
# no-phone-gpt2

This is a test to remove memorized private information, such as phone numbers, from a small GPT-2 model. This should not generate valid phone numbers.

Inspired by BAIR privacy research:
- https://bair.berkeley.edu/blog/2019/08/13/memorization/
- https://bair.berkeley.edu/blog/2020/12/20/lmmem/

[Blog post](https://mapmeld.medium.com/scrambling-memorized-info-in-gpt-2-60753d7652d8)

## Process

- All +## and +### tokens were replaced with new, randomly-selected 2- and 3-digit numbers in the vocab.json and tokenizer.json. You can identify these in outputs because the new tokens start with ^^.
- Input and output embeddings for +## and +### tokens were moved to the +00 and +000 embeddings.
- Removed associations between numbers from merges.txt

Using a library such as [ecco](https://github.com/jalammar/ecco), probabilities for next number token look equally likely, with +000 preferred.

Code: https://colab.research.google.com/drive/1X31TIZjmxlXMXAzQrR3Fl1AnLzGBCpWf#scrollTo=0GVFwrAgY68J

### Future goals

- Add new +### tokens to rebuild number generation
- Fine-tune new tokens on counting numbers and ended phone numbers
- Use [gpt2-large](https://huggingface.co/gpt2-large)

### BibTeX entry and citation info

Original GPT-2:

```bibtex
@article{radford2019language,
  title={Language Models are Unsupervised Multitask Learners},
  author={Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
  year={2019}
}
```

{"language": "en", "license": "mit", "tags": ["exbert"]}
transformers pytorch jax safetensors gpt2 text-generation exbert en license:mit autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #jax #safetensors #gpt2 #text-generation #exbert #en #license-mit #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 


# no-phone-gpt2

This is a test to remove memorized private information, such as phone numbers, from a small GPT-2 model. This should not generate valid phone numbers.

Inspired by BAIR privacy research:
- URL
- URL

Blog post

## Process

- All +## and +### tokens were replaced with new, randomly-selected 2- and 3-digit numbers in the URL and URL. You can identify these in outputs because the new tokens start with ^^.
- Input and output embeddings for +## and +### tokens were moved to the +00 and +000 embeddings.
- Removed associations between numbers from URL

Using a library such as ecco, probabilities for next number token look equally likely, with +000 preferred.

Code: URL

### Future goals

- Add new +### tokens to rebuild number generation
- Fine-tune new tokens on counting numbers and ended phone numbers
- Use gpt2-large

### BibTeX entry and citation info

Original GPT-2:



# no-phone-gpt2

This is a test to remove memorized private information, such as phone numbers, from a small GPT-2 model. This should not generate valid phone numbers.

Inspired by BAIR privacy research:
- URL
- URL

Blog post ## Process

- All +## and +### tokens were replaced with new, randomly-selected 2- and 3-digit numbers in the URL and URL. You can identify these in outputs because the new tokens start with ^^.
- Input and output embeddings for +## and +### tokens were moved to the +00 and +000 embeddings.
- Removed associations between numbers from URL

Using a library such as ecco, probabilities for next number token look equally likely, with +000 preferred.

Code: URL ### Future goals

- Add new +### tokens to rebuild number generation
- Fine-tune new tokens on counting numbers and ended phone numbers
- Use gpt2-large ### BibTeX entry and citation info

Original GPT-2:
TAGS
#transformers #pytorch #jax #safetensors #gpt2 #text-generation #exbert #en #license-mit #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
 # no-phone-gpt2

This is a test to remove memorized private information, such as phone numbers, from a small GPT-2 model. This should not generate valid phone numbers.

Inspired by BAIR privacy research:
- URL
- URL

Blog post ## Process

- All +## and +### tokens were replaced with new, randomly-selected 2- and 3-digit numbers in the URL and URL. You can identify these in outputs because the new tokens start with ^^.
- Input and output embeddings for +## and +### tokens were moved to the +00 and +000 embeddings.
- Removed associations between numbers from URL

Using a library such as ecco, probabilities for next number token look equally likely, with +000 preferred.

Code: URL ### Future goals

- Add new +### tokens to rebuild number generation
- Fine-tune new tokens on counting numbers and ended phone numbers
- Use gpt2-large ### BibTeX entry and citation info

Original GPT-2:
passage: TAGS
#transformers #pytorch #jax #safetensors #gpt2 #text-generation #exbert #en #license-mit #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
# no-phone-gpt2

This is a test to remove memorized private information, such as phone numbers, from a small GPT-2 model. This should not generate valid phone numbers.

Inspired by BAIR privacy research:
- URL
- URL

Blog post## Process

- All +## and +### tokens were replaced with new, randomly-selected 2- and 3-digit numbers in the URL and URL. You can identify these in outputs because the new tokens start with ^^.
- Input and output embeddings for +## and +### tokens were moved to the +00 and +000 embeddings.
- Removed associations between numbers from URL

Using a library such as ecco, probabilities for next number token look equally likely, with +000 preferred.

Code: URL### Future goals

- Add new +### tokens to rebuild number generation
- Fine-tune new tokens on counting numbers and ended phone numbers
- Use gpt2-large### BibTeX entry and citation info

Original GPT-2: