This is a `gpt2` model, finetuned on the Wikitext-103 dataset.

It achieves a perplexity of **14.84** using a "sliding window" context, using the `run_clm.py` script at [https://github.com/neulab/knn-transformers](https://github.com/neulab/knn-transformers).

| Base LM:        | `distilgpt2` | `gpt2` | 
| :---            |    ----:   |     ---: |
| base perplexity | 18.25      | 14.84    |
| +kNN-LM          |  15.03     |   12.57  |
| +RetoMaton       | **14.70**  |  **12.46**    |

This model was released as part of the paper ["Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval"](https://arxiv.org/pdf/2201.12431.pdf) (ICML'2022).

For more information, see: [https://github.com/neulab/knn-transformers](https://github.com/neulab/knn-transformers)

If you use this model, please cite:
```
@inproceedings{alon2022neuro,
  title={Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval},
  author={Alon, Uri and Xu, Frank and He, Junxian and Sengupta, Sudipta and Roth, Dan and Neubig, Graham},
  booktitle={International Conference on Machine Learning},
  pages={468--485},
  year={2022},
  organization={PMLR}
}
```
{}
transformers pytorch gpt2 text-generation arxiv:2201.12431 autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #gpt2 #text-generation #arxiv-2201.12431 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

This is a 'gpt2' model, finetuned on the Wikitext-103 dataset.


It achieves a perplexity of 14.84 using a "sliding window" context, using the 'run\_clm.py' script at URL



This model was released as part of the paper "Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval" (ICML'2022).


For more information, see: URL


If you use this model, please cite:



TAGS
#transformers #pytorch #gpt2 #text-generation #arxiv-2201.12431 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

passage: TAGS
#transformers #pytorch #gpt2 #text-generation #arxiv-2201.12431 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
