This is a model checkpoint for ["Should You Mask 15% in Masked Language Modeling"](https://arxiv.org/abs/2202.08005) [(code)](https://github.com/princeton-nlp/DinkyTrain.git). We use pre layer norm, which is not supported by HuggingFace. To use our model, go to our [github repo](https://github.com/princeton-nlp/DinkyTrain.git), download our code, and import the RoBERTa class from `huggingface/modeling_roberta_prelayernorm.py`. For example,

``` bash
from huggingface.modeling_roberta_prelayernorm import RobertaForMaskedLM, RobertaForSequenceClassification
```
{"inference": false}
transformers pytorch roberta fill-mask arxiv:2202.08005 autotrain_compatible region:us
TAGS
#transformers #pytorch #roberta #fill-mask #arxiv-2202.08005 #autotrain_compatible #region-us 

This is a model checkpoint for "Should You Mask 15% in Masked Language Modeling" (code). We use pre layer norm, which is not supported by HuggingFace. To use our model, go to our github repo, download our code, and import the RoBERTa class from 'huggingface/modeling_roberta_prelayernorm.py'. For example,


TAGS
#transformers #pytorch #roberta #fill-mask #arxiv-2202.08005 #autotrain_compatible #region-us 

passage: TAGS
#transformers #pytorch #roberta #fill-mask #arxiv-2202.08005 #autotrain_compatible #region-us 
