(Beta) RoBERTa and XLM-R Models
TorchText has added support for pre-trained RoBERTa and XLM-R models. It would allow users to train end-2-end Transformer Encoder based models on standard NLP tasks using TorchText.
More specifically:

The models are torchscriptable and hence can be employed for production use-cases.
The model APIs let users to easily attach custom task-specific heads with pre-trained encoders.
The API also comes equipped with data pre-processing transforms to match the pre-trained weights and model configuration.

We have added a tutorial to demonstrate SST-2 binary text classification task with pre-trained XLM-R base architecture.
For additional details on model APIs and usage examples, please refer to the documentation.
(Beta) byte-level BPE tokenizer