This project pretrains a [`roberta-base`](https://huggingface.co/roberta-base) on the *Alemannic* (`als`) data subset of the [OSCAR](https://oscar-corpus.com/) corpus in JAX/Flax.

We will be using the masked-language modeling loss for pretraining.
{}
transformers jax tensorboard roberta fill-mask autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #jax #tensorboard #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

This project pretrains a 'roberta-base' on the *Alemannic* ('als') data subset of the OSCAR corpus in JAX/Flax.

We will be using the masked-language modeling loss for pretraining.
TAGS
#transformers #jax #tensorboard #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

passage: TAGS
#transformers #jax #tensorboard #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 
