#Slovak RoBERTA Masked Language Model

###83Mil Parameters in small model

Medium and Large models coming soon!

RoBERTA pretrained tokenizer vocab and merges included.

---

##Training params:
- **Dataset**:
  8GB Slovak Monolingual dataset including ParaCrawl (monolingual), OSCAR, and several gigs of my own findings and cleaning.
- **Preprocessing**:
  Tokenized with a pretrained ByteLevelBPETokenizer trained on the same dataset. Uncased, with s, pad, /s, unk, and mask special tokens.
- **Evaluation results**:
  - Mnoho ľudí tu MASK
    - žije.
    - žijú.
    - je.
    - trpí.
  - Ako sa MASK
    - máte
    - máš
    - má
    - hovorí
  - Plážová sezóna pod Zoborom patrí medzi MASK obdobia.
    - ročné
    - najkrajšie
    - najobľúbenejšie
    - najnáročnejšie
    
- **Limitations**:
  The current model is fairly small, although it works very well. This model is meant to be finetuned on downstream tasks e.g. Part-of-Speech tagging, Question Answering, anything in GLUE or SUPERGLUE.
  
- **Credit**:
  If you use this or any of my models in research or professional work, please credit me - Christopher Brousseau in said work.
{}
transformers pytorch safetensors roberta fill-mask autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

#Slovak RoBERTA Masked Language Model

###83Mil Parameters in small model

Medium and Large models coming soon!

RoBERTA pretrained tokenizer vocab and merges included.

---

##Training params:
- Dataset:
  8GB Slovak Monolingual dataset including ParaCrawl (monolingual), OSCAR, and several gigs of my own findings and cleaning.
- Preprocessing:
  Tokenized with a pretrained ByteLevelBPETokenizer trained on the same dataset. Uncased, with s, pad, /s, unk, and mask special tokens.
- Evaluation results:
  - Mnoho ľudí tu MASK
    - žije.
    - žijú.
    - je.
    - trpí.
  - Ako sa MASK
    - máte
    - máš
    - má
    - hovorí
  - Plážová sezóna pod Zoborom patrí medzi MASK obdobia.
    - ročné
    - najkrajšie
    - najobľúbenejšie
    - najnáročnejšie
    
- Limitations:
  The current model is fairly small, although it works very well. This model is meant to be finetuned on downstream tasks e.g. Part-of-Speech tagging, Question Answering, anything in GLUE or SUPERGLUE.
  
- Credit:
  If you use this or any of my models in research or professional work, please credit me - Christopher Brousseau in said work.
TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

passage: TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 
