# Transformer language model for Croatian and Serbian
Trained on 43GB datasets that contain Croatian and Serbian language for one epochs (9.6 mil. steps, 3 epochs).
Leipzig Corpus, OSCAR, srWac, hrWac, cc100-hr and cc100-sr  datasets

Validation number of exampels run for perplexity:1620487 sentences
Perplexity:6.02
Start loss: 8.6
Final loss: 2.0
Thoughts: Model could be trained more, the training did not stagnate.

| Model                          | #params                        | Arch. | Training data                     |
|--------------------------------|--------------------------------|-------|-----------------------------------|
| `Andrija/SRoBERTa-F` | 80M   | Fifth | Leipzig Corpus, OSCAR, srWac, hrWac, cc100-hr and cc100-sr  (43 GB of text)             |
{"language": ["hr", "sr", "multilingual"], "license": "apache-2.0", "tags": ["masked-lm"], "datasets": ["oscar", "srwac", "leipzig", "cc100", "hrwac"], "widget": [{"text": "Ovo je po\u010detak <mask>."}]}
transformers pytorch tensorboard roberta fill-mask masked-lm hr sr multilingual dataset:oscar dataset:srwac dataset:leipzig dataset:cc100 dataset:hrwac license:apache-2.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #masked-lm #hr #sr #multilingual #dataset-oscar #dataset-srwac #dataset-leipzig #dataset-cc100 #dataset-hrwac #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 

Transformer language model for Croatian and Serbian
===================================================


Trained on 43GB datasets that contain Croatian and Serbian language for one epochs (9.6 mil. steps, 3 epochs).
Leipzig Corpus, OSCAR, srWac, hrWac, cc100-hr and cc100-sr datasets


Validation number of exampels run for perplexity:1620487 sentences
Perplexity:6.02
Start loss: 8.6
Final loss: 2.0
Thoughts: Model could be trained more, the training did not stagnate.




TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #masked-lm #hr #sr #multilingual #dataset-oscar #dataset-srwac #dataset-leipzig #dataset-cc100 #dataset-hrwac #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 

passage: TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #masked-lm #hr #sr #multilingual #dataset-oscar #dataset-srwac #dataset-leipzig #dataset-cc100 #dataset-hrwac #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
