
# ScandiBERT

Note note: The model has been updated on 2022-09-27

The model was trained on the data shown in the table below. Batch size was 8.8k, the model was trained for 72 epochs on 24 V100 cards for about 2 weeks.

| Language  | Data                                  | Size   |
|-----------|---------------------------------------|--------|
| Icelandic | See IceBERT paper                     | 16 GB  |
| Danish    | Danish Gigaword Corpus (incl Twitter) | 4,7 GB |
| Norwegian | NCC corpus                            | 42 GB  |
| Swedish   | Swedish Gigaword Corpus               | 3,4 GB |
| Faroese   | FC3 + Sosialurinn + Bible             | 69 MB  |


Note: At an earlier date a half trained model went up here, it has since been removed. The model has since been updated.

This is a Scandinavian BERT model trained on a large collection of Danish, Faroese, Icelandic, Norwegian and Swedish text. It is currently the highest ranking model on the ScandEval leaderbord https://scandeval.github.io/pretrained/

If you find this model useful, please cite

```
@inproceedings{snaebjarnarson-etal-2023-transfer,
    title = "{T}ransfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese",
    author = "Snæbjarnarson, Vésteinn  and
      Simonsen, Annika  and
      Glavaš, Goran  and
      Vulić, Ivan",
    booktitle = "Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)",
    month = "may 22--24",
    year = "2023",
    address = "Tórshavn, Faroe Islands",
    publisher = {Link{\"o}ping University Electronic Press, Sweden},
}
```
{"language": ["is", "da", "sv", "no", "fo"], "license": "agpl-3.0", "tags": ["roberta", "icelandic", "norwegian", "faroese", "danish", "swedish", "masked-lm", "pytorch"], "datasets": ["vesteinn/FC3", "vesteinn/IC3", "mideind/icelandic-common-crawl-corpus-IC3", "NbAiLab/NCC", "DDSC/partial-danish-gigaword-no-twitter"], "widget": [{"text": "Fina lilla<mask>, jag vill inte bliva stur."}, {"text": "Nu ved jeg, at du frygter<mask> og end ikke vil n\u00e6gte mig din eneste s\u00f8n.."}, {"text": "\u00dea\u00f0 er vorhret \u00e1<mask>, napur vindur sem hv\u00edn."}, {"text": "Ja, Gud signi<mask>, m\u00edtt land."}, {"text": "Alle dyrene i<mask> m\u00e5 v\u00e6re venner."}]}
transformers pytorch safetensors xlm-roberta fill-mask roberta icelandic norwegian faroese danish swedish masked-lm is da sv no fo dataset:vesteinn/FC3 dataset:vesteinn/IC3 dataset:mideind/icelandic-common-crawl-corpus-IC3 dataset:NbAiLab/NCC dataset:DDSC/partial-danish-gigaword-no-twitter doi:10.57967/hf/0382 license:agpl-3.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #safetensors #xlm-roberta #fill-mask #roberta #icelandic #norwegian #faroese #danish #swedish #masked-lm #is #da #sv #no #fo #dataset-vesteinn/FC3 #dataset-vesteinn/IC3 #dataset-mideind/icelandic-common-crawl-corpus-IC3 #dataset-NbAiLab/NCC #dataset-DDSC/partial-danish-gigaword-no-twitter #doi-10.57967/hf/0382 #license-agpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 

ScandiBERT
==========


Note note: The model has been updated on 2022-09-27


The model was trained on the data shown in the table below. Batch size was 8.8k, the model was trained for 72 epochs on 24 V100 cards for about 2 weeks.


Language: Icelandic, Data: See IceBERT paper, Size: 16 GB
Language: Danish, Data: Danish Gigaword Corpus (incl Twitter), Size: 4,7 GB
Language: Norwegian, Data: NCC corpus, Size: 42 GB
Language: Swedish, Data: Swedish Gigaword Corpus, Size: 3,4 GB
Language: Faroese, Data: FC3 + Sosialurinn + Bible, Size: 69 MB


Note: At an earlier date a half trained model went up here, it has since been removed. The model has since been updated.


This is a Scandinavian BERT model trained on a large collection of Danish, Faroese, Icelandic, Norwegian and Swedish text. It is currently the highest ranking model on the ScandEval leaderbord URL


If you find this model useful, please cite



TAGS
#transformers #pytorch #safetensors #xlm-roberta #fill-mask #roberta #icelandic #norwegian #faroese #danish #swedish #masked-lm #is #da #sv #no #fo #dataset-vesteinn/FC3 #dataset-vesteinn/IC3 #dataset-mideind/icelandic-common-crawl-corpus-IC3 #dataset-NbAiLab/NCC #dataset-DDSC/partial-danish-gigaword-no-twitter #doi-10.57967/hf/0382 #license-agpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 

passage: TAGS
#transformers #pytorch #safetensors #xlm-roberta #fill-mask #roberta #icelandic #norwegian #faroese #danish #swedish #masked-lm #is #da #sv #no #fo #dataset-vesteinn/FC3 #dataset-vesteinn/IC3 #dataset-mideind/icelandic-common-crawl-corpus-IC3 #dataset-NbAiLab/NCC #dataset-DDSC/partial-danish-gigaword-no-twitter #doi-10.57967/hf/0382 #license-agpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 
