
# IceBERT

IceBERT was trained with fairseq using the RoBERTa-base architecture. The training data used is shown in the table below.

| Dataset                                              | Size    | Tokens |
|------------------------------------------------------|---------|--------|
| Icelandic Gigaword Corpus v20.05 (IGC)               | 8.2 GB  | 1,388M |
| Icelandic Common Crawl Corpus (IC3)                  | 4.9 GB  | 824M   |
| Greynir News articles                                | 456 MB  | 76M    |
| Icelandic Sagas                                      | 9 MB    | 1.7M   |
| Open Icelandic e-books (Rafbókavefurinn)             | 14 MB   | 2.6M   |
| Data from the medical library of Landspitali         | 33 MB   | 5.2M   |
| Student theses from Icelandic universities (Skemman) | 2.2 GB  | 367M   |
| Total                                                | 15.8 GB | 2,664M |


If you find this model useful, please cite

```
@inproceedings{snaebjarnarson-etal-2022-warm,
    title = "A Warm Start and a Clean Crawled Corpus - A Recipe for Good Language Models",
    author = "Sn{\ae}bjarnarson, V{\'e}steinn  and
      S{\'\i}monarson, Haukur Barri  and
      Ragnarsson, P{\'e}tur Orri  and
      Ing{\'o}lfsd{\'o}ttir, Svanhv{\'\i}t Lilja  and
      J{\'o}nsson, Haukur  and
      Thorsteinsson, Vilhjalmur  and
      Einarsson, Hafsteinn",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.464",
    pages = "4356--4366",
}
```
{"language": "is", "license": "agpl-3.0", "tags": ["roberta", "icelandic", "masked-lm", "pytorch"], "datasets": ["mideind/icelandic-common-crawl-corpus-IC3"], "widget": [{"text": "M\u00e1 bj\u00f3\u00f0a \u00fe\u00e9r <mask> \u00ed kv\u00f6ld?"}, {"text": "Forseti <mask> er \u00e1g\u00e6t."}, {"text": "S\u00fapan var <mask> \u00e1 brag\u00f0i\u00f0."}]}
transformers pytorch safetensors roberta fill-mask icelandic masked-lm is dataset:mideind/icelandic-common-crawl-corpus-IC3 license:agpl-3.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #icelandic #masked-lm #is #dataset-mideind/icelandic-common-crawl-corpus-IC3 #license-agpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 

IceBERT
=======


IceBERT was trained with fairseq using the RoBERTa-base architecture. The training data used is shown in the table below.


Dataset: Icelandic Gigaword Corpus v20.05 (IGC), Size: 8.2 GB, Tokens: 1,388M
Dataset: Icelandic Common Crawl Corpus (IC3), Size: 4.9 GB, Tokens: 824M
Dataset: Greynir News articles, Size: 456 MB, Tokens: 76M
Dataset: Icelandic Sagas, Size: 9 MB, Tokens: 1.7M
Dataset: Open Icelandic e-books (Rafbókavefurinn), Size: 14 MB, Tokens: 2.6M
Dataset: Data from the medical library of Landspitali, Size: 33 MB, Tokens: 5.2M
Dataset: Student theses from Icelandic universities (Skemman), Size: 2.2 GB, Tokens: 367M
Dataset: Total, Size: 15.8 GB, Tokens: 2,664M


If you find this model useful, please cite



TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #icelandic #masked-lm #is #dataset-mideind/icelandic-common-crawl-corpus-IC3 #license-agpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 

passage: TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #icelandic #masked-lm #is #dataset-mideind/icelandic-common-crawl-corpus-IC3 #license-agpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 
