Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
This is a **RoBERTa-base** model trained from scratch in Spanish.
|
| 2 |
+
|
| 3 |
+
The training dataset is mc4 (1) subsampling documents to a total of about 50 million examples. Sampling is biased towards average perplexity values, discarding more often documents with very large values (poor quality) of very small values (short, repetitive texts).
|
| 4 |
+
|
| 5 |
+
This model takes the one using sequence length 128 (2) and trains during 25.000 steps using sequence length 512.
|
| 6 |
+
|
| 7 |
+
(1) https://huggingface.co/datasets/bertin-project/mc4-es-sampled
|
| 8 |
+
(2) https://huggingface.co/bertin-project/bertin-base-gaussian
|