
# RoBERTa Greek base model

Pretrained model on Greek language with the Masked Language Modeling (MLM) objective using [Hugging Face's](https://huggingface.co/) [Transformers](https://github.com/huggingface/transformers) library. This model is *NOT* case-sensitive and all Greek diacritics retained.

### How to use

You can use this model directly with a pipeline for masked language modeling:

```python
# example url 
# https://www.news247.gr/politiki/misologa-maximoy-gia-tin-ekthesi-tsiodra-lytra-gia-ti-thnitotita-ektos-meth.9462425.html 
# not present in train/eval set
from transformers import pipeline
pipe = pipeline('fill-mask', model='cvcio/roberta-el-news')
pipe(
    'Η κυβέρνηση μουδιασμένη από τη <mask> της έκθεσης Τσιόδρα-Λύτρα, '
    'επιχειρεί χωρίς να απαντά ουσιαστικά να ρίξει ευθύνες στον ΣΥΡΙΖΑ, που κυβερνούσε πριν... 2 χρόνια.'
)
# outputs
[
    {
        'sequence': 'Η κυβέρνηση μουδιασμένη από τη δημοσιοποίηση της έκθεσης Τσιόδρα-Λύτρα, επιχειρεί χωρίς να απαντά ουσιαστικά να ρίξει ευθύνες στον ΣΥΡΙΖΑ, που κυβερνούσε πριν... 2 χρόνια.', 
        'score': 0.5881184339523315, 'token': 20235, 'token_str': ' δημοσιοποίηση'
    }, 
    {
        'sequence': 'Η κυβέρνηση μουδιασμένη από τη δημοσίευση της έκθεσης Τσιόδρα-Λύτρα, επιχειρεί χωρίς να απαντά ουσιαστικά να ρίξει ευθύνες στον ΣΥΡΙΖΑ, που κυβερνούσε πριν... 2 χρόνια.', 
        'score': 0.05952141433954239, 'token': 9696, 'token_str': ' δημοσίευση'
    }, 
    {
        'sequence': 'Η κυβέρνηση μουδιασμένη από τη διαχείριση της έκθεσης Τσιόδρα-Λύτρα, επιχειρεί χωρίς να απαντά ουσιαστικά να ρίξει ευθύνες στον ΣΥΡΙΖΑ, που κυβερνούσε πριν... 2 χρόνια.', 
        'score': 0.029887061566114426, 'token': 4315, 'token_str': ' διαχείριση'
    }, 
    {
        'sequence': 'Η κυβέρνηση μουδιασμένη από τη διαρροή της έκθεσης Τσιόδρα-Λύτρα, επιχειρεί χωρίς να απαντά ουσιαστικά να ρίξει ευθύνες στον ΣΥΡΙΖΑ, που κυβερνούσε πριν... 2 χρόνια.', 
        'score': 0.022848669439554214, 'token': 24940, 'token_str': ' διαρροή'
    }, 
    {
        'sequence': 'Η κυβέρνηση μουδιασμένη από τη ματαίωση της έκθεσης Τσιόδρα-Λύτρα, επιχειρεί χωρίς να απαντά ουσιαστικά να ρίξει ευθύνες στον ΣΥΡΙΖΑ, που κυβερνούσε πριν... 2 χρόνια.', 
        'score': 0.01729060709476471, 'token': 46913, 'token_str': ' ματαίωση'
    }
]
```

## Training data

The model was pretrained on 8 millon unique news articles (~ approx 160M sentences, 33GB of text), collected with [MediaWatch](https://mediawatch.io/), from October 2016 upto December 2021.

## Preprocessing

The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50,265. During the preprocessing we only unescaped html text to the correspoing Unicode characters (ex. `&amp;` => `&`).

## Pretraining

The model was pretrained using an NVIDIA A10 GPU for 3 epochs (~ approx 760K steps, 182 hours) with a batch size of 14 (x2 gradient accumulation steps = 28) and a sequence length of 512 tokens. The optimizer used is Adam with a learning rate of 5e-5, and linear decay of the learning rate.

### Training results

| epochs | steps   | train/train_loss | train/loss  | eval/loss |
|-------:|--------:|-----------------:|------------:|----------:|
| 3      | 765,414 | 0.3960           | 1.2356      | 0.9028    |

### Evaluation results

The model fine-tuned on ner task using the [elNER](https://github.com/nmpartzio/elner) dataset and achieved the following results:

| task | epochs | lr   | batch | dataset | precision | recall | f1     | accuracy |
|-----:|-------:|-----:|------:|--------:|----------:|-------:|-------:|---------:|
| ner  | 5      | 1e-5 | 16/16 | elNER4  | 0.8954    | 0.9280 | 0.9114 | 0.9872   |
| ner  | 5      | 1e-4 | 16/16 | elNER18 | 0.9069    | 0.9268 | 0.9168 | 0.9823   |

### Training hyperparameters

The following hyperparameters were used during training:

- learning_rate: 5e-5
- train_batch_size: 14
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 28
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Framework versions

- Transformers 4.13.0
- Pytorch 1.9.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3

## Authors

Dimitris Papaevagelou - [@andefined](https://github.com/andefined)

## About Us

[Civic Information Office](https://cvcio.org/) is a Non Profit Organization based in Athens, Greece focusing on creating technology and research products for the public interest.
{"language": "el", "license": "gpl-3.0", "tags": ["generated_from_trainer", "roberta", "Greek", "news", "transformers"], "widget": [{"text": "\u0397 \u03ba\u03c5\u03b2\u03ad\u03c1\u03bd\u03b7\u03c3\u03b7 \u03bc\u03bf\u03c5\u03b4\u03b9\u03b1\u03c3\u03bc\u03ad\u03bd\u03b7 \u03b1\u03c0\u03cc \u03c4\u03b7 <mask> \u03c4\u03b7\u03c2 \u03ad\u03ba\u03b8\u03b5\u03c3\u03b7\u03c2 \u03a4\u03c3\u03b9\u03cc\u03b4\u03c1\u03b1-\u039b\u03cd\u03c4\u03c1\u03b1, \u03b5\u03c0\u03b9\u03c7\u03b5\u03b9\u03c1\u03b5\u03af \u03c7\u03c9\u03c1\u03af\u03c2 \u03bd\u03b1 \u03b1\u03c0\u03b1\u03bd\u03c4\u03ac \u03bf\u03c5\u03c3\u03b9\u03b1\u03c3\u03c4\u03b9\u03ba\u03ac \u03bd\u03b1 \u03c1\u03af\u03be\u03b5\u03b9 \u03b5\u03c5\u03b8\u03cd\u03bd\u03b5\u03c2 \u03c3\u03c4\u03bf\u03bd \u03a3\u03a5\u03a1\u0399\u0396\u0391, \u03c0\u03bf\u03c5 \u03ba\u03c5\u03b2\u03b5\u03c1\u03bd\u03bf\u03cd\u03c3\u03b5 \u03c0\u03c1\u03b9\u03bd... 2 \u03c7\u03c1\u03cc\u03bd\u03b9\u03b1."}], "model-index": [{"name": "roberta-el-news", "results": []}]}
transformers pytorch safetensors roberta fill-mask generated_from_trainer Greek news el doi:10.57967/hf/0712 license:gpl-3.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #generated_from_trainer #Greek #news #el #doi-10.57967/hf/0712 #license-gpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 

RoBERTa Greek base model
========================


Pretrained model on Greek language with the Masked Language Modeling (MLM) objective using Hugging Face's Transformers library. This model is *NOT* case-sensitive and all Greek diacritics retained.


### How to use


You can use this model directly with a pipeline for masked language modeling:


Training data
-------------


The model was pretrained on 8 millon unique news articles (~ approx 160M sentences, 33GB of text), collected with MediaWatch, from October 2016 upto December 2021.


Preprocessing
-------------


The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50,265. During the preprocessing we only unescaped html text to the correspoing Unicode characters (ex. '&' => '&').


Pretraining
-----------


The model was pretrained using an NVIDIA A10 GPU for 3 epochs (~ approx 760K steps, 182 hours) with a batch size of 14 (x2 gradient accumulation steps = 28) and a sequence length of 512 tokens. The optimizer used is Adam with a learning rate of 5e-5, and linear decay of the learning rate.


### Training results



### Evaluation results


The model fine-tuned on ner task using the elNER dataset and achieved the following results:



### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-5
* train\_batch\_size: 14
* eval\_batch\_size: 8
* seed: 42
* gradient\_accumulation\_steps: 2
* total\_train\_batch\_size: 28
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 3.0


### Framework versions


* Transformers 4.13.0
* Pytorch 1.9.0+cu111
* Datasets 1.16.1
* Tokenizers 0.10.3


Authors
-------


Dimitris Papaevagelou - @andefined


About Us
--------


Civic Information Office is a Non Profit Organization based in Athens, Greece focusing on creating technology and research products for the public interest.



### How to use


You can use this model directly with a pipeline for masked language modeling:


Training data
-------------


The model was pretrained on 8 millon unique news articles (~ approx 160M sentences, 33GB of text), collected with MediaWatch, from October 2016 upto December 2021.


Preprocessing
-------------


The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50,265. During the preprocessing we only unescaped html text to the correspoing Unicode characters (ex. '&' => '&').


Pretraining
-----------


The model was pretrained using an NVIDIA A10 GPU for 3 epochs (~ approx 760K steps, 182 hours) with a batch size of 14 (x2 gradient accumulation steps = 28) and a sequence length of 512 tokens. The optimizer used is Adam with a learning rate of 5e-5, and linear decay of the learning rate. ### Training results ### Evaluation results


The model fine-tuned on ner task using the elNER dataset and achieved the following results: ### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-5
* train\_batch\_size: 14
* eval\_batch\_size: 8
* seed: 42
* gradient\_accumulation\_steps: 2
* total\_train\_batch\_size: 28
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 3.0 ### Framework versions


* Transformers 4.13.0
* Pytorch 1.9.0+cu111
* Datasets 1.16.1
* Tokenizers 0.10.3


Authors
-------


Dimitris Papaevagelou - @andefined


About Us
--------


Civic Information Office is a Non Profit Organization based in Athens, Greece focusing on creating technology and research products for the public interest.
TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #generated_from_trainer #Greek #news #el #doi-10.57967/hf/0712 #license-gpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 
 ### How to use


You can use this model directly with a pipeline for masked language modeling:


Training data
-------------


The model was pretrained on 8 millon unique news articles (~ approx 160M sentences, 33GB of text), collected with MediaWatch, from October 2016 upto December 2021.


Preprocessing
-------------


The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50,265. During the preprocessing we only unescaped html text to the correspoing Unicode characters (ex. '&' => '&').


Pretraining
-----------


The model was pretrained using an NVIDIA A10 GPU for 3 epochs (~ approx 760K steps, 182 hours) with a batch size of 14 (x2 gradient accumulation steps = 28) and a sequence length of 512 tokens. The optimizer used is Adam with a learning rate of 5e-5, and linear decay of the learning rate. ### Training results ### Evaluation results


The model fine-tuned on ner task using the elNER dataset and achieved the following results: ### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-5
* train\_batch\_size: 14
* eval\_batch\_size: 8
* seed: 42
* gradient\_accumulation\_steps: 2
* total\_train\_batch\_size: 28
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 3.0 ### Framework versions


* Transformers 4.13.0
* Pytorch 1.9.0+cu111
* Datasets 1.16.1
* Tokenizers 0.10.3


Authors
-------


Dimitris Papaevagelou - @andefined


About Us
--------


Civic Information Office is a Non Profit Organization based in Athens, Greece focusing on creating technology and research products for the public interest.
passage: TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #generated_from_trainer #Greek #news #el #doi-10.57967/hf/0712 #license-gpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 
### How to use


You can use this model directly with a pipeline for masked language modeling:


Training data
-------------


The model was pretrained on 8 millon unique news articles (~ approx 160M sentences, 33GB of text), collected with MediaWatch, from October 2016 upto December 2021.


Preprocessing
-------------


The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50,265. During the preprocessing we only unescaped html text to the correspoing Unicode characters (ex. '&' => '&').


Pretraining
-----------


The model was pretrained using an NVIDIA A10 GPU for 3 epochs (~ approx 760K steps, 182 hours) with a batch size of 14 (x2 gradient accumulation steps = 28) and a sequence length of 512 tokens. The optimizer used is Adam with a learning rate of 5e-5, and linear decay of the learning rate.### Training results### Evaluation results


The model fine-tuned on ner task using the elNER dataset and achieved the following results:### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-5
* train\_batch\_size: 14
* eval\_batch\_size: 8
* seed: 42
* gradient\_accumulation\_steps: 2
* total\_train\_batch\_size: 28
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 3.0