

# aristoBERTo

aristoBERTo is a transformer model for ancient Greek, a low resource  language. We initialized the pre-training with weights from [GreekBERT](https://huggingface.co/nlpaueb/bert-base-greek-uncased-v1), a Greek version of BERT which was trained on a large corpus of modern Greek (~ 30 GB of texts). We continued the pre-training with an ancient Greek corpus of about 900 MB, which was scrapped from the web and post-processed. Duplicate texts and editorial punctuation were removed. 

Applied to the processing of ancient Greek, aristoBERTo outperforms xlm-roberta-base and mdeberta in most downstream tasks like the labeling of POS, MORPH, DEP and LEMMA. 

aristoBERTo is provided by the [Diogenet project](https://diogenet.ucsd.edu) of the University of California, San Diego. 
 

## Intended uses

This model was created for fine-tuning with spaCy and the ancient Greek Universal Dependency datasets as well as a NER corpus produced by the [Diogenet project](https://diogenet.ucsd.edu). As a fill-mask model, AristoBERTo can also be used in the restoration of damaged Greek papyri, inscriptions, and manuscripts. 


It achieves the following results on the evaluation set:
- Loss: 1.6323

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| 1.377         | 20.0  | 3414220 | 1.6314          |


### Framework versions

- Transformers 4.14.0.dev0
- Pytorch 1.10.0+cu102
- Datasets 1.16.1
- Tokenizers 0.10.3

{"language": ["grc"], "widget": [{"text": "\u03a0\u03bb\u03ac\u03c4\u03c9\u03bd \u1f41 \u03a0\u03b5\u03c1\u03b9\u03ba\u03c4\u03b9\u03cc\u03bd\u03b7\u03c2 [MASK] \u03b3\u03ad\u03bd\u03bf\u03c2 \u1f00\u03bd\u03ad\u03c6\u03b5\u03c1\u03b5\u03bd \u03b5\u1f30\u03c2 \u03a3\u03cc\u03bb\u03c9\u03bd\u03b1."}, {"text": "\u1f41 \u039a\u03c1\u03b9\u03c4\u03af\u03b1\u03c2 \u1f00\u03c0\u03ad\u03b2\u03bb\u03b5\u03c8\u03b5 [MASK] \u03c4\u1f74\u03bd \u03b8\u03cd\u03c1\u03b1\u03bd."}, {"text": "\u03c0\u03c1\u1ff6\u03c4\u03bf\u03b9 \u03b4\u1f72 \u03ba\u03b1\u1f76 \u03bf\u1f50\u03bd\u03cc\u03bc\u03b1\u03c4\u03b1 \u1f31\u03c1\u1f70 \u1f14\u03b3\u03bd\u03c9\u03c3\u03b1\u03bd \u03ba\u03b1\u1f76 [MASK] \u1f31\u03c1\u03bf\u1f7a\u03c2 \u1f14\u03bb\u03b5\u03be\u03b1\u03bd."}], "model-index": [{"name": "aristoBERTo", "results": []}]}
transformers pytorch bert fill-mask grc autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #bert #fill-mask #grc #autotrain_compatible #endpoints_compatible #region-us 

aristoBERTo
===========


aristoBERTo is a transformer model for ancient Greek, a low resource language. We initialized the pre-training with weights from GreekBERT, a Greek version of BERT which was trained on a large corpus of modern Greek (~ 30 GB of texts). We continued the pre-training with an ancient Greek corpus of about 900 MB, which was scrapped from the web and post-processed. Duplicate texts and editorial punctuation were removed.


Applied to the processing of ancient Greek, aristoBERTo outperforms xlm-roberta-base and mdeberta in most downstream tasks like the labeling of POS, MORPH, DEP and LEMMA.


aristoBERTo is provided by the Diogenet project of the University of California, San Diego.


Intended uses
-------------


This model was created for fine-tuning with spaCy and the ancient Greek Universal Dependency datasets as well as a NER corpus produced by the Diogenet project. As a fill-mask model, AristoBERTo can also be used in the restoration of damaged Greek papyri, inscriptions, and manuscripts.


It achieves the following results on the evaluation set:


* Loss: 1.6323


Training procedure
------------------


### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-05
* train\_batch\_size: 16
* eval\_batch\_size: 16
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 20.0
* mixed\_precision\_training: Native AMP


### Training results



### Framework versions


* Transformers 4.14.0.dev0
* Pytorch 1.10.0+cu102
* Datasets 1.16.1
* Tokenizers 0.10.3



### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-05
* train\_batch\_size: 16
* eval\_batch\_size: 16
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 20.0
* mixed\_precision\_training: Native AMP ### Training results ### Framework versions


* Transformers 4.14.0.dev0
* Pytorch 1.10.0+cu102
* Datasets 1.16.1
* Tokenizers 0.10.3
TAGS
#transformers #pytorch #bert #fill-mask #grc #autotrain_compatible #endpoints_compatible #region-us 
 ### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-05
* train\_batch\_size: 16
* eval\_batch\_size: 16
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 20.0
* mixed\_precision\_training: Native AMP ### Training results ### Framework versions


* Transformers 4.14.0.dev0
* Pytorch 1.10.0+cu102
* Datasets 1.16.1
* Tokenizers 0.10.3
passage: TAGS
#transformers #pytorch #bert #fill-mask #grc #autotrain_compatible #endpoints_compatible #region-us 
### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-05
* train\_batch\_size: 16
* eval\_batch\_size: 16
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 20.0
* mixed\_precision\_training: Native AMP### Training results### Framework versions


* Transformers 4.14.0.dev0
* Pytorch 1.10.0+cu102
* Datasets 1.16.1
* Tokenizers 0.10.3