Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,49 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- sr
|
| 5 |
+
metrics:
|
| 6 |
+
- accuracy
|
| 7 |
+
- wer
|
| 8 |
+
library_name: transformers
|
| 9 |
+
tags:
|
| 10 |
+
- legal
|
| 11 |
---
|
| 12 |
+
|
| 13 |
+
# BERTić-COMtext-SR-legal-MSD-ijekavica
|
| 14 |
+
|
| 15 |
+
**BERTić-COMtext-SR-legal-MSD-ijekavica** is a variant of the [BERTić](https://huggingface.co/classla/bcms-bertic) model, fine-tuned on the task of morphosyntactic (MSD) tag prediction in Serbian legal texts written in the Ijekavian pronunciation.
|
| 16 |
+
The model was fine-tuned for 15 epochs on the Ijekavian variant of the [COMtext.SR.legal](https://github.com/ICEF-NLP/COMtext.SR) dataset.
|
| 17 |
+
|
| 18 |
+
# Benchmarking
|
| 19 |
+
|
| 20 |
+
This model was evaluated on the tasks of MSD prediction and lemmatization of Serbian legal texts.
|
| 21 |
+
Lemmatization was performed using the predicted MSD tags and the [hrLex](http://hdl.handle.net/11356/1232) inflectional lexicon.
|
| 22 |
+
|
| 23 |
+
Accuracy and Word Error Rate were used as evaluation metrics.
|
| 24 |
+
|
| 25 |
+
This model was compared to:
|
| 26 |
+
- The [CLASSLA](http://pypi.org/project/classla/) library
|
| 27 |
+
- A variant of [BERTić](https://huggingface.co/classla/bcms-bertic) fine-tuned for MSD prediction using the [SETimes.SR 2.0](http://hdl.handle.net/11356/1843) corpus of newswire texts
|
| 28 |
+
- [SrBERTa](http://huggingface.co/nemanjaPetrovic/SrBERTa), a model specially trained on Serbian legal texts
|
| 29 |
+
|
| 30 |
+
All large language models were fine-tuned for 15 epochs.
|
| 31 |
+
CLASSLA and BERTić-SETimes were directly tested on the entire COMtext.SR.legal.ijekavica corpus.
|
| 32 |
+
BERTić-COMtext-SR-legal-MSD-ijekavica and SrBERTa were fine-tuned and evaluated on the COMtext.SR.legal.ijekavica corpus using 10-fold CV.
|
| 33 |
+
|
| 34 |
+
The code and data to run these experiments is available on the [COMtext.SR GitHub repository](https://github.com/ICEF-NLP/COMtext.SR).
|
| 35 |
+
|
| 36 |
+
## Results
|
| 37 |
+
|
| 38 |
+
| Model | MSD ACC | MSD WER | Lemma ACC | Lemma WER |
|
| 39 |
+
| ----------------------------------------------------------- | -------- | ---------- | --------- | ---------- |
|
| 40 |
+
| CLASSLA-SR (gold tokens) | 0.9150 | 0.0850 | 0.9036 | 0.0964 |
|
| 41 |
+
| *CLASSLA-SR (CLASSLA tokenizer)* | / | *0.0977* | / | *0.1135* |
|
| 42 |
+
| CLASSLA-HR (gold tokens) | 0.9062 | 0.0938 | 0.9353 | 0.0647 |
|
| 43 |
+
| *CLASSLA-HR (CLASSLA tokenizer)* | / | *0.1076* | / | *0.0827* |
|
| 44 |
+
| BERTić-SETimes.SR (gold tokens) | 0.9234 | 0.0766 | 0.9412 | 0.0588 |
|
| 45 |
+
| *BERTić-SETimes.SR (CLASSLA tokenizer)* | / | *0.0883* | / | *0.0780* |
|
| 46 |
+
| BERTić-COMtext-SR-legal-MSD-ijekavica (gold tokens) |**0.9674**| **0.0326** |**0.9429** | **0.0571** |
|
| 47 |
+
| *BERTić-COMtext-SR-legal-MSD-ijekavica (CLASSLA tokenizer)* | / |***0.0447***| / |***0.0763***|
|
| 48 |
+
| SrBERTa (gold tokens) | 0.9300 | 0.0700 | 0.9187 | 0.0813 |
|
| 49 |
+
|*SrBERTa (CLASSLA tokenizer)* | / | *0.0840* | / | *0.1024* |
|