pytorch bert fill-mask arxiv:1911.03090 arxiv:2010.02559 transformers autotrain_compatible
# Legal-HeBERT
Legal-HeBERT is a BERT model for Hebrew legal and legislative domains. It is intended to improve the legal NLP research and tools development in Hebrew. We release two versions of Legal-HeBERT. The first version is a fine-tuned model of [HeBERT](https://github.com/avichaychriqui/HeBERT) applied on legal and legislative documents. The second version uses [HeBERT](https://github.com/avichaychriqui/HeBERT)'s architecture guidlines to train a BERT model from scratch. <br>
We continue collecting legal data, examining different architectural designs, and performing tagged datasets and legal tasks for evaluating and to development of a Hebrew legal tools.

## Training Data
Our training datasets are:

| Name                                                                                                                              	| Hebrew Description                                                       	| Size (GB) 	| Documents 	| Sentences  	| Words       	| Notes                                   	|
|-----------------------------------------------------------------------------------------------------------------------------------	|--------------------------------------------------------------------------	|-----------	|-----------	|------------	|-------------	|-----------------------------------------	|
| The Israeli Law Book                                                                                                              	| ספר החוקים הישראלי                                                       	| 0.05      	| 2338      	| 293352     	| 4851063     	|                                         	|
| Judgments of the Supreme Court                                                                                                    	| מאגר פסקי   הדין של בית המשפט העליון                                     	| 0.7       	| 212348    	| 5790138    	| 79672415    	|                                         	|
| custody courts                                                                                                                    	| החלטות בתי   הדין למשמורת                                                	| 2.46      	| 169,708   	| 8,555,893  	| 213,050,492 	|                                         	|
| Law memoranda, drafts of secondary legislation and drafts of   support tests that have been distributed to the public for comment 	| תזכירי חוק,   טיוטות חקיקת משנה וטיוטות מבחני תמיכה שהופצו להערות הציבור 	| 0.4       	| 3,291     	| 294,752    	| 7,218,960   	|                                         	|
| Supervisors of Land Registration judgments                                                                                        	| מאגר פסקי   דין של המפקחים על רישום המקרקעין                             	| 0.02      	| 559       	| 67,639     	| 1,785,446   	|                                         	|
| Decisions of the Labor Court - Corona                                                                                             	|          מאגר החלטות   בית הדין לעניין שירות התעסוקה – קורונה            	| 0.001     	| 146       	| 3505       	| 60195       	|                                         	|
| Decisions of the Israel Lands Council                                                                                             	| החלטות   מועצת מקרקעי ישראל                                              	|           	| 118       	| 11283      	| 162692      	| aggregate file                          	|
| Judgments of the Disciplinary Tribunal and the Israel Police Appeals Tribunal                                                     	| פסקי  דין של בית הדין למשמעת ובית הדין לערעורים של משטרת ישראל           	| 0.02      	| 54        	| 83724      	| 1743419     	| aggregate files                         	|
| Disciplinary Appeals Committee in the   Ministry of Health                                                                        	| ועדת   ערר לדין משמעתי במשרד הבריאות                                     	| 0.004     	| 252       	| 21010      	| 429807      	| 465 files are scanned and didn't parser 	|
| Attorney General's Positions                                                                                                      	| מאגר התייצבויות היועץ המשפטי לממשלה                                      	| 0.008     	| 281       	| 32724      	| 813877      	|                                         	|
| Legal-Opinion of the Attorney General                                                                                             	| מאגר חוות דעת היועץ המשפטי לממשלה                                        	| 0.002     	| 44        	| 7132       	| 188053      	|                                         	|
|                                                                                                                                   	|                                                                          	|           	|           	|            	|             	|                                         	|
| total                                                                                                                             	|                                                                          	| 3.665     	| 389,139   	| 15,161,152 	| 309,976,419 	|                                         	|

We thank <b>Yair Gardin</b> for the referring to the governance data, <b>Elhanan Schwarts</b> for collecting and parsing The Israeli law book, and <b>Jonathan Schler</b> for collecting the judgments of the supreme court.


## Training process
* Vocabulary size: 50,000 tokens
* 4 epochs (1M steps±)
* lr=5e-5
* mlm_probability=0.15
* batch size = 32 (for each gpu)
* NVIDIA GeForce RTX 2080 TI + NVIDIA GeForce RTX 3090 (1 week training)

### Additional training settings: 
<b>Fine-tuned [HeBERT](https://github.com/avichaychriqui/HeBERT) model:</b> The first eight layers were freezed (like [Lee et al. (2019)](https://arxiv.org/abs/1911.03090) suggest)<br>
<b>Legal-HeBERT trained from scratch:</b> The training process is similar to [HeBERT](https://github.com/avichaychriqui/HeBERT) and inspired by [Chalkidis et al. (2020)](https://arxiv.org/abs/2010.02559) <br>

## How to use
The models can be found in huggingface hub and can be fine-tunned to any down-stream task:
```
# !pip install transformers==4.14.1
from transformers import AutoTokenizer, AutoModel

model_name = 'avichr/Legal-heBERT_ft' # for the fine-tuned HeBERT model 
model_name = 'avichr/Legal-heBERT' # for legal HeBERT model trained from scratch

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

from transformers import pipeline
fill_mask = pipeline(
    "fill-mask",
    model=model_name,
)
fill_mask("הקורונה לקחה את [MASK] ולנו לא נשאר דבר.")
```
## Stay tuned!
We are still working on our models and the datasets. We will edit this page as we progress. We are open for collaborations.

## If you used this model please cite us as :
Chriqui, Avihay, Yahav, Inbal and Bar-Siman-Tov, Ittai, Legal HeBERT: A BERT-based NLP Model for Hebrew Legal, Judicial and Legislative Texts (June 27, 2022). Available at: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4147127

```
@article{chriqui2021hebert,
  title={Legal HeBERT: A BERT-based NLP Model for Hebrew Legal, Judicial and Legislative Texts},
  author={Chriqui, Avihay, Yahav, Inbal and Bar-Siman-Tov, Ittai},
  journal={SSRN preprint:4147127},
  year={2022}
}
```

## Contact us
[Avichay Chriqui](mailto:avichayc@mail.tau.ac.il), The Coller AI Lab <br>
[Inbal yahav](mailto:inbalyahav@tauex.tau.ac.il), The Coller AI Lab <br>
[Ittai Bar-Siman-Tov](mailto:Ittai.Bar-Siman-Tov@biu.ac.il), the BIU Innovation Lab for Law, Data-Science and Digital Ethics <br>

Thank you, תודה, شكرا <br>
