
# Arabic BERT Model
**AraBERTMo** is an Arabic pre-trained language model based on [Google's BERT architechture](https://github.com/google-research/bert). 
AraBERTMo_base uses the same BERT-Base config. 
AraBERTMo_base now comes in 10 new variants
All models are available on the `HuggingFace` model page under the [Ebtihal](https://huggingface.co/Ebtihal/) name. 
Checkpoints are available in PyTorch formats.

## Pretraining Corpus
`AraBertMo_base_V5' model was pre-trained on ~3 million words:
- [OSCAR](https://traces1.inria.fr/oscar/) - Arabic version "unshuffled_deduplicated_ar". 

## Training results
this model achieves the following results:

| Task | Num examples | Num Epochs  | Batch Size | steps | Wall time  | training loss| 
|:----:|:----:|:----:|:----:|:-----:|:----:|:-----:|
| Fill-Mask| 50046|  5  | 64 | 3910  | 6h 49m 59s | 7.4599  | 

## Load Pretrained Model
You can use this model by installing `torch` or `tensorflow` and Huggingface library `transformers`. And you can use it directly by initializing it like this:  
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("Ebtihal/AraBertMo_base_V5")
model = AutoModelForMaskedLM.from_pretrained("Ebtihal/AraBertMo_base_V5")
```

 ## This model was built for master's degree research in an organization:
- [University of kufa](https://uokufa.edu.iq/).
- [Faculty of Computer Science and Mathematics](https://mathcomp.uokufa.edu.iq/).
- **Department of Computer Science**


{"language": "ar", "tags": "Fill-Mask", "datasets": "OSCAR", "widget": [{"text": " \u0627\u0644\u0633\u0644\u0627\u0645 \u0639\u0644\u064a\u0643\u0645 \u0648\u0631\u062d\u0645\u0629[MASK] \u0648\u0628\u0631\u0643\u0627\u062a\u0629"}, {"text": " \u0627\u0647\u0644\u0627 \u0648\u0633\u0647\u0644\u0627 \u0628\u0643\u0645 \u0641\u064a [MASK] \u0645\u0646 \u0633\u064a\u0631\u0628\u062d \u0627\u0644\u0645\u0644\u064a\u0648\u0646"}, {"text": " \u0645\u0631\u062d\u0628\u0627 \u0628\u0643 \u0639\u0632\u064a\u0632\u064a \u0627\u0644\u0632\u0627\u0626\u0631 [MASK] \u0645\u0648\u0642\u0639\u0646\u0627 "}]}
transformers pytorch bert fill-mask Fill-Mask ar dataset:OSCAR autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #bert #fill-mask #Fill-Mask #ar #dataset-OSCAR #autotrain_compatible #endpoints_compatible #region-us 

Arabic BERT Model
=================


AraBERTMo is an Arabic pre-trained language model based on Google's BERT architechture.
AraBERTMo\_base uses the same BERT-Base config.
AraBERTMo\_base now comes in 10 new variants
All models are available on the 'HuggingFace' model page under the Ebtihal name.
Checkpoints are available in PyTorch formats.


Pretraining Corpus
------------------


'AraBertMo\_base\_V5' model was pre-trained on ~3 million words:


* OSCAR - Arabic version "unshuffled\_deduplicated\_ar".


Training results
----------------


this model achieves the following results:



Load Pretrained Model
---------------------


You can use this model by installing 'torch' or 'tensorflow' and Huggingface library 'transformers'. And you can use it directly by initializing it like this:


This model was built for master's degree research in an organization:
---------------------------------------------------------------------


* University of kufa.
* Faculty of Computer Science and Mathematics.
* Department of Computer Science



TAGS
#transformers #pytorch #bert #fill-mask #Fill-Mask #ar #dataset-OSCAR #autotrain_compatible #endpoints_compatible #region-us 

passage: TAGS
#transformers #pytorch #bert #fill-mask #Fill-Mask #ar #dataset-OSCAR #autotrain_compatible #endpoints_compatible #region-us 
