

# Arabic-ALBERT Xlarge

Arabic edition of ALBERT Xlarge pretrained language model

_If you use any of these models in your work, please cite this work as:_

```
@software{ali_safaya_2020_4718724,
  author       = {Ali Safaya},
  title        = {Arabic-ALBERT},
  month        = aug,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.4718724},
  url          = {https://doi.org/10.5281/zenodo.4718724}
}
```

## Pretraining data

The models were pretrained on ~4.4 Billion words:

- Arabic version of [OSCAR](https://oscar-corpus.com/) (unshuffled version of the corpus) - filtered from [Common Crawl](http://commoncrawl.org/)
- Recent dump of Arabic [Wikipedia](https://dumps.wikimedia.org/backup-index.html)

__Notes on training data:__

- Our final version of corpus contains some non-Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER.
- Although non-Arabic characters were lowered as a preprocessing step, since Arabic characters do not have upper or lower case, there is no cased and uncased version of the model.
- The corpus and vocabulary set are not restricted to Modern Standard Arabic, they contain some dialectical Arabic too.

## Pretraining details

- These models were trained using Google ALBERT's github [repository](https://github.com/google-research/albert) on a single TPU v3-8 provided for free from [TFRC](https://www.tensorflow.org/tfrc).
- Our pretraining procedure follows training settings of bert with some changes: trained for 7M training steps with batchsize of 64, instead of 125K with batchsize of 4096.

## Models

|  | albert-base | albert-large | albert-xlarge |
|:---:|:---:|:---:|:---:|
| Hidden Layers | 12 | 24 | 24 |
| Attention heads | 12 | 16 | 32 |
| Hidden size | 768 | 1024 | 2048 |

## Results

For further details on the models performance or any other queries, please refer to [Arabic-ALBERT](https://github.com/KUIS-AI-Lab/Arabic-ALBERT/)

## How to use

You can use these models by installing `torch` or `tensorflow` and Huggingface library `transformers`. And you can use it directly by initializing it like this:  

```python

from transformers import AutoTokenizer, AutoModel

# loading the tokenizer
tokenizer    = AutoTokenizer.from_pretrained("kuisailab/albert-xlarge-arabic")

# loading the model
model   = AutoModelForMaskedLM.from_pretrained("kuisailab/albert-xlarge-arabic")

```

## Acknowledgement

Thanks to Google for providing free TPU for the training process and for Huggingface for hosting these models on their servers 😊

{"language": "ar", "tags": ["ar", "masked-lm"], "datasets": ["oscar", "wikipedia"]}
transformers pytorch tf safetensors albert fill-mask ar masked-lm dataset:oscar dataset:wikipedia autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #tf #safetensors #albert #fill-mask #ar #masked-lm #dataset-oscar #dataset-wikipedia #autotrain_compatible #endpoints_compatible #region-us 

Arabic-ALBERT Xlarge
====================


Arabic edition of ALBERT Xlarge pretrained language model


*If you use any of these models in your work, please cite this work as:*


Pretraining data
----------------


The models were pretrained on ~4.4 Billion words:


* Arabic version of OSCAR (unshuffled version of the corpus) - filtered from Common Crawl
* Recent dump of Arabic Wikipedia


**Notes on training data:**


* Our final version of corpus contains some non-Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER.
* Although non-Arabic characters were lowered as a preprocessing step, since Arabic characters do not have upper or lower case, there is no cased and uncased version of the model.
* The corpus and vocabulary set are not restricted to Modern Standard Arabic, they contain some dialectical Arabic too.


Pretraining details
-------------------


* These models were trained using Google ALBERT's github repository on a single TPU v3-8 provided for free from TFRC.
* Our pretraining procedure follows training settings of bert with some changes: trained for 7M training steps with batchsize of 64, instead of 125K with batchsize of 4096.


Models
------



Results
-------


For further details on the models performance or any other queries, please refer to Arabic-ALBERT


How to use
----------


You can use these models by installing 'torch' or 'tensorflow' and Huggingface library 'transformers'. And you can use it directly by initializing it like this:


Acknowledgement
---------------


Thanks to Google for providing free TPU for the training process and for Huggingface for hosting these models on their servers



TAGS
#transformers #pytorch #tf #safetensors #albert #fill-mask #ar #masked-lm #dataset-oscar #dataset-wikipedia #autotrain_compatible #endpoints_compatible #region-us 

TAGS
#transformers #pytorch #tf #safetensors #albert #fill-mask #ar #masked-lm #dataset-oscar #dataset-wikipedia #autotrain_compatible #endpoints_compatible #region-us 
