|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- arbml/SANAD |
|
language: |
|
- ar |
|
base_model: |
|
- answerdotai/ModernBERT-base |
|
pipeline_tag: text-classification |
|
library_name: transformers |
|
tags: |
|
- modernbert |
|
- arabic |
|
--- |
|
|
|
|
|
# ModernBERT Arabic Model Card |
|
|
|
## Overview |
|
This is an Arabic version of ModernBERT, a modernized bidirectional encoder-only Transformer model (BERT-style). ModernBERT was pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. You can find more about the base ModernBERT model here: [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base). |
|
|
|
For this proof of concept, a tokenizer trained on Arabic Wikipedia was utilized: |
|
- **Dataset:** Arabic Wikipedia |
|
- **Size:** 1.8 GB |
|
- **Tokens:** 228,788,529 tokens |
|
|
|
This model demonstrates how ModernBERT can be adapted to Arabic for tasks like topic classification. |
|
|
|
## Model Details |
|
- **Epochs:** 3 |
|
- **Evaluation Metrics:** |
|
- **F1 Score:** 0.9587811491105839 |
|
- **Loss:** 0.19986020028591156 |
|
- **Runtime:** 46.4942 seconds |
|
- **Samples per second:** 305.006 |
|
- **Steps per second:** 38.134 |
|
- **Training Step:** 47,862 |
|
|
|
## How to Use |
|
The model can be used for text classification using the `transformers` library. Below is an example: |
|
|
|
```python |
|
from transformers import pipeline |
|
|
|
# Load model from huggingface.co/models using our repository ID |
|
classifier = pipeline( |
|
task="text-classification", |
|
model="ModernBERT-domain-classifier/checkpoint-47862", |
|
) |
|
|
|
sample = ''' |
|
اسلام عددا من الوافدين الى الممكلة العربية السعوديه |
|
''' |
|
|
|
classifier(sample) |
|
# [{'label': 'health', 'score': 0.6779336333274841}] |
|
|
|
|
|
|
