|
|
--- |
|
|
pipeline_tag: text-classification |
|
|
language: |
|
|
- multilingual |
|
|
license: apache-2.0 |
|
|
library_name: transformers |
|
|
--- |
|
|
|
|
|
# Model Description |
|
|
|
|
|
This model was build by translating the fine-Edu annotations into 15 languages using the best proprietary LLM for translation in the world: Tower LLM 70B. |
|
|
|
|
|
The translation model excels at translating entire documents and thus its the perfect fit to translate the texts we will use to train our classifier. |
|
|
|
|
|
The classifier is trained for English, German, Spanish, Japanese, Chinese, Russian, Hindi, Czech, Ukrainian, Icelandic, Portuguese, French, Dutch, Italian and Korean. Since its build on top of [mdeberta-v3-base](https://huggingface.co/microsoft/mdeberta-v3-base) it should be able to generalize across other languages. |
|
|
|
|
|
## Running Model: |
|
|
To run inference you must install |
|
|
``` |
|
|
pip install transformers[torch] |
|
|
pip install datasets |
|
|
pip install pandas |
|
|
pip install tqdm |
|
|
``` |
|
|
|
|
|
After installing those libraries you can sun the following code: |
|
|
|
|
|
```python |
|
|
import pandas as pd |
|
|
import torch |
|
|
from transformers import AutoModelForSequenceClassification, AutoTokenizer |
|
|
from tqdm import tqdm |
|
|
|
|
|
|
|
|
device = "cuda" |
|
|
path = "Unbabel/mfineweb-edu-classifier" |
|
|
model = AutoModelForSequenceClassification.from_pretrained( |
|
|
path, |
|
|
device_map=device, |
|
|
trust_remote_code=True, |
|
|
torch_dtype=torch.bfloat16 |
|
|
) |
|
|
tokenizer = AutoTokenizer.from_pretrained(path, use_fast=True) |
|
|
|
|
|
def get_model_outputs(texts): |
|
|
inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt", max_length=512).to(model.device) |
|
|
with torch.no_grad(): |
|
|
outputs = model(**inputs) |
|
|
score = outputs.logits |
|
|
prob = torch.nn.functional.sigmoid(outputs.binary_logits) |
|
|
return score.cpu(), prob.cpu() |
|
|
|
|
|
def batchify_texts(texts, batch_size): |
|
|
for i in range(0, len(texts), batch_size): |
|
|
yield texts[i:i + batch_size] |
|
|
|
|
|
# TODO: replace the next line with the texts you want to classify |
|
|
texts = LIST_WITH_TEXTS_TO_CLASSIFY |
|
|
batch_size = 64 # Adjust based on your available memory and model capacity |
|
|
num_batches = (len(texts) + batch_size - 1) // batch_size |
|
|
|
|
|
all_scores = [] |
|
|
all_probs = [] |
|
|
with tqdm(total=num_batches, dynamic_ncols=True) as pbar: |
|
|
for batch_num, batch in enumerate(batchify_texts(texts, batch_size), 1): |
|
|
score, probs = get_model_outputs(batch) |
|
|
all_scores.append(score) |
|
|
all_probs.append(probs) |
|
|
pbar.set_description(f"Processing Batch {batch_num}/{num_batches}") |
|
|
pbar.update(1) |
|
|
|
|
|
# SCORES is the output of the regression head and should reflect the |
|
|
# educational score of the text! |
|
|
scores = torch.cat(all_scores, dim=0).squeeze() |
|
|
|
|
|
## BINARY_PRED is the output of the classification head that tells |
|
|
# if a text has an acceptable educational score or not. |
|
|
# NOTE: Converting the scores into binary predictions is also possible |
|
|
all_probs = torch.cat(all_probs, dim=0).squeeze() |
|
|
binary_pred = (all_probs >= 0.5).numpy().astype(int) |
|
|
``` |
|
|
|
|
|
## English Results: |
|
|
|
|
|
When testing the model on an english partition with 37537 samples the results are comparable to the original FineEdu-classifier. |
|
|
|
|
|
Regression head results: |
|
|
``` |
|
|
precision recall f1-score support |
|
|
|
|
|
0 0.80 0.53 0.64 5130 |
|
|
1 0.80 0.88 0.83 21602 |
|
|
2 0.63 0.58 0.61 7849 |
|
|
3 0.54 0.62 0.58 2310 |
|
|
4 0.62 0.48 0.54 645 |
|
|
5 0.00 0.00 0.00 1 |
|
|
|
|
|
accuracy 0.74 37537 |
|
|
macro avg 0.56 0.51 0.53 37537 |
|
|
weighted avg 0.74 0.74 0.74 37537 |
|
|
``` |
|
|
|
|
|
Binary head results: |
|
|
``` |
|
|
precision recall f1-score support |
|
|
|
|
|
0 0.98 0.97 0.98 34581 |
|
|
1 0.71 0.74 0.73 2956 |
|
|
|
|
|
accuracy 0.96 37537 |
|
|
macro avg 0.85 0.86 0.85 37537 |
|
|
weighted avg 0.96 0.96 0.96 37537 |
|
|
``` |
|
|
|
|
|
## Multilingual Results: |
|
|
|
|
|
If we evaluate on the same texts translated into 15 different languages are almost identical! |
|
|
|
|
|
Regression head results: |
|
|
``` |
|
|
precision recall f1-score support |
|
|
|
|
|
0 0.80 0.50 0.61 5130 |
|
|
1 0.79 0.87 0.83 21602 |
|
|
2 0.61 0.58 0.59 7849 |
|
|
3 0.52 0.61 0.56 2310 |
|
|
4 0.61 0.38 0.47 645 |
|
|
5 0.00 0.00 0.00 1 |
|
|
|
|
|
accuracy 0.73 37537 |
|
|
macro avg 0.55 0.49 0.51 37537 |
|
|
weighted avg 0.73 0.73 0.73 37537 |
|
|
``` |
|
|
|
|
|
Binary head results: |
|
|
``` |
|
|
precision recall f1-score support |
|
|
|
|
|
0 0.98 0.97 0.97 34581 |
|
|
1 0.70 0.71 0.71 2956 |
|
|
|
|
|
accuracy 0.95 37537 |
|
|
macro avg 0.84 0.84 0.84 37537 |
|
|
weighted avg 0.95 0.95 0.95 37537 |
|
|
``` |
|
|
|
