|
--- |
|
license: mit |
|
language: |
|
- multilingual |
|
- en |
|
- it |
|
- sl |
|
metrics: |
|
- f1 |
|
- accuracy |
|
base_model: FacebookAI/xlm-roberta-large |
|
pipeline_tag: text-classification |
|
tags: |
|
- hate-speech |
|
- xlm-roberta |
|
- Youtube |
|
- Twitter |
|
--- |
|
|
|
# Multilingual Hate Speech Classifier for Social Media with Disagreement-Aware Training |
|
|
|
A multilingual [XLM-R-based (100 languages)](https://huggingface.co/FacebookAI/xlm-roberta-large) hate speech classification model fine-tuned on English, Italian and Slovenian with inter-annotator disagreement-aware training. |
|
|
|
The details of the model and the disagreement-aware training are described in our [paper](https://www.researchgate.net/publication/384628421_Multilingual_Hate_Speech_Modeling_by_Leveraging_Inter-Annotator_Disagreement): |
|
|
|
@inproceedings{ |
|
grigor2024multilingual, |
|
title={Multilingual Hate Speech Modeling by Leveraging Inter-Annotator Disagreement}, |
|
author={Grigor, Patricia-Carla and Evkoski, Bojan and Kralj Novak, Petra}, |
|
url={http://dx.doi.org/10.70314/is.2024.sikdd.7}, |
|
DOI={10.70314/is.2024.sikdd.7}, |
|
booktitle={Proceedings of Data Mining and Data Warehouses – Sikdd 2024}, |
|
publisher={Jožef Stefan Instutute}, |
|
year={2024} |
|
} |
|
|
|
Authors: Patricia-Carla Grigor, Bojan Evkoski, Petra Kralj Novak |
|
|
|
Data available here: [English](https://www.clarin.si/repository/xmlui/handle/11356/1454); [Italian](https://www.clarin.si/repository/xmlui/handle/11356/1450); [Slovenian](https://www.clarin.si/repository/xmlui/handle/11356/1398) |
|
|
|
**Model output** |
|
The model classifies each input into one of four distinct classes: |
|
|
|
* 0 - appropriate |
|
* 1 - inappropriate |
|
* 2 - offensive |
|
* 3 - violent |
|
|
|
**Training data*** |
|
* 51k English Youtube comments |
|
* 60k Italian Youtube comments |
|
* 50k Slovenian Twitter comments |
|
|
|
**Evaluation data*** |
|
* 10k English Youtube comments |
|
* 10k Italian Youtube comments |
|
* 10k Slovenian Twitter comments |
|
|
|
\* each comment is manually labeled by two different annotators |
|
|
|
**Fine-tuning hyperparameters** |
|
|
|
num_train_epochs=3, |
|
train_batch_size=8, |
|
learning_rate=6e-6 |
|
|
|
**Evaluation Results** |
|
Model agreement (accuracy) vs. Inter-annotator agreement (0 - no agreement; 100 - perfect agreement): |
|
| | Model-annotator Agreement | Inter-annotator Agreement | |
|
|-----------|---------------------------|---------------------------| |
|
| English | 79.97 | 82.91 | |
|
| Italian | 82.00 | 81.79 | |
|
| Slovenian | 78.84 | 79.43 | |
|
|
|
Class-specific model F1-scores: |
|
| | Appropriate | Inappropriate | Offensive | Violent | |
|
|-----------|-------------|---------------|-----------|---------| |
|
| English | 86.10 | 39.16 | 68.24 | 27.82 | |
|
| Italian | 89.77 | 58.45 | 60.42 | 44.97 | |
|
| Slovenian | 84.30 | 45.22 | 69.69 | 24.79 | |
|
|
|
**Usage** |
|
|
|
from transformers import AutoModelForSequenceClassification, TextClassificationPipeline, AutoTokenizer, AutoConfig |
|
|
|
MODEL = "IMSyPP/hate_speech_multilingual" |
|
tokenizer = AutoTokenizer.from_pretrained(MODEL) |
|
config = AutoConfig.from_pretrained(MODEL) |
|
model = AutoModelForSequenceClassification.from_pretrained(MODEL) |
|
|
|
pipe = TextClassificationPipeline(model=model, tokenizer=tokenizer, return_all_scores=True, |
|
task='sentiment_analysis', device=0, function_to_apply="none") |
|
pipe([ |
|
"Thank you for using our model", |
|
"Grazie per aver utilizzato il nostro modello" |
|
"Hvala za uporabo našega modela" |
|
]) |
