File size: 3,366 Bytes
fda01b4 c46d81c b397691 c46d81c bb3fa96 2045782 bb3fa96 c46d81c df684c1 179503d 33c6c44 2750f3b c46d81c 2750f3b c46d81c 2750f3b c46d81c 8290a53 c46d81c b397691 c46d81c df684c1 8290a53 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 |
---
license: mit
language:
- multilingual
- en
- it
- sl
metrics:
- f1
- accuracy
base_model: FacebookAI/xlm-roberta-large
pipeline_tag: text-classification
tags:
- hate-speech
- xlm-roberta
- Youtube
- Twitter
---
# Multilingual Hate Speech Classifier for Social Media with Disagreement-Aware Training
A multilingual [XLM-R-based (100 languages)](https://huggingface.co/FacebookAI/xlm-roberta-large) hate speech classification model fine-tuned on English, Italian and Slovenian with inter-annotator disagreement-aware training.
The details of the model and the disagreement-aware training are described in our [paper](https://www.researchgate.net/publication/384628421_Multilingual_Hate_Speech_Modeling_by_Leveraging_Inter-Annotator_Disagreement):
@inproceedings{
grigor2024multilingual,
title={Multilingual Hate Speech Modeling by Leveraging Inter-Annotator Disagreement},
author={Grigor, Patricia-Carla and Evkoski, Bojan and Kralj Novak, Petra},
url={http://dx.doi.org/10.70314/is.2024.sikdd.7},
DOI={10.70314/is.2024.sikdd.7},
booktitle={Proceedings of Data Mining and Data Warehouses – Sikdd 2024},
publisher={Jožef Stefan Instutute},
year={2024}
}
Authors: Patricia-Carla Grigor, Bojan Evkoski, Petra Kralj Novak
Data available here: [English](https://www.clarin.si/repository/xmlui/handle/11356/1454); [Italian](https://www.clarin.si/repository/xmlui/handle/11356/1450); [Slovenian](https://www.clarin.si/repository/xmlui/handle/11356/1398)
**Model output**
The model classifies each input into one of four distinct classes:
* 0 - appropriate
* 1 - inappropriate
* 2 - offensive
* 3 - violent
**Training data***
* 51k English Youtube comments
* 60k Italian Youtube comments
* 50k Slovenian Twitter comments
**Evaluation data***
* 10k English Youtube comments
* 10k Italian Youtube comments
* 10k Slovenian Twitter comments
\* each comment is manually labeled by two different annotators
**Fine-tuning hyperparameters**
num_train_epochs=3,
train_batch_size=8,
learning_rate=6e-6
**Evaluation Results**
Model agreement (accuracy) vs. Inter-annotator agreement (0 - no agreement; 100 - perfect agreement):
| | Model-annotator Agreement | Inter-annotator Agreement |
|-----------|---------------------------|---------------------------|
| English | 79.97 | 82.91 |
| Italian | 82.00 | 81.79 |
| Slovenian | 78.84 | 79.43 |
Class-specific model F1-scores:
| | Appropriate | Inappropriate | Offensive | Violent |
|-----------|-------------|---------------|-----------|---------|
| English | 86.10 | 39.16 | 68.24 | 27.82 |
| Italian | 89.77 | 58.45 | 60.42 | 44.97 |
| Slovenian | 84.30 | 45.22 | 69.69 | 24.79 |
**Usage**
from transformers import AutoModelForSequenceClassification, TextClassificationPipeline, AutoTokenizer, AutoConfig
MODEL = "IMSyPP/hate_speech_multilingual"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
config = AutoConfig.from_pretrained(MODEL)
model = AutoModelForSequenceClassification.from_pretrained(MODEL)
pipe = TextClassificationPipeline(model=model, tokenizer=tokenizer, return_all_scores=True,
task='sentiment_analysis', device=0, function_to_apply="none")
pipe([
"Thank you for using our model",
"Grazie per aver utilizzato il nostro modello"
"Hvala za uporabo našega modela"
]) |