
# beto-emoji
Fine-tunning [BETO](https://github.com/dccuchile/beto) for emoji-prediction.

## Repository
Details with training and a use example are shown in [github.com/camilocarvajalreyes/beto-emoji](https://github.com/camilocarvajalreyes/beto-emoji). A deeper analysis of this and other models on the full dataset can be found in [github.com/furrutiav/data-mining-2022](https://github.com/furrutiav/data-mining-2022). We have used this model for a project for [CC5205 Data Mining](https://github.com/dccuchile/CC5205) course.

## Example
Inspired by model card from [cardiffnlp/twitter-roberta-base-emoji](https://huggingface.co/cardiffnlp/twitter-roberta-base-emoji).

```python
from transformers import AutoModelForSequenceClassification
from transformers import TFAutoModelForSequenceClassification
from transformers import AutoTokenizer
import numpy as np
from scipy.special import softmax
import csv
import urllib.request
# Preprocess text (username and link placeholders)
def preprocess(text):
    new_text = []
    for t in text.split(" "):
        t = '@user' if t.startswith('@') and len(t) > 1 else t
        t = 'http' if t.startswith('http') else t
        new_text.append(t)
    return " ".join(new_text)

MODEL = f"ccarvajal/beto-emoji"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
# download label mapping
labels=[]
mapping_link = f"https://raw.githubusercontent.com/camilocarvajalreyes/beto-emoji/main/es_mapping.txt"
with urllib.request.urlopen(mapping_link) as f:
    html = f.read().decode('utf-8').split("\n")
    csvreader = csv.reader(html, delimiter='\t')
labels = [row[1] for row in csvreader if len(row) > 1]

model = AutoModelForSequenceClassification.from_pretrained(MODEL)
model.save_pretrained(MODEL)
text =  "que viva españa"
text = preprocess(text)
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
scores = output[0][0].detach().numpy()
scores = softmax(scores)

ranking = np.argsort(scores)
ranking = ranking[::-1]
for i in range(scores.shape[0]):
    l = labels[ranking[i]]
    s = scores[ranking[i]]
    print(f"{i+1}) {l} {np.round(float(s), 4)}")
```
Output

```python
1) 🇪🇸 0.2508
2) 😍 0.238
3) 👌 0.2225
4) 😂 0.0806
5) ❤ 0.0489
6) 😁 0.0415
7) 😜 0.0232
8) 😎 0.0229
9) 😊 0.0156
10) 😉 0.0119
11) 💜 0.0079
12) 💕 0.0077
13) 💪 0.0066
14) 💘 0.0054
15) 💙 0.0052
16) 💞 0.005
17) 😘 0.0034
18) 🎶 0.0022
19) ✨ 0.0007
```

## Results in test set
                 precision    recall  f1-score   support

           ❤       0.39      0.43      0.41      2141
           😍       0.29      0.39      0.33      1408
           😂       0.51      0.51      0.51      1499
           💕       0.09      0.05      0.06       352
           😊       0.12      0.23      0.16       514
           😘       0.24      0.23      0.24       397
           💪       0.37      0.43      0.40       307
           😉       0.15      0.17      0.16       453
           👌       0.09      0.16      0.11       180
           🇪🇸       0.46      0.46      0.46       424
           😎       0.12      0.11      0.11       339
           💙       0.36      0.02      0.04       413
           💜       0.00      0.00      0.00       235
           😜       0.04      0.02      0.02       274
           💞       0.00      0.00      0.00        93
           ✨       0.26      0.12      0.17       416
           🎶       0.25      0.24      0.24       212
           💘       0.00      0.00      0.00       134
           😁       0.05      0.03      0.04       209

     accuracy                           0.30     10000
    macro_avg       0.20      0.19      0.18     10000
    weighted avg    0.29      0.30      0.29     10000

[Another example](https://github.com/camilocarvajalreyes/beto-emoji/blob/main/attention_visualisation.ipynb) with a visualisation of the attention modules within this model is carried out using [bertviz](https://github.com/jessevig/bertviz).

## Reproducibility
The Multilingual Emoji Prediction dataset (Barbieri et al. 2010) consists of tweets in English and Spanish that originally had a single emoji, which is later used as a tag. Test and trial sets can be downloaded [here](https://github.com/fvancesco/Semeval2018-Task2-Emoji-Detection/blob/master/dataset/Semeval2018-Task2-EmojiPrediction.zip?raw=true), but the train set needs to be downloaded using a [twitter crawler](https://github.com/fra82/twitter-crawler/blob/master/semeval2018task2TwitterCrawlerHOWTO.md). The goal is to predict that single emoji that was originally in the tweet using the text in it (out of a fixed set of possible emojis, 20 for English and 19 for Spanish).

Training parameters:
```python
training_args = TrainingArguments(
    output_dir="./results",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=5,
    weight_decay=0.01
)
```

{"language": ["es"]}
transformers pytorch bert text-classification es autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #bert #text-classification #es #autotrain_compatible #endpoints_compatible #region-us 


# beto-emoji
Fine-tunning BETO for emoji-prediction.

## Repository
Details with training and a use example are shown in URL A deeper analysis of this and other models on the full dataset can be found in URL We have used this model for a project for CC5205 Data Mining course.

## Example
Inspired by model card from cardiffnlp/twitter-roberta-base-emoji.


Output



## Results in test set
                 precision    recall  f1-score   support

                  0.39      0.43      0.41      2141
                  0.29      0.39      0.33      1408
                  0.51      0.51      0.51      1499
                  0.09      0.05      0.06       352
                  0.12      0.23      0.16       514
                  0.24      0.23      0.24       397
                  0.37      0.43      0.40       307
                  0.15      0.17      0.16       453
                  0.09      0.16      0.11       180
           🇪🇸       0.46      0.46      0.46       424
                  0.12      0.11      0.11       339
                  0.36      0.02      0.04       413
                  0.00      0.00      0.00       235
                  0.04      0.02      0.02       274
                  0.00      0.00      0.00        93
                  0.26      0.12      0.17       416
                  0.25      0.24      0.24       212
                  0.00      0.00      0.00       134
                  0.05      0.03      0.04       209

     accuracy                           0.30     10000
    macro_avg       0.20      0.19      0.18     10000
    weighted avg    0.29      0.30      0.29     10000

Another example with a visualisation of the attention modules within this model is carried out using bertviz.

## Reproducibility
The Multilingual Emoji Prediction dataset (Barbieri et al. 2010) consists of tweets in English and Spanish that originally had a single emoji, which is later used as a tag. Test and trial sets can be downloaded here, but the train set needs to be downloaded using a twitter crawler. The goal is to predict that single emoji that was originally in the tweet using the text in it (out of a fixed set of possible emojis, 20 for English and 19 for Spanish).

Training parameters:


# beto-emoji
Fine-tunning BETO for emoji-prediction. ## Repository
Details with training and a use example are shown in URL A deeper analysis of this and other models on the full dataset can be found in URL We have used this model for a project for CC5205 Data Mining course. ## Example
Inspired by model card from cardiffnlp/twitter-roberta-base-emoji.


Output ## Results in test set
                 precision    recall  f1-score   support

                  0.39      0.43      0.41      2141
                  0.29      0.39      0.33      1408
                  0.51      0.51      0.51      1499
                  0.09      0.05      0.06       352
                  0.12      0.23      0.16       514
                  0.24      0.23      0.24       397
                  0.37      0.43      0.40       307
                  0.15      0.17      0.16       453
                  0.09      0.16      0.11       180
           🇪🇸       0.46      0.46      0.46       424
                  0.12      0.11      0.11       339
                  0.36      0.02      0.04       413
                  0.00      0.00      0.00       235
                  0.04      0.02      0.02       274
                  0.00      0.00      0.00        93
                  0.26      0.12      0.17       416
                  0.25      0.24      0.24       212
                  0.00      0.00      0.00       134
                  0.05      0.03      0.04       209

     accuracy                           0.30     10000
    macro_avg       0.20      0.19      0.18     10000
    weighted avg    0.29      0.30      0.29     10000

Another example with a visualisation of the attention modules within this model is carried out using bertviz. ## Reproducibility
The Multilingual Emoji Prediction dataset (Barbieri et al. 2010) consists of tweets in English and Spanish that originally had a single emoji, which is later used as a tag. Test and trial sets can be downloaded here, but the train set needs to be downloaded using a twitter crawler. The goal is to predict that single emoji that was originally in the tweet using the text in it (out of a fixed set of possible emojis, 20 for English and 19 for Spanish).

Training parameters:
TAGS
#transformers #pytorch #bert #text-classification #es #autotrain_compatible #endpoints_compatible #region-us 
 # beto-emoji
Fine-tunning BETO for emoji-prediction. ## Repository
Details with training and a use example are shown in URL A deeper analysis of this and other models on the full dataset can be found in URL We have used this model for a project for CC5205 Data Mining course. ## Example
Inspired by model card from cardiffnlp/twitter-roberta-base-emoji.


Output ## Results in test set
                 precision    recall  f1-score   support

                  0.39      0.43      0.41      2141
                  0.29      0.39      0.33      1408
                  0.51      0.51      0.51      1499
                  0.09      0.05      0.06       352
                  0.12      0.23      0.16       514
                  0.24      0.23      0.24       397
                  0.37      0.43      0.40       307
                  0.15      0.17      0.16       453
                  0.09      0.16      0.11       180
           🇪🇸       0.46      0.46      0.46       424
                  0.12      0.11      0.11       339
                  0.36      0.02      0.04       413
                  0.00      0.00      0.00       235
                  0.04      0.02      0.02       274
                  0.00      0.00      0.00        93
                  0.26      0.12      0.17       416
                  0.25      0.24      0.24       212
                  0.00      0.00      0.00       134
                  0.05      0.03      0.04       209

     accuracy                           0.30     10000
    macro_avg       0.20      0.19      0.18     10000
    weighted avg    0.29      0.30      0.29     10000

Another example with a visualisation of the attention modules within this model is carried out using bertviz. ## Reproducibility
The Multilingual Emoji Prediction dataset (Barbieri et al. 2010) consists of tweets in English and Spanish that originally had a single emoji, which is later used as a tag. Test and trial sets can be downloaded here, but the train set needs to be downloaded using a twitter crawler. The goal is to predict that single emoji that was originally in the tweet using the text in it (out of a fixed set of possible emojis, 20 for English and 19 for Spanish).

Training parameters:
passage: TAGS
#transformers #pytorch #bert #text-classification #es #autotrain_compatible #endpoints_compatible #region-us 
# beto-emoji
Fine-tunning BETO for emoji-prediction.## Repository
Details with training and a use example are shown in URL A deeper analysis of this and other models on the full dataset can be found in URL We have used this model for a project for CC5205 Data Mining course.## Example
Inspired by model card from cardiffnlp/twitter-roberta-base-emoji.


Output## Results in test set
                 precision    recall  f1-score   support

                  0.39      0.43      0.41      2141
                  0.29      0.39      0.33      1408
                  0.51      0.51      0.51      1499
                  0.09      0.05      0.06       352
                  0.12      0.23      0.16       514
                  0.24      0.23      0.24       397
                  0.37      0.43      0.40       307
                  0.15      0.17      0.16       453
                  0.09      0.16      0.11       180
           🇪🇸       0.46      0.46      0.46       424
                  0.12      0.11      0.11       339
                  0.36      0.02      0.04       413
                  0.00      0.00      0.00       235
                  0.04      0.02      0.02       274
                  0.00      0.00      0.00        93
                  0.26      0.12      0.17       416
                  0.25      0.24      0.24       212
                  0.00      0.00      0.00       134
                  0.05      0.03      0.04       209

     accuracy                           0.30     10000
    macro_avg       0.20      0.19      0.18     10000
    weighted avg    0.29      0.30      0.29     10000

Another example with a visualisation of the attention modules within this model is carried out using bertviz.## Reproducibility
The Multilingual Emoji Prediction dataset (Barbieri et al. 2010) consists of tweets in English and Spanish that originally had a single emoji, which is later used as a tag. Test and trial sets can be downloaded here, but the train set needs to be downloaded using a twitter crawler. The goal is to predict that single emoji that was originally in the tweet using the text in it (out of a fixed set of possible emojis, 20 for English and 19 for Spanish).

Training parameters: