pytorch encoder-decoder text2text-generation es dataset:mlsum arxiv:1907.12461 transformers summarization seq2seq model-index autotrain_compatible
---
language:
- es
tags:
- summarization  # Example: audio
- seq2seq  # Example: automatic-speech-recognition
datasets:
- mlsum
metrics:
- rouge2
- rouge1
- rougel
- rougelsum

# Optional. Add this if you want to encode your eval results in a structured way.
model-index:
- name: marimari-r2r-mlsum
  results:
  - task: 
      type: summarization  # Required. Example: automatic-speech-recognition
      name: abstractive summarization  # Optional. Example: Speech Recognition
    dataset:
      type: mlsum  # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
      name: mlsum-es  # Required. Example: Common Voice zh-CN
      args: es         # Optional. Example: zh-CN
    metrics:
      - type: rouge1    # Required. Example: wer
        value: 28.7802  # Required. Example: 20.90
        name: rouge1    # Optional. Example: Test WER
      - type: rouge2
        value: 10.6748
        name: rouge2
      - type: rougeL
        value: 23.0447
        name: rougeL
      - type: rougeLsum
        value: 23.4055
        name: rougeLsum
---

<img src="https://huggingface.co/IIC/marimari-r2r-mlsum/resolve/main/marimariLogo.png"/>

This is a model for text summarization in Spanish. It has been trained on the spanish portion of [mlsum](https://huggingface.co/datasets/mlsum). For that, MariMari was created. It is called like that because it is an EncoderDecoder model built from Maria model, specifically, the [roberta model from the Maria Project](https://huggingface.co/PlanTL-GOB-ES/roberta-base-bne). For building the Encoder Decoder model, [this paper was followed](https://arxiv.org/abs/1907.12461), which has a [direct implementation in transformers](https://huggingface.co/docs/transformers/master/model_doc/encoder-decoder). As there are no natural encoder decoder models in Spanish, such as BART or T5, we decided to leverage the capacity of the Roberta model of the MarIA project, as it has shown great results on several NLU tasks, therefore it was natural to think it could perform well on NLG tasks when trained properly.

For tuning the hyperparameters  of the model we used [Optuna](https://optuna.org/), with only 10 different trials and 7 initial random trials, as [the dataset chosen for training the model, mlsum](https://huggingface.co/datasets/mlsum) was huge. The set of hyperparameters used was the following:

```python

    def hp_space(trial):
        return {
            "learning_rate": trial.suggest_float(
                "learning_rate", 3e-5, 7e-5, log=True
            ),
            "num_train_epochs": trial.suggest_categorical(
                "num_train_epochs", [7]
            ),
            "per_device_train_batch_size": trial.suggest_categorical(
                "per_device_train_batch_size", [16]),
            "per_device_eval_batch_size": trial.suggest_categorical(
                "per_device_eval_batch_size", [32]),
            "gradient_accumulation_steps": trial.suggest_categorical(
                "gradient_accumulation_steps", [2, 4, 8]),
            "warmup_steps": trial.suggest_categorical(
                "warmup_steps", [50, 100, 500, 1000]
            ),
            "weight_decay": trial.suggest_float(
                 "weight_decay", 0.0, 0.1
            ),
```

The reported results are on the test split of mlsum. As you can see, MariMari-r2r-mlsum works better for summarization on mlsum than the previous best model in this regard, [beto2beto](https://huggingface.co/LeoCordoba/beto2beto-mlsum). The complete metrics on test are:

```json
{"rouge1": 28.7802, "rouge2": 10.6748, "rougeL": 23.0447, "rougeLsum": 23.4055, "gen_len": 25.7803}
```

This model is really easy to use, and with the following lines of code you can just start summarizing your documents in Spanish:

```python
from transformers import EncoderDecoderModel, AutoTokenizer

text = "Hola esto es un ejemplo de texto a resumir. Poco hay que resumir aquí, pero es sólo de muestra."

tokenizer = AutoTokenizer.from_pretrained("IIC/marimari-r2r-mlsum")
model = EncoderDecoderModel.from_pretrained("IIC/marimari-r2r-mlsum")

input_ids = tokenizer(text, return_tensors="pt").input_ids

output_ids = model.generate(input_ids)[0]
print(tokenizer.decode(output_ids, skip_special_tokens=True))
```

### Contributions
Thanks to [@avacaondata](https://huggingface.co/avacaondata), [@alborotis](https://huggingface.co/alborotis), [@albarji](https://huggingface.co/albarji), [@Dabs](https://huggingface.co/Dabs), [@GuillemGSubies](https://huggingface.co/GuillemGSubies) for adding this model.