metadata

language:
  - it
tags:
  - summarization
datasets:
  - ARTeLab/mlsum-it
metrics:
  - rouge
base_model: gsarti/it5-base
model-index:
  - name: summarization_mlsum
    results:
      - task:
          type: summarization
          name: Summarization
        dataset:
          name: cnn_dailymail
          type: cnn_dailymail
          config: 3.0.0
          split: test
        metrics:
          - type: rouge
            value: 10.775
            name: ROUGE-1
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2MxZWFhOTMxYTBmMjVhZmE0MzU5NDQzMGVlYWU5MjJmYjBiMGI5Y2U1ZmMwZDQzYWUwNDEwNDI1ZjY2ODA5MyIsInZlcnNpb24iOjF9.EqOHhgehPD-OVAK2vCdFndZlhiyh3-Vc89D_ujisSgK-shrouep7JhKV4hYtp-m5PbEvAQSk8PWJsYBlwV00Bw
          - type: rouge
            value: 3.0633
            name: ROUGE-2
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYjNiZGNlNjEyOTI4YjQ2OWIzNjczNjU5ZWUyOTk3MTNhMDY0ZmI5ZDcyOGEwOTVjNjFmYjJiMjBhZDBiY2Y1YiIsInZlcnNpb24iOjF9.Hznn3spvnWWUpR4KVQ20UP-rM2MFDtRCCjtaiUxRRvC_46KpsPoyKme2h_X3QFW-xKPMLj4BLaJOLRPTrXO0Dw
          - type: rouge
            value: 9.2018
            name: ROUGE-L
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzVhOGUwYjYxYzQwYTVhZDUzYzIwMjc1ZWE5ZjQ3ZWRkNzlmZGY1MmY5ODk3ZGNiY2Q4YjJiOWY3OTJjOWU4NyIsInZlcnNpb24iOjF9.yKGKdtzMlv81Ym7bCPlEjDMrWhwKO7GHuog9I5PjvnmOwtM2TVTLk8XqIvU3_GnlSBfffNEe12pCJ-zQ27Q3BQ
          - type: rouge
            value: 10.1469
            name: ROUGE-LSUM
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZWExM2I0ZjhmYjE5YzQ1YjhiMTdmOTQzNDc5ODM0NjQ5MzMyNDVkMTUxMzg0MDVhNDU2OTNlN2EzNTc1ZGFlYyIsInZlcnNpb24iOjF9.-6_6JVdsBYdDN9Gi5iuEPchyaY0K3az06nTylQKA22bX1mBziQ2Y4z8crdzxF_hf_z1pPunWnhyLj3yUn4KPCA
          - type: loss
            value: 4.3302483558654785
            name: loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNWI4OWNlYTJiNGU5OWRhYmE4MTFiY2YxNTk4NWY2M2RjMDE2ODY4NzBmYWVmNjUxYmY2MzZiNjBmMTY1ODgxMiIsInZlcnNpb24iOjF9.QDPl5BGgPsu9XdEC0TA_Zjhb47nEHFM9ysBTvDs75-1kp_Y6aqB-xIPFp03llsXBHGnbyAr4WQhFRtxDdlrKDw
          - type: gen_len
            value: 18.9984
            name: gen_len
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiODYxZGRhZWFiYzQwODlhNzM0NTlkNzVhN2FkOWYzZDczMmE1MjU0ODdhNGRmYjc1NTE0NTIyYjhhZjRhOGQ1YyIsInZlcnNpb24iOjF9.z5jkmfzM0btlSKulTh7w6FZk66q0JRS803mVgBT1nL88vDaCTezb4wRYbXazssdfo2V8J-EY3r_VwVTAxQxgAw

summarization_mlsum

This model is a fine-tuned version of gsarti/it5-base on MLSum-it for Abstractive Summarization.

It achieves the following results:

Loss: 2.0190
Rouge1: 19.3739
Rouge2: 5.9753
Rougel: 16.691
Rougelsum: 16.7862
Gen Len: 32.5268

Usage

from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained("ARTeLab/it5-summarization-mlsum")
model = T5ForConditionalGeneration.from_pretrained("ARTeLab/it5-summarization-mlsum")

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 6
eval_batch_size: 6
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 4.0

Framework versions

Transformers 4.12.0.dev0
Pytorch 1.9.1+cu102
Datasets 1.12.1
Tokenizers 0.10.3

Citation

More details and results in published work

@Article{info13050228,
    AUTHOR = {Landro, Nicola and Gallo, Ignazio and La Grassa, Riccardo and Federici, Edoardo},
    TITLE = {Two New Datasets for Italian-Language Abstractive Text Summarization},
    JOURNAL = {Information},
    VOLUME = {13},
    YEAR = {2022},
    NUMBER = {5},
    ARTICLE-NUMBER = {228},
    URL = {https://www.mdpi.com/2078-2489/13/5/228},
    ISSN = {2078-2489},
    ABSTRACT = {Text summarization aims to produce a short summary containing relevant parts from a given text. Due to the lack of data for abstractive summarization on low-resource languages such as Italian, we propose two new original datasets collected from two Italian news websites with multi-sentence summaries and corresponding articles, and from a dataset obtained by machine translation of a Spanish summarization dataset. These two datasets are currently the only two available in Italian for this task. To evaluate the quality of these two datasets, we used them to train a T5-base model and an mBART model, obtaining good results with both. To better evaluate the results obtained, we also compared the same models trained on automatically translated datasets, and the resulting summaries in the same training language, with the automatically translated summaries, which demonstrated the superiority of the models obtained from the proposed datasets.},
    DOI = {10.3390/info13050228}
}