pytorch t5 text2text-generation transformers generated_from_trainer license:apache-2.0 model-index autotrain_compatible
---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- rouge
model-index:
- name: t5-small-finetuned-qa-google-en_v1
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-small-finetuned-qa-google-en_v1

This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 2.1413
- Rouge1: 40.2873
- Rouge2: 30.6667
- Rougel: 40.1625
- Rougelsum: 40.2529
- Gen Len: 6.352

## Model description

Model finetuned to generate answers based on an input paragraph.

Example input:


> extract answer: \<hl\> The most recent major version of Python is Python 3, which we shall be using in this tutorial. \<hl\> However, Python 2, although not being updated with anything other than security updates, is still quite popular.

## Intended uses & limitations

More information needed

## Training and evaluation data

Model trained on Google Natural Questions in this repo: https://huggingface.co/datasets/IsaacBot/Natural-Questions-Sentence-Highlight. The dataset allows training for question and/or answer generation. This model was trained to generate answer based on an input paragraph (no question hint).

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 256
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5
- mixed_precision_training: Native AMP
- label_smoothing_factor: 0.1

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rouge1  | Rouge2  | Rougel  | Rougelsum | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
| No log        | 0.27  | 100  | 2.3595          | 33.588  | 25.61   | 33.5609 | 33.6388   | 6.141   |
| No log        | 0.53  | 200  | 2.2774          | 34.5528 | 26.0686 | 34.5097 | 34.58     | 6.3135  |
| No log        | 0.8   | 300  | 2.2435          | 36.1641 | 27.857  | 36.134  | 36.1435   | 6.5255  |
| No log        | 1.06  | 400  | 2.2117          | 36.7651 | 28.2274 | 36.7426 | 36.7278   | 6.3635  |
| 2.3999        | 1.33  | 500  | 2.1987          | 38.0006 | 29.1016 | 37.9354 | 37.9785   | 6.485   |
| 2.3999        | 1.6   | 600  | 2.1846          | 37.978  | 28.9293 | 37.8811 | 37.9299   | 6.321   |
| 2.3999        | 1.86  | 700  | 2.1782          | 38.6531 | 29.582  | 38.5923 | 38.5797   | 6.484   |
| 2.3999        | 2.13  | 800  | 2.1726          | 38.8536 | 29.9533 | 38.7785 | 38.8452   | 6.537   |
| 2.3999        | 2.39  | 900  | 2.1640          | 38.7099 | 29.9414 | 38.6318 | 38.6736   | 6.4635  |
| 2.2365        | 2.66  | 1000 | 2.1563          | 39.2126 | 30.0471 | 39.1838 | 39.1994   | 6.4135  |
| 2.2365        | 2.93  | 1100 | 2.1579          | 39.6397 | 30.5926 | 39.5747 | 39.6701   | 6.4395  |
| 2.2365        | 3.19  | 1200 | 2.1525          | 39.3201 | 30.1897 | 39.2362 | 39.3391   | 6.4305  |
| 2.2365        | 3.46  | 1300 | 2.1514          | 39.4479 | 30.2987 | 39.3595 | 39.374    | 6.311   |
| 2.2365        | 3.72  | 1400 | 2.1478          | 39.7449 | 30.6635 | 39.6403 | 39.6993   | 6.372   |
| 2.1979        | 3.99  | 1500 | 2.1453          | 39.7789 | 30.4461 | 39.6917 | 39.8043   | 6.4215  |
| 2.1979        | 4.26  | 1600 | 2.1427          | 39.8127 | 30.3727 | 39.6894 | 39.7522   | 6.3165  |
| 2.1979        | 4.52  | 1700 | 2.1426          | 40.3472 | 30.7014 | 40.2174 | 40.3229   | 6.452   |
| 2.1979        | 4.79  | 1800 | 2.1413          | 40.2873 | 30.6667 | 40.1625 | 40.2529   | 6.352   |


### Framework versions

- Transformers 4.17.0
- Pytorch 1.10.0+cu111
- Datasets 2.0.0
- Tokenizers 0.11.6
