pytorch tensorboard t5 text2text-generation transformers generated_from_trainer license:apache-2.0 model-index autotrain_compatible
---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: t5-small-mlm-pubmed-35
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-small-mlm-pubmed-35

This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 1.1101
- Rouge2 Precision: 0.4758
- Rouge2 Recall: 0.3498
- Rouge2 Fmeasure: 0.3927

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 40
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Rouge2 Precision | Rouge2 Recall | Rouge2 Fmeasure |
|:-------------:|:-----:|:-----:|:---------------:|:----------------:|:-------------:|:---------------:|
| 1.8404        | 0.75  | 500   | 1.5005          | 0.4265           | 0.2786        | 0.3273          |
| 1.6858        | 1.51  | 1000  | 1.4216          | 0.4318           | 0.2946        | 0.3404          |
| 1.6071        | 2.26  | 1500  | 1.3777          | 0.4472           | 0.3148        | 0.3598          |
| 1.5551        | 3.02  | 2000  | 1.3360          | 0.4406           | 0.3168        | 0.3586          |
| 1.5116        | 3.77  | 2500  | 1.3128          | 0.4523           | 0.3234        | 0.3671          |
| 1.4837        | 4.52  | 3000  | 1.2937          | 0.4477           | 0.3215        | 0.3645          |
| 1.4513        | 5.28  | 3500  | 1.2766          | 0.4511           | 0.3262        | 0.3689          |
| 1.4336        | 6.03  | 4000  | 1.2626          | 0.4548           | 0.3283        | 0.3718          |
| 1.4149        | 6.79  | 4500  | 1.2449          | 0.4495           | 0.3274        | 0.3687          |
| 1.3977        | 7.54  | 5000  | 1.2349          | 0.4507           | 0.3305        | 0.3712          |
| 1.3763        | 8.3   | 5500  | 1.2239          | 0.4519           | 0.3266        | 0.3688          |
| 1.371         | 9.05  | 6000  | 1.2171          | 0.4546           | 0.3305        | 0.3727          |
| 1.3501        | 9.8   | 6500  | 1.2080          | 0.4575           | 0.3329        | 0.3755          |
| 1.3443        | 10.56 | 7000  | 1.2017          | 0.4576           | 0.3314        | 0.3742          |
| 1.326         | 11.31 | 7500  | 1.1926          | 0.4578           | 0.333         | 0.3757          |
| 1.3231        | 12.07 | 8000  | 1.1866          | 0.4606           | 0.3357        | 0.3782          |
| 1.3089        | 12.82 | 8500  | 1.1816          | 0.4591           | 0.3338        | 0.3765          |
| 1.3007        | 13.57 | 9000  | 1.1764          | 0.4589           | 0.3361        | 0.3777          |
| 1.2943        | 14.33 | 9500  | 1.1717          | 0.4641           | 0.3382        | 0.3811          |
| 1.2854        | 15.08 | 10000 | 1.1655          | 0.4617           | 0.3378        | 0.38            |
| 1.2777        | 15.84 | 10500 | 1.1612          | 0.464            | 0.3401        | 0.3823          |
| 1.2684        | 16.59 | 11000 | 1.1581          | 0.4608           | 0.3367        | 0.3789          |
| 1.2612        | 17.35 | 11500 | 1.1554          | 0.4623           | 0.3402        | 0.3818          |
| 1.2625        | 18.1  | 12000 | 1.1497          | 0.4613           | 0.3381        | 0.3802          |
| 1.2529        | 18.85 | 12500 | 1.1465          | 0.4671           | 0.3419        | 0.3848          |
| 1.2461        | 19.61 | 13000 | 1.1431          | 0.4646           | 0.3399        | 0.3824          |
| 1.2415        | 20.36 | 13500 | 1.1419          | 0.4659           | 0.341         | 0.3835          |
| 1.2375        | 21.12 | 14000 | 1.1377          | 0.4693           | 0.3447        | 0.3873          |
| 1.2315        | 21.87 | 14500 | 1.1353          | 0.4672           | 0.3433        | 0.3855          |
| 1.2263        | 22.62 | 15000 | 1.1333          | 0.467            | 0.3433        | 0.3854          |
| 1.2214        | 23.38 | 15500 | 1.1305          | 0.4682           | 0.3446        | 0.3869          |
| 1.2202        | 24.13 | 16000 | 1.1291          | 0.4703           | 0.3465        | 0.3888          |
| 1.2155        | 24.89 | 16500 | 1.1270          | 0.472            | 0.348         | 0.3903          |
| 1.2064        | 25.64 | 17000 | 1.1261          | 0.4724           | 0.3479        | 0.3905          |
| 1.2173        | 26.4  | 17500 | 1.1236          | 0.4734           | 0.3485        | 0.3912          |
| 1.1994        | 27.15 | 18000 | 1.1220          | 0.4739           | 0.3486        | 0.3915          |
| 1.2018        | 27.9  | 18500 | 1.1217          | 0.4747           | 0.3489        | 0.3921          |
| 1.2045        | 28.66 | 19000 | 1.1194          | 0.4735           | 0.3488        | 0.3916          |
| 1.1949        | 29.41 | 19500 | 1.1182          | 0.4732           | 0.3484        | 0.3911          |
| 1.19          | 30.17 | 20000 | 1.1166          | 0.4724           | 0.3479        | 0.3904          |
| 1.1932        | 30.92 | 20500 | 1.1164          | 0.4753           | 0.3494        | 0.3924          |
| 1.1952        | 31.67 | 21000 | 1.1147          | 0.4733           | 0.3485        | 0.3911          |
| 1.1922        | 32.43 | 21500 | 1.1146          | 0.475            | 0.3494        | 0.3923          |
| 1.1889        | 33.18 | 22000 | 1.1132          | 0.4765           | 0.3499        | 0.3933          |
| 1.1836        | 33.94 | 22500 | 1.1131          | 0.4768           | 0.351         | 0.3939          |
| 1.191         | 34.69 | 23000 | 1.1127          | 0.4755           | 0.3495        | 0.3926          |
| 1.1811        | 35.44 | 23500 | 1.1113          | 0.4748           | 0.349         | 0.3919          |
| 1.1864        | 36.2  | 24000 | 1.1107          | 0.4751           | 0.3494        | 0.3921          |
| 1.1789        | 36.95 | 24500 | 1.1103          | 0.4756           | 0.3499        | 0.3927          |
| 1.1819        | 37.71 | 25000 | 1.1101          | 0.4758           | 0.35          | 0.3932          |
| 1.1862        | 38.46 | 25500 | 1.1099          | 0.4755           | 0.3497        | 0.3926          |
| 1.1764        | 39.22 | 26000 | 1.1101          | 0.4759           | 0.3498        | 0.3928          |
| 1.1819        | 39.97 | 26500 | 1.1101          | 0.4758           | 0.3498        | 0.3927          |


### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3
