pytorch tensorboard t5 text2text-generation transformers generated_from_trainer license:apache-2.0 model-index autotrain_compatible
---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: t5-small-mlm-pubmed-15
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# t5-small-mlm-pubmed-15

This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5389
- Rouge2 Precision: 0.7165
- Rouge2 Recall: 0.5375
- Rouge2 Fmeasure: 0.5981

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 40
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Rouge2 Precision | Rouge2 Recall | Rouge2 Fmeasure |
|:-------------:|:-----:|:-----:|:---------------:|:----------------:|:-------------:|:---------------:|
| 1.1024        | 0.75  | 500   | 0.7890          | 0.6854           | 0.4813        | 0.5502          |
| 0.8788        | 1.51  | 1000  | 0.7176          | 0.6906           | 0.4989        | 0.5638          |
| 0.8086        | 2.26  | 1500  | 0.6830          | 0.6872           | 0.5052        | 0.5663          |
| 0.7818        | 3.02  | 2000  | 0.6650          | 0.6912           | 0.5104        | 0.5711          |
| 0.7466        | 3.77  | 2500  | 0.6458          | 0.6965           | 0.5167        | 0.5774          |
| 0.731         | 4.52  | 3000  | 0.6355          | 0.6955           | 0.5161        | 0.5763          |
| 0.7126        | 5.28  | 3500  | 0.6249          | 0.6924           | 0.517         | 0.576           |
| 0.6998        | 6.03  | 4000  | 0.6166          | 0.6995           | 0.5207        | 0.5809          |
| 0.6855        | 6.79  | 4500  | 0.6076          | 0.6981           | 0.5215        | 0.5813          |
| 0.676         | 7.54  | 5000  | 0.6015          | 0.7003           | 0.5242        | 0.5836          |
| 0.6688        | 8.3   | 5500  | 0.5962          | 0.7004           | 0.5235        | 0.583           |
| 0.6569        | 9.05  | 6000  | 0.5900          | 0.6997           | 0.5234        | 0.5827          |
| 0.6503        | 9.8   | 6500  | 0.5880          | 0.703            | 0.5257        | 0.5856          |
| 0.6455        | 10.56 | 7000  | 0.5818          | 0.7008           | 0.5259        | 0.5849          |
| 0.635         | 11.31 | 7500  | 0.5796          | 0.7017           | 0.5271        | 0.5861          |
| 0.6323        | 12.07 | 8000  | 0.5769          | 0.7053           | 0.5276        | 0.5877          |
| 0.6241        | 12.82 | 8500  | 0.5730          | 0.7011           | 0.5243        | 0.5838          |
| 0.6224        | 13.57 | 9000  | 0.5696          | 0.7046           | 0.5286        | 0.5879          |
| 0.6139        | 14.33 | 9500  | 0.5685          | 0.7047           | 0.5295        | 0.5886          |
| 0.6118        | 15.08 | 10000 | 0.5653          | 0.704            | 0.5297        | 0.5886          |
| 0.6089        | 15.84 | 10500 | 0.5633          | 0.703            | 0.5272        | 0.5865          |
| 0.598         | 16.59 | 11000 | 0.5613          | 0.7059           | 0.5293        | 0.5889          |
| 0.6003        | 17.35 | 11500 | 0.5602          | 0.7085           | 0.532         | 0.5918          |
| 0.5981        | 18.1  | 12000 | 0.5587          | 0.7106           | 0.5339        | 0.5938          |
| 0.5919        | 18.85 | 12500 | 0.5556          | 0.708            | 0.5319        | 0.5914          |
| 0.5897        | 19.61 | 13000 | 0.5556          | 0.7106           | 0.5327        | 0.5931          |
| 0.5899        | 20.36 | 13500 | 0.5526          | 0.7114           | 0.534         | 0.5939          |
| 0.5804        | 21.12 | 14000 | 0.5521          | 0.7105           | 0.5328        | 0.5928          |
| 0.5764        | 21.87 | 14500 | 0.5520          | 0.715            | 0.537         | 0.5976          |
| 0.5793        | 22.62 | 15000 | 0.5506          | 0.713            | 0.5346        | 0.5951          |
| 0.5796        | 23.38 | 15500 | 0.5492          | 0.7124           | 0.5352        | 0.5952          |
| 0.5672        | 24.13 | 16000 | 0.5482          | 0.7124           | 0.5346        | 0.5948          |
| 0.5737        | 24.89 | 16500 | 0.5470          | 0.7134           | 0.5352        | 0.5956          |
| 0.5685        | 25.64 | 17000 | 0.5463          | 0.7117           | 0.5346        | 0.5946          |
| 0.5658        | 26.4  | 17500 | 0.5457          | 0.7145           | 0.5359        | 0.5965          |
| 0.5657        | 27.15 | 18000 | 0.5447          | 0.7145           | 0.5367        | 0.597           |
| 0.5645        | 27.9  | 18500 | 0.5441          | 0.7141           | 0.5362        | 0.5964          |
| 0.565         | 28.66 | 19000 | 0.5436          | 0.7151           | 0.5367        | 0.5972          |
| 0.5579        | 29.41 | 19500 | 0.5426          | 0.7162           | 0.5378        | 0.5982          |
| 0.563         | 30.17 | 20000 | 0.5424          | 0.7155           | 0.5373        | 0.5977          |
| 0.556         | 30.92 | 20500 | 0.5418          | 0.7148           | 0.536         | 0.5966          |
| 0.5576        | 31.67 | 21000 | 0.5411          | 0.7141           | 0.5356        | 0.5961          |
| 0.5546        | 32.43 | 21500 | 0.5409          | 0.7149           | 0.5364        | 0.5967          |
| 0.556         | 33.18 | 22000 | 0.5405          | 0.7143           | 0.5356        | 0.596           |
| 0.5536        | 33.94 | 22500 | 0.5401          | 0.7165           | 0.5377        | 0.5982          |
| 0.5527        | 34.69 | 23000 | 0.5397          | 0.7188           | 0.5389        | 0.5999          |
| 0.5531        | 35.44 | 23500 | 0.5395          | 0.7172           | 0.538         | 0.5989          |
| 0.5508        | 36.2  | 24000 | 0.5392          | 0.7166           | 0.538         | 0.5985          |
| 0.5495        | 36.95 | 24500 | 0.5391          | 0.7176           | 0.5387        | 0.5993          |
| 0.5539        | 37.71 | 25000 | 0.5391          | 0.7169           | 0.5372        | 0.598           |
| 0.5452        | 38.46 | 25500 | 0.5390          | 0.7179           | 0.5384        | 0.5991          |
| 0.5513        | 39.22 | 26000 | 0.5390          | 0.717            | 0.5377        | 0.5984          |
| 0.5506        | 39.97 | 26500 | 0.5389          | 0.7165           | 0.5375        | 0.5981          |


### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3
