parallel-mean-bottleneck-gpt2-medium-wikitext

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 3.1861
  • Accuracy: 0.4193
  • Perplexity: 24.1930
  • Bleu: 0.1440

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 64
  • eval_batch_size: 64
  • seed: 42
  • optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 5

Training results

Training Loss Epoch Step Validation Loss Accuracy Perplexity Bleu
6.0438 0.2806 500 5.9200 0.1897 372.4009 0.0359
5.0422 0.5612 1000 4.8934 0.2636 133.4091 0.0610
4.3494 0.8418 1500 4.2389 0.3183 69.3337 0.0833
3.9486 1.1223 2000 3.8856 0.3521 48.6953 0.1037
3.7605 1.4029 2500 3.7143 0.3671 41.0301 0.1206
3.6544 1.6835 3000 3.5898 0.3781 36.2282 0.1332
3.5527 1.9641 3500 3.5051 0.3862 33.2836 0.1349
3.4346 2.2447 4000 3.4410 0.3919 31.2181 0.1335
3.374 2.5253 4500 3.3867 0.3972 29.5672 0.1354
3.3442 2.8058 5000 3.3410 0.4017 28.2468 0.1405
3.2251 3.0864 5500 3.3072 0.4055 27.3093 0.1404
3.2187 3.3670 6000 3.2781 0.4088 26.5242 0.1401
3.1975 3.6476 6500 3.2494 0.4118 25.7753 0.1433
3.172 3.9282 7000 3.2276 0.4142 25.2178 0.1445
3.1055 4.2088 7500 3.2109 0.4163 24.8014 0.1447
3.0676 4.4893 8000 3.1977 0.4178 24.4763 0.1453
3.0779 4.7699 8500 3.1861 0.4193 24.1930 0.1440

Framework versions

  • Transformers 4.49.0
  • Pytorch 2.6.0+cu124
  • Datasets 3.3.2
  • Tokenizers 0.21.0
Downloads last month
3
Safetensors
Model size
355M params
Tensor type
F32
·
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The HF Inference API does not support model that require custom code execution.