pytorch distilbert fill-mask transformers generated_from_trainer model-index autotrain_compatible
---
tags:
- generated_from_trainer
model-index:
- name: distilbert-base-uncased-continued_training-medqa
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-uncased-continued_training-medqa

This model is a fine-tuned version of [Shaier/distilbert-base-uncased-continued_training-medqa](https://huggingface.co/Shaier/distilbert-base-uncased-continued_training-medqa) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5389

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 512
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 220
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| No log        | 1.0   | 333   | 0.4516          |
| No log        | 2.0   | 666   | 0.4277          |
| No log        | 3.0   | 999   | 0.3734          |
| No log        | 4.0   | 1332  | 0.4083          |
| No log        | 5.0   | 1665  | 0.4134          |
| No log        | 6.0   | 1998  | 0.5093          |
| No log        | 7.0   | 2331  | 0.4639          |
| 0.4564        | 8.0   | 2664  | 0.5132          |
| 0.4564        | 9.0   | 2997  | 0.3483          |
| 0.4564        | 10.0  | 3330  | 0.4174          |
| 0.4564        | 11.0  | 3663  | 0.4975          |
| 0.4564        | 12.0  | 3996  | 0.4030          |
| 0.4564        | 13.0  | 4329  | 0.4476          |
| 0.4564        | 14.0  | 4662  | 0.3692          |
| 0.4564        | 15.0  | 4995  | 0.4474          |
| 0.4533        | 16.0  | 5328  | 0.3289          |
| 0.4533        | 17.0  | 5661  | 0.4647          |
| 0.4533        | 18.0  | 5994  | 0.4873          |
| 0.4533        | 19.0  | 6327  | 0.5323          |
| 0.4533        | 20.0  | 6660  | 0.4273          |
| 0.4533        | 21.0  | 6993  | 0.3426          |
| 0.4533        | 22.0  | 7326  | 0.3892          |
| 0.4533        | 23.0  | 7659  | 0.4297          |
| 0.4493        | 24.0  | 7992  | 0.4162          |
| 0.4493        | 25.0  | 8325  | 0.4424          |
| 0.4493        | 26.0  | 8658  | 0.4575          |
| 0.4493        | 27.0  | 8991  | 0.4192          |
| 0.4493        | 28.0  | 9324  | 0.4151          |
| 0.4493        | 29.0  | 9657  | 0.4321          |
| 0.4493        | 30.0  | 9990  | 0.4129          |
| 0.4493        | 31.0  | 10323 | 0.4869          |
| 0.4456        | 32.0  | 10656 | 0.4510          |
| 0.4456        | 33.0  | 10989 | 0.5263          |
| 0.4456        | 34.0  | 11322 | 0.3908          |
| 0.4456        | 35.0  | 11655 | 0.5016          |
| 0.4456        | 36.0  | 11988 | 0.4454          |
| 0.4456        | 37.0  | 12321 | 0.4011          |
| 0.4456        | 38.0  | 12654 | 0.4714          |
| 0.4456        | 39.0  | 12987 | 0.4972          |
| 0.443         | 40.0  | 13320 | 0.4200          |
| 0.443         | 41.0  | 13653 | 0.4659          |
| 0.443         | 42.0  | 13986 | 0.4758          |
| 0.443         | 43.0  | 14319 | 0.4509          |
| 0.443         | 44.0  | 14652 | 0.4211          |
| 0.443         | 45.0  | 14985 | 0.4007          |
| 0.443         | 46.0  | 15318 | 0.3205          |
| 0.443         | 47.0  | 15651 | 0.4479          |
| 0.4402        | 48.0  | 15984 | 0.4723          |
| 0.4402        | 49.0  | 16317 | 0.4956          |
| 0.4402        | 50.0  | 16650 | 0.4103          |
| 0.4402        | 51.0  | 16983 | 0.4234          |
| 0.4402        | 52.0  | 17316 | 0.4052          |
| 0.4402        | 53.0  | 17649 | 0.4033          |
| 0.4402        | 54.0  | 17982 | 0.4139          |
| 0.4402        | 55.0  | 18315 | 0.3618          |
| 0.4372        | 56.0  | 18648 | 0.5102          |
| 0.4372        | 57.0  | 18981 | 0.4166          |
| 0.4372        | 58.0  | 19314 | 0.4475          |
| 0.4372        | 59.0  | 19647 | 0.4259          |
| 0.4372        | 60.0  | 19980 | 0.4018          |
| 0.4372        | 61.0  | 20313 | 0.5005          |
| 0.4372        | 62.0  | 20646 | 0.4445          |
| 0.4372        | 63.0  | 20979 | 0.4280          |
| 0.434         | 64.0  | 21312 | 0.4533          |
| 0.434         | 65.0  | 21645 | 0.3672          |
| 0.434         | 66.0  | 21978 | 0.4726          |
| 0.434         | 67.0  | 22311 | 0.4084          |
| 0.434         | 68.0  | 22644 | 0.4508          |
| 0.434         | 69.0  | 22977 | 0.3746          |
| 0.434         | 70.0  | 23310 | 0.4703          |
| 0.434         | 71.0  | 23643 | 0.4789          |
| 0.4314        | 72.0  | 23976 | 0.3963          |
| 0.4314        | 73.0  | 24309 | 0.3800          |
| 0.4314        | 74.0  | 24642 | 0.5051          |
| 0.4314        | 75.0  | 24975 | 0.4245          |
| 0.4314        | 76.0  | 25308 | 0.4745          |
| 0.4314        | 77.0  | 25641 | 0.4351          |
| 0.4314        | 78.0  | 25974 | 0.4367          |
| 0.4314        | 79.0  | 26307 | 0.4200          |
| 0.4291        | 80.0  | 26640 | 0.4985          |
| 0.4291        | 81.0  | 26973 | 0.5058          |
| 0.4291        | 82.0  | 27306 | 0.4154          |
| 0.4291        | 83.0  | 27639 | 0.4837          |
| 0.4291        | 84.0  | 27972 | 0.3865          |
| 0.4291        | 85.0  | 28305 | 0.4357          |
| 0.4291        | 86.0  | 28638 | 0.3978          |
| 0.4291        | 87.0  | 28971 | 0.4413          |
| 0.4263        | 88.0  | 29304 | 0.4223          |
| 0.4263        | 89.0  | 29637 | 0.4241          |
| 0.4263        | 90.0  | 29970 | 0.4525          |
| 0.4263        | 91.0  | 30303 | 0.3895          |
| 0.4263        | 92.0  | 30636 | 0.4207          |
| 0.4263        | 93.0  | 30969 | 0.3217          |
| 0.4263        | 94.0  | 31302 | 0.3725          |
| 0.4263        | 95.0  | 31635 | 0.4354          |
| 0.4239        | 96.0  | 31968 | 0.4169          |
| 0.4239        | 97.0  | 32301 | 0.4873          |
| 0.4239        | 98.0  | 32634 | 0.4219          |
| 0.4239        | 99.0  | 32967 | 0.4984          |
| 0.4239        | 100.0 | 33300 | 0.4078          |
| 0.4239        | 101.0 | 33633 | 0.4463          |
| 0.4239        | 102.0 | 33966 | 0.3371          |
| 0.4239        | 103.0 | 34299 | 0.3896          |
| 0.422         | 104.0 | 34632 | 0.4743          |
| 0.422         | 105.0 | 34965 | 0.4931          |
| 0.422         | 106.0 | 35298 | 0.3574          |
| 0.422         | 107.0 | 35631 | 0.4127          |
| 0.422         | 108.0 | 35964 | 0.3892          |
| 0.422         | 109.0 | 36297 | 0.3881          |
| 0.422         | 110.0 | 36630 | 0.4221          |
| 0.422         | 111.0 | 36963 | 0.3924          |
| 0.4204        | 112.0 | 37296 | 0.4067          |
| 0.4204        | 113.0 | 37629 | 0.4357          |
| 0.4204        | 114.0 | 37962 | 0.4175          |
| 0.4204        | 115.0 | 38295 | 0.4424          |
| 0.4204        | 116.0 | 38628 | 0.3925          |
| 0.4204        | 117.0 | 38961 | 0.4693          |
| 0.4204        | 118.0 | 39294 | 0.3503          |
| 0.4204        | 119.0 | 39627 | 0.4761          |
| 0.4183        | 120.0 | 39960 | 0.3816          |
| 0.4183        | 121.0 | 40293 | 0.3903          |
| 0.4183        | 122.0 | 40626 | 0.3535          |
| 0.4183        | 123.0 | 40959 | 0.4388          |
| 0.4183        | 124.0 | 41292 | 0.4519          |
| 0.4183        | 125.0 | 41625 | 0.4241          |
| 0.4183        | 126.0 | 41958 | 0.4085          |
| 0.4183        | 127.0 | 42291 | 0.4836          |
| 0.4168        | 128.0 | 42624 | 0.4101          |
| 0.4168        | 129.0 | 42957 | 0.4749          |
| 0.4168        | 130.0 | 43290 | 0.4022          |
| 0.4168        | 131.0 | 43623 | 0.4861          |
| 0.4168        | 132.0 | 43956 | 0.4376          |
| 0.4168        | 133.0 | 44289 | 0.4597          |
| 0.4168        | 134.0 | 44622 | 0.4154          |
| 0.4168        | 135.0 | 44955 | 0.4431          |
| 0.415         | 136.0 | 45288 | 0.4887          |
| 0.415         | 137.0 | 45621 | 0.4229          |
| 0.415         | 138.0 | 45954 | 0.3997          |
| 0.415         | 139.0 | 46287 | 0.4185          |
| 0.415         | 140.0 | 46620 | 0.4633          |
| 0.415         | 141.0 | 46953 | 0.4061          |
| 0.415         | 142.0 | 47286 | 0.4604          |
| 0.415         | 143.0 | 47619 | 0.4047          |
| 0.4139        | 144.0 | 47952 | 0.4272          |
| 0.4139        | 145.0 | 48285 | 0.4783          |
| 0.4139        | 146.0 | 48618 | 0.3954          |
| 0.4139        | 147.0 | 48951 | 0.4501          |
| 0.4139        | 148.0 | 49284 | 0.4941          |
| 0.4139        | 149.0 | 49617 | 0.4112          |
| 0.4139        | 150.0 | 49950 | 0.4582          |
| 0.4139        | 151.0 | 50283 | 0.4361          |
| 0.4126        | 152.0 | 50616 | 0.3535          |
| 0.4126        | 153.0 | 50949 | 0.3797          |
| 0.4126        | 154.0 | 51282 | 0.4080          |
| 0.4126        | 155.0 | 51615 | 0.4049          |
| 0.4126        | 156.0 | 51948 | 0.4255          |
| 0.4126        | 157.0 | 52281 | 0.4303          |
| 0.4126        | 158.0 | 52614 | 0.4950          |
| 0.4126        | 159.0 | 52947 | 0.3721          |
| 0.4114        | 160.0 | 53280 | 0.2861          |
| 0.4114        | 161.0 | 53613 | 0.3775          |
| 0.4114        | 162.0 | 53946 | 0.4274          |
| 0.4114        | 163.0 | 54279 | 0.3904          |
| 0.4114        | 164.0 | 54612 | 0.4687          |
| 0.4114        | 165.0 | 54945 | 0.4013          |
| 0.4114        | 166.0 | 55278 | 0.4760          |
| 0.4114        | 167.0 | 55611 | 0.3554          |
| 0.4104        | 168.0 | 55944 | 0.5193          |
| 0.4104        | 169.0 | 56277 | 0.4476          |
| 0.4104        | 170.0 | 56610 | 0.5011          |
| 0.4104        | 171.0 | 56943 | 0.4441          |
| 0.4104        | 172.0 | 57276 | 0.4457          |
| 0.4104        | 173.0 | 57609 | 0.3792          |
| 0.4104        | 174.0 | 57942 | 0.5116          |
| 0.4104        | 175.0 | 58275 | 0.4249          |
| 0.4097        | 176.0 | 58608 | 0.3804          |
| 0.4097        | 177.0 | 58941 | 0.3886          |
| 0.4097        | 178.0 | 59274 | 0.4420          |
| 0.4097        | 179.0 | 59607 | 0.3573          |
| 0.4097        | 180.0 | 59940 | 0.3635          |
| 0.4097        | 181.0 | 60273 | 0.4596          |
| 0.4097        | 182.0 | 60606 | 0.3674          |
| 0.4097        | 183.0 | 60939 | 0.3869          |
| 0.409         | 184.0 | 61272 | 0.3909          |
| 0.409         | 185.0 | 61605 | 0.4339          |
| 0.409         | 186.0 | 61938 | 0.4475          |
| 0.409         | 187.0 | 62271 | 0.3218          |
| 0.409         | 188.0 | 62604 | 0.3771          |
| 0.409         | 189.0 | 62937 | 0.4007          |
| 0.409         | 190.0 | 63270 | 0.4520          |
| 0.409         | 191.0 | 63603 | 0.3980          |
| 0.4077        | 192.0 | 63936 | 0.4572          |
| 0.4077        | 193.0 | 64269 | 0.3952          |
| 0.4077        | 194.0 | 64602 | 0.4384          |
| 0.4077        | 195.0 | 64935 | 0.4795          |
| 0.4077        | 196.0 | 65268 | 0.3743          |
| 0.4077        | 197.0 | 65601 | 0.4445          |
| 0.4077        | 198.0 | 65934 | 0.3925          |
| 0.4077        | 199.0 | 66267 | 0.4564          |
| 0.4075        | 200.0 | 66600 | 0.4580          |
| 0.4075        | 201.0 | 66933 | 0.4446          |
| 0.4075        | 202.0 | 67266 | 0.4289          |
| 0.4075        | 203.0 | 67599 | 0.3722          |
| 0.4075        | 204.0 | 67932 | 0.4810          |
| 0.4075        | 205.0 | 68265 | 0.4004          |
| 0.4075        | 206.0 | 68598 | 0.4219          |
| 0.4075        | 207.0 | 68931 | 0.3926          |
| 0.407         | 208.0 | 69264 | 0.6043          |
| 0.407         | 209.0 | 69597 | 0.3835          |
| 0.407         | 210.0 | 69930 | 0.3791          |
| 0.407         | 211.0 | 70263 | 0.4152          |
| 0.407         | 212.0 | 70596 | 0.3654          |
| 0.407         | 213.0 | 70929 | 0.4434          |
| 0.407         | 214.0 | 71262 | 0.3613          |
| 0.407         | 215.0 | 71595 | 0.5103          |
| 0.4069        | 216.0 | 71928 | 0.3733          |
| 0.4069        | 217.0 | 72261 | 0.4881          |
| 0.4069        | 218.0 | 72594 | 0.3375          |
| 0.4069        | 219.0 | 72927 | 0.4766          |
| 0.4069        | 220.0 | 73260 | 0.4604          |


### Framework versions

- Transformers 4.18.0
- Pytorch 1.11.0
- Datasets 2.3.2
- Tokenizers 0.11.0
