End of training

README.md

@@ -15,14 +15,14 @@ This student model is distilled from the teacher model [roneneldan/TinyStories-3
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 172.3267
-- eval_frwikippl: 37035.1875
-- eval_zhwikippl: 194088.4531
-- eval_tinystoriesppl: 10.7883
-- eval_loss: 1.3421
-- eval_runtime: 6.5001
-- eval_samples_per_second: 76.922
-- eval_steps_per_second: 9.692
+- eval_enwikippl: 3607.3171
+- eval_frwikippl: 29425.125
+- eval_zhwikippl: 52510.3125
+- eval_tinystoriesppl: 1167.9218
+- eval_loss: 5.1093
+- eval_runtime: 6.505
+- eval_samples_per_second: 76.864
+- eval_steps_per_second: 9.685
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -47,7 +47,7 @@ More information needed
 The following hyperparameters were used during training:
 - distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
 - train_embeddings: True
-- learning_rate: 0.004
+- learning_rate: 0.0004
 - train_batch_size: 8
 - eval_batch_size: 8
 - seed: 42
@@ -62,20 +62,20 @@ Peak GPU Memory: 8.0568 GB
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 169.9865 | 47377.9414 |  |  |  |  | 3.9789 | 4998.1294 |
-| 0 | 0 | 21321.3555 | 56774.5312 | 6.6010 | 6.4987 | 76.938 | 9.694 | 11289.9248 | 60744.7383 |
-| 500 | 0.0808 | 209.4913 | 62706.3438 | 1.4389 | 6.5968 | 75.794 | 9.55 | 11.3247 | 298068.5312 |
-| 1000 | 0.1616 | 182.2395 | 44100.0312 | 1.3516 | 6.4978 | 76.95 | 9.696 | 10.7194 | 255063.875 |
-| 1500 | 0.2424 | 174.5435 | 39099.0508 | 1.3434 | 6.5516 | 76.317 | 9.616 | 10.7310 | 199230.0 |
-| 2000 | 0.3232 | 173.0893 | 37756.8164 | 1.3422 | 6.4982 | 76.945 | 9.695 | 10.7545 | 194918.7188 |
-| 2500 | 0.4040 | 171.9666 | 36889.3945 | 1.3422 | 6.5098 | 76.807 | 9.678 | 10.7906 | 195543.75 |
-| 3000 | 0.4848 | 171.2156 | 36931.0 | 1.3418 | 6.4894 | 77.049 | 9.708 | 10.7314 | 190904.3438 |
-| 3500 | 0.5656 | 172.5805 | 37171.0625 | 1.3418 | 6.4818 | 77.14 | 9.72 | 10.8124 | 193984.8281 |
-| 4000 | 0.6464 | 171.9800 | 37035.1875 | 1.3417 | 6.4879 | 77.067 | 9.71 | 10.7732 | 191414.4375 |
-| 4500 | 0.7272 | 172.1532 | 37056.0664 | 1.3423 | 6.5945 | 75.821 | 9.553 | 10.7879 | 193984.8281 |
-| 5000 | 0.8080 | 172.3400 | 37035.1875 | 1.3422 | 6.5862 | 75.916 | 9.565 | 10.7968 | 196799.8281 |
-| 5500 | 0.8888 | 172.2065 | 37035.1875 | 1.3422 | 6.4984 | 76.943 | 9.695 | 10.7714 | 193984.8281 |
-| 6000 | 0.9696 | 172.3267 | 37035.1875 | 1.3419 | 6.4894 | 77.048 | 9.708 | 10.7910 | 193984.8281 |
-| 6188 | 1.0 | 172.3267 | 37035.1875 | 1.3421 | 6.5001 | 76.922 | 9.692 | 10.7883 | 194088.4531 |
+| 0 | 0 | 21321.3555 | 56774.5312 | 6.6010 | 6.5635 | 76.178 | 9.598 | 11289.9248 | 60744.7383 |
+| 500 | 0.0808 | 3754.7207 | 29512.3027 | 5.1110 | 6.5245 | 76.634 | 9.656 | 1235.4543 | 53915.7461 |
+| 1000 | 0.1616 | 3629.7410 | 29470.7617 | 5.1093 | 6.5374 | 76.483 | 9.637 | 1179.3701 | 52678.6953 |
+| 1500 | 0.2424 | 3604.8032 | 29425.125 | 5.1093 | 6.5363 | 76.496 | 9.638 | 1167.5359 | 52510.3125 |
+| 2000 | 0.3232 | 3604.8032 | 29425.125 | 5.1093 | 6.5198 | 76.689 | 9.663 | 1167.3427 | 52510.3125 |
+| 2500 | 0.4040 | 3607.3171 | 29425.125 | 5.1093 | 6.5054 | 76.86 | 9.684 | 1167.9218 | 52510.3125 |
+| 3000 | 0.4848 | 3607.3171 | 29425.125 | 5.1093 | 6.5103 | 76.801 | 9.677 | 1167.9218 | 52510.3125 |
+| 3500 | 0.5656 | 3607.3171 | 29425.125 | 5.1093 | 6.4942 | 76.992 | 9.701 | 1167.9218 | 52510.3125 |
+| 4000 | 0.6464 | 3607.3171 | 29425.125 | 5.1093 | 6.516 | 76.734 | 9.669 | 1167.9218 | 52510.3125 |
+| 4500 | 0.7272 | 3607.3171 | 29425.125 | 5.1093 | 6.5209 | 76.677 | 9.661 | 1167.9218 | 52510.3125 |
+| 5000 | 0.8080 | 3607.3171 | 29425.125 | 5.1093 | 6.502 | 76.899 | 9.689 | 1167.9218 | 52510.3125 |
+| 5500 | 0.8888 | 3607.3171 | 29425.125 | 5.1093 | 6.5361 | 76.498 | 9.639 | 1167.9218 | 52510.3125 |
+| 6000 | 0.9696 | 3607.3171 | 29425.125 | 5.1093 | 6.4994 | 76.93 | 9.693 | 1167.9218 | 52510.3125 |
+| 6188 | 1.0 | 3607.3171 | 29425.125 | 5.1093 | 6.505 | 76.864 | 9.685 | 1167.9218 | 52510.3125 |
 
 ### Framework versions
 - Distily 0.2.0

logs/dropout=0, learning_rate=0.0004, weight_decay=0.1/events.out.tfevents.1723870375.5f530b1cf724

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a14592e0d6c72a3ab368153e1a868275469b0d7ac0d61e8d181e1c8b3383ab9
+size 307