End of training

README.md

@@ -15,14 +15,14 @@ This student model is distilled from the teacher model [roneneldan/TinyStories-3
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 172.3267
-- eval_frwikippl: 37035.1875
-- eval_zhwikippl: 194088.4531
-- eval_tinystoriesppl: 10.7883
-- eval_loss: 1.3421
-- eval_runtime: 6.4999
-- eval_samples_per_second: 76.924
-- eval_steps_per_second: 9.692
+- eval_enwikippl: 3607.3171
+- eval_frwikippl: 29425.125
+- eval_zhwikippl: 52510.3125
+- eval_tinystoriesppl: 1167.9218
+- eval_loss: 5.1093
+- eval_runtime: 6.526
+- eval_samples_per_second: 76.617
+- eval_steps_per_second: 9.654
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -47,7 +47,7 @@ More information needed
 The following hyperparameters were used during training:
 - distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
 - train_embeddings: True
-- learning_rate: 0.004
+- learning_rate: 0.0004
 - train_batch_size: 8
 - eval_batch_size: 8
 - seed: 42
@@ -62,20 +62,20 @@ Peak GPU Memory: 8.0568 GB
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 169.9865 | 47377.9414 |  |  |  |  | 3.9789 | 4998.1294 |
-| 0 | 0 | 21321.3555 | 56774.5312 | 6.6010 | 6.5064 | 76.847 | 9.683 | 11289.9248 | 60744.7383 |
-| 500 | 0.0808 | 209.4913 | 62706.3438 | 1.4389 | 6.5754 | 76.041 | 9.581 | 11.3247 | 298068.5312 |
-| 1000 | 0.1616 | 182.2395 | 44100.0312 | 1.3516 | 6.5043 | 76.872 | 9.686 | 10.7194 | 255063.875 |
-| 1500 | 0.2424 | 174.5435 | 39099.0508 | 1.3434 | 6.5028 | 76.89 | 9.688 | 10.7310 | 199230.0 |
-| 2000 | 0.3232 | 173.0893 | 37756.8164 | 1.3422 | 6.5267 | 76.609 | 9.653 | 10.7545 | 194918.7188 |
-| 2500 | 0.4040 | 171.9666 | 36889.3945 | 1.3422 | 6.5337 | 76.526 | 9.642 | 10.7906 | 195543.75 |
-| 3000 | 0.4848 | 171.2156 | 36931.0 | 1.3418 | 6.526 | 76.616 | 9.654 | 10.7314 | 190904.3438 |
-| 3500 | 0.5656 | 172.5805 | 37171.0625 | 1.3418 | 6.51 | 76.805 | 9.677 | 10.8124 | 193984.8281 |
-| 4000 | 0.6464 | 171.9800 | 37035.1875 | 1.3417 | 6.5087 | 76.821 | 9.679 | 10.7732 | 191414.4375 |
-| 4500 | 0.7272 | 172.1532 | 37056.0664 | 1.3423 | 6.5796 | 75.992 | 9.575 | 10.7879 | 193984.8281 |
-| 5000 | 0.8080 | 172.3400 | 37035.1875 | 1.3422 | 6.4904 | 77.036 | 9.707 | 10.7968 | 196799.8281 |
-| 5500 | 0.8888 | 172.2065 | 37035.1875 | 1.3422 | 6.4994 | 76.93 | 9.693 | 10.7714 | 193984.8281 |
-| 6000 | 0.9696 | 172.3267 | 37035.1875 | 1.3419 | 6.4995 | 76.929 | 9.693 | 10.7910 | 193984.8281 |
-| 6188 | 1.0 | 172.3267 | 37035.1875 | 1.3421 | 6.4999 | 76.924 | 9.692 | 10.7883 | 194088.4531 |
+| 0 | 0 | 21321.3555 | 56774.5312 | 6.6010 | 6.5485 | 76.353 | 9.621 | 11289.9248 | 60744.7383 |
+| 500 | 0.0808 | 3754.7207 | 29512.3027 | 5.1110 | 6.493 | 77.007 | 9.703 | 1235.4543 | 53915.7461 |
+| 1000 | 0.1616 | 3629.7410 | 29470.7617 | 5.1093 | 6.5015 | 76.906 | 9.69 | 1179.3701 | 52678.6953 |
+| 1500 | 0.2424 | 3604.8032 | 29425.125 | 5.1093 | 6.4868 | 77.08 | 9.712 | 1167.5359 | 52510.3125 |
+| 2000 | 0.3232 | 3604.8032 | 29425.125 | 5.1093 | 6.4978 | 76.949 | 9.696 | 1167.3427 | 52510.3125 |
+| 2500 | 0.4040 | 3607.3171 | 29425.125 | 5.1093 | 6.5048 | 76.866 | 9.685 | 1167.9218 | 52510.3125 |
+| 3000 | 0.4848 | 3607.3171 | 29425.125 | 5.1093 | 6.5218 | 76.666 | 9.66 | 1167.9218 | 52510.3125 |
+| 3500 | 0.5656 | 3607.3171 | 29425.125 | 5.1093 | 6.5134 | 76.764 | 9.672 | 1167.9218 | 52510.3125 |
+| 4000 | 0.6464 | 3607.3171 | 29425.125 | 5.1093 | 6.6885 | 74.755 | 9.419 | 1167.9218 | 52510.3125 |
+| 4500 | 0.7272 | 3607.3171 | 29425.125 | 5.1093 | 6.5018 | 76.902 | 9.69 | 1167.9218 | 52510.3125 |
+| 5000 | 0.8080 | 3607.3171 | 29425.125 | 5.1093 | 6.5019 | 76.9 | 9.689 | 1167.9218 | 52510.3125 |
+| 5500 | 0.8888 | 3607.3171 | 29425.125 | 5.1093 | 6.5496 | 76.34 | 9.619 | 1167.9218 | 52510.3125 |
+| 6000 | 0.9696 | 3607.3171 | 29425.125 | 5.1093 | 6.5045 | 76.87 | 9.686 | 1167.9218 | 52510.3125 |
+| 6188 | 1.0 | 3607.3171 | 29425.125 | 5.1093 | 6.526 | 76.617 | 9.654 | 1167.9218 | 52510.3125 |
 
 ### Framework versions
 - Distily 0.2.0

logs/dropout=0.1, learning_rate=0.0004, weight_decay=0/events.out.tfevents.1723882095.5f530b1cf724

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53c80e7bcaff10a7407cb3134460f5773b389e1b6c5c091634a7a66676033c7d
+size 307