End of training

README.md

@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 399.2138
-- eval_frwikippl: 2966.5173
-- eval_zhwikippl: 10538.9238
-- eval_loss: 1988.1440
-- eval_runtime: 21.3924
-- eval_samples_per_second: 46.746
-- eval_steps_per_second: 11.686
+- eval_enwikippl: 527.0228
+- eval_frwikippl: 3796.0032
+- eval_zhwikippl: 4795.4683
+- eval_loss: 2376.6721
+- eval_runtime: 21.817
+- eval_samples_per_second: 45.836
+- eval_steps_per_second: 11.459
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- distillation_objective: LinearObjective(logits_weight=1, logits_loss_fn=<function kl_divergence_loss at 0x7f57c4b07910>, activations_weight=10, activations_loss_fn=<function mse_loss at 0x7f57c4b07880>, attentions_weight=0, attentions_loss_fn=<function mse_loss at 0x7f57c4b07880>)
+- distillation_objective: LinearObjective(logits_weight=1, logits_loss_fn=<function kl_divergence_loss at 0x7f57c4b07910>, activations_weight=1, activations_loss_fn=<function kl_divergence_loss at 0x7f57c4b07910>, attentions_weight=0, attentions_loss_fn=<function mse_loss at 0x7f57c4b07880>)
 - train_embeddings: True
 - learning_rate: 4e-05
 - train_batch_size: 4
@@ -64,20 +64,20 @@ Peak GPU Memory: 4.5067 GB
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 30.2385 | 57.2728 |  |  |  |  | 18.1772 |
-| 0 | 0 | 55339.3672 | 57682.5742 | 6775.2959 | 21.1575 | 47.265 | 11.816 | 57080.2930 |
-| 500 | 0.0808 | 1023.4225 | 6192.8911 | 2836.3201 | 21.7865 | 45.9 | 11.475 | 48060.6055 |
-| 1000 | 0.1616 | 775.0658 | 5052.5142 | 2521.5359 | 21.2571 | 47.043 | 11.761 | 45170.0781 |
-| 1500 | 0.2424 | 671.2710 | 4404.4277 | 2386.8799 | 21.4196 | 46.686 | 11.672 | 34499.8906 |
-| 2000 | 0.3232 | 614.9977 | 4029.4568 | 2301.9199 | 21.8274 | 45.814 | 11.453 | 30409.7441 |
-| 2500 | 0.4040 | 569.0704 | 4036.1411 | 2240.4800 | 21.4135 | 46.7 | 11.675 | 165707.2031 |
-| 3000 | 0.4848 | 535.3444 | 3657.6721 | 2192.7681 | 21.3797 | 46.773 | 11.693 | 11745.1631 |
-| 3500 | 0.5657 | 500.6540 | 3441.3875 | 2149.4719 | 21.2691 | 47.017 | 11.754 | 9315015.0 |
-| 4000 | 0.6465 | 482.9359 | 3418.8945 | 2116.2720 | 21.4203 | 46.685 | 11.671 | 35355.7070 |
-| 4500 | 0.7273 | 458.2032 | 3282.2515 | 2079.2319 | 21.277 | 46.999 | 11.75 | 203786.2188 |
-| 5000 | 0.8081 | 433.2373 | 3179.2988 | 2048.7361 | 21.2654 | 47.025 | 11.756 | 9217.1963 |
-| 5500 | 0.8889 | 419.5392 | 3143.1909 | 2024.0320 | 21.3022 | 46.944 | 11.736 | 7640.9155 |
-| 6000 | 0.9697 | 402.3651 | 3010.7686 | 1996.6240 | 21.3926 | 46.745 | 11.686 | 5998.7275 |
-| 6187 | 0.9999 | 399.2138 | 2966.5173 | 1988.1440 | 21.3924 | 46.746 | 11.686 | 10538.9238 |
+| 0 | 0 | 55339.3672 | 57682.5742 | 31197.1836 | 21.7082 | 46.065 | 11.516 | 57080.2930 |
+| 500 | 0.0808 | 1509.5735 | 7497.0439 | 3194.9919 | 21.4587 | 46.601 | 11.65 | 50589.3438 |
+| 1000 | 0.1616 | 1083.2607 | 5620.3037 | 2923.7439 | 21.5879 | 46.322 | 11.581 | 29616.2285 |
+| 1500 | 0.2424 | 906.6083 | 4937.0078 | 2796.2080 | 21.6636 | 46.16 | 11.54 | 21403.5996 |
+| 2000 | 0.3232 | 813.4678 | 4877.3267 | 2706.0481 | 21.5303 | 46.446 | 11.612 | 20010.4863 |
+| 2500 | 0.4040 | 750.0352 | 4512.8765 | 2636.6079 | 21.6059 | 46.284 | 11.571 | 16546.3457 |
+| 3000 | 0.4848 | 704.7218 | 4373.6377 | 2583.7920 | 21.6069 | 46.281 | 11.57 | 14758.0859 |
+| 3500 | 0.5657 | 667.2821 | 4153.7866 | 2537.5520 | 21.59 | 46.318 | 11.579 | 14131.2881 |
+| 4000 | 0.6465 | 635.3494 | 4060.9749 | 2505.6001 | 21.554 | 46.395 | 11.599 | 13081.5996 |
+| 4500 | 0.7273 | 605.6495 | 4037.2766 | 2468.9121 | 21.795 | 45.882 | 11.471 | 11453.9658 |
+| 5000 | 0.8081 | 573.4954 | 3881.2524 | 2437.7439 | 21.6801 | 46.125 | 11.531 | 8931.2441 |
+| 5500 | 0.8889 | 557.2740 | 3918.3730 | 2413.4880 | 21.5054 | 46.5 | 11.625 | 6643.0454 |
+| 6000 | 0.9697 | 549.7523 | 4035.1443 | 2392.2400 | 21.6194 | 46.255 | 11.564 | 5330.4404 |
+| 6187 | 0.9999 | 527.0228 | 3796.0032 | 2376.6721 | 21.817 | 45.836 | 11.459 | 4795.4683 |
 
 ### Framework versions
 - Distily 0.2.0

logs/distillation_objective=LinearObjective_logits_weight_1__logits_loss_fn__function_kl_divergence_loss_at_0x7f57c4b07910___activations_weight_1__activations_loss_fn__function_kl_divergence_loss_at_0x7f57/events.out.tfevents.1723370868.715f24f8d8b8

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8fe78fb2bbf0f8be25cbc66294910303c91fe0392802f1f8835e49ca72269e3a
+size 249