End of training

README.md

@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 400.4869
-- eval_frwikippl: 2969.3420
-- eval_zhwikippl: 6008.7476
-- eval_loss: 1989.5520
-- eval_runtime: 21.3179
-- eval_samples_per_second: 46.909
-- eval_steps_per_second: 11.727
+- eval_enwikippl: 399.2138
+- eval_frwikippl: 2966.5173
+- eval_zhwikippl: 10538.9238
+- eval_loss: 1988.1440
+- eval_runtime: 21.3924
+- eval_samples_per_second: 46.746
+- eval_steps_per_second: 11.686
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- distillation_objective: LinearObjective(logits_weight=1, logits_loss_fn=<function kl_divergence_loss at 0x7f57c4b07910>, activations_weight=1, activations_loss_fn=<function mse_loss at 0x7f57c4b07880>, attentions_weight=0, attentions_loss_fn=<function mse_loss at 0x7f57c4b07880>)
+- distillation_objective: LinearObjective(logits_weight=1, logits_loss_fn=<function kl_divergence_loss at 0x7f57c4b07910>, activations_weight=10, activations_loss_fn=<function mse_loss at 0x7f57c4b07880>, attentions_weight=0, attentions_loss_fn=<function mse_loss at 0x7f57c4b07880>)
 - train_embeddings: True
 - learning_rate: 4e-05
 - train_batch_size: 4
@@ -64,20 +64,20 @@ Peak GPU Memory: 4.5067 GB
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 30.2385 | 57.2728 |  |  |  |  | 18.1772 |
-| 0 | 0 | 55339.3672 | 57682.5742 | 6775.2959 | 21.2939 | 46.962 | 11.74 | 57080.2930 |
-| 500 | 0.0808 | 1019.9713 | 6165.6597 | 2829.9199 | 21.7268 | 46.026 | 11.507 | 49348.3086 |
-| 1000 | 0.1616 | 776.1801 | 5053.2275 | 2518.8479 | 21.7049 | 46.072 | 11.518 | 43976.7227 |
-| 1500 | 0.2424 | 668.2543 | 4423.4121 | 2381.3120 | 21.5969 | 46.303 | 11.576 | 45161.0352 |
-| 2000 | 0.3232 | 617.0906 | 4036.9917 | 2304.1279 | 21.5749 | 46.35 | 11.588 | 40062.9219 |
-| 2500 | 0.4040 | 572.6278 | 4018.1096 | 2242.3999 | 21.5368 | 46.432 | 11.608 | 24775.3672 |
-| 3000 | 0.4848 | 534.8770 | 3649.6860 | 2192.1919 | 21.7099 | 46.062 | 11.515 | 135085.7969 |
-| 3500 | 0.5657 | 504.0086 | 3549.8198 | 2150.6721 | 21.6613 | 46.165 | 11.541 | 46912.9102 |
-| 4000 | 0.6465 | 483.1796 | 3357.9766 | 2115.4399 | 21.688 | 46.108 | 11.527 | 24034.0039 |
-| 4500 | 0.7273 | 458.0607 | 3258.0415 | 2080.0 | 21.5498 | 46.404 | 11.601 | 44794.6328 |
-| 5000 | 0.8081 | 432.8254 | 3191.2034 | 2050.5439 | 21.3113 | 46.923 | 11.731 | 9512.9375 |
-| 5500 | 0.8889 | 422.7440 | 3201.5676 | 2027.1520 | 21.5677 | 46.366 | 11.591 | 14696.1348 |
-| 6000 | 0.9697 | 406.4080 | 3030.5735 | 2000.3840 | 21.1806 | 47.213 | 11.803 | 3418.1089 |
-| 6187 | 0.9999 | 400.4869 | 2969.3420 | 1989.5520 | 21.3179 | 46.909 | 11.727 | 6008.7476 |
+| 0 | 0 | 55339.3672 | 57682.5742 | 6775.2959 | 21.1575 | 47.265 | 11.816 | 57080.2930 |
+| 500 | 0.0808 | 1023.4225 | 6192.8911 | 2836.3201 | 21.7865 | 45.9 | 11.475 | 48060.6055 |
+| 1000 | 0.1616 | 775.0658 | 5052.5142 | 2521.5359 | 21.2571 | 47.043 | 11.761 | 45170.0781 |
+| 1500 | 0.2424 | 671.2710 | 4404.4277 | 2386.8799 | 21.4196 | 46.686 | 11.672 | 34499.8906 |
+| 2000 | 0.3232 | 614.9977 | 4029.4568 | 2301.9199 | 21.8274 | 45.814 | 11.453 | 30409.7441 |
+| 2500 | 0.4040 | 569.0704 | 4036.1411 | 2240.4800 | 21.4135 | 46.7 | 11.675 | 165707.2031 |
+| 3000 | 0.4848 | 535.3444 | 3657.6721 | 2192.7681 | 21.3797 | 46.773 | 11.693 | 11745.1631 |
+| 3500 | 0.5657 | 500.6540 | 3441.3875 | 2149.4719 | 21.2691 | 47.017 | 11.754 | 9315015.0 |
+| 4000 | 0.6465 | 482.9359 | 3418.8945 | 2116.2720 | 21.4203 | 46.685 | 11.671 | 35355.7070 |
+| 4500 | 0.7273 | 458.2032 | 3282.2515 | 2079.2319 | 21.277 | 46.999 | 11.75 | 203786.2188 |
+| 5000 | 0.8081 | 433.2373 | 3179.2988 | 2048.7361 | 21.2654 | 47.025 | 11.756 | 9217.1963 |
+| 5500 | 0.8889 | 419.5392 | 3143.1909 | 2024.0320 | 21.3022 | 46.944 | 11.736 | 7640.9155 |
+| 6000 | 0.9697 | 402.3651 | 3010.7686 | 1996.6240 | 21.3926 | 46.745 | 11.686 | 5998.7275 |
+| 6187 | 0.9999 | 399.2138 | 2966.5173 | 1988.1440 | 21.3924 | 46.746 | 11.686 | 10538.9238 |
 
 ### Framework versions
 - Distily 0.2.0

logs/distillation_objective=LinearObjective_logits_weight_1__logits_loss_fn__function_kl_divergence_loss_at_0x7f57c4b07910___activations_weight_10__activations_loss_fn__function_mse_loss_at_0x7f57c4b07880_/events.out.tfevents.1723366555.715f24f8d8b8

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f04f1f6a04dd27bd89e7d9e9c6f402967aab8194099b7351068dbd4412d6b942
+size 249