End of training

README.md

@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 314.5309
-- eval_frwikippl: 2375.5381
-- eval_zhwikippl: 1510.4966
-- eval_loss: 1556.0640
-- eval_runtime: 21.4342
-- eval_samples_per_second: 46.654
-- eval_steps_per_second: 11.664
+- eval_enwikippl: 322.4071
+- eval_frwikippl: 2411.2278
+- eval_zhwikippl: 1802.7419
+- eval_loss: 1569.2800
+- eval_runtime: 21.3907
+- eval_samples_per_second: 46.749
+- eval_steps_per_second: 11.687
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- distillation_objective: LinearObjective(logits_weight=10, logits_loss_fn=<function kl_divergence_loss at 0x7f57c4b07910>, activations_weight=0, activations_loss_fn=<function mse_loss at 0x7f57c4b07880>, attentions_weight=0, attentions_loss_fn=<function mse_loss at 0x7f57c4b07880>)
+- distillation_objective: LinearObjective(logits_weight=100, logits_loss_fn=<function kl_divergence_loss at 0x7f57c4b07910>, activations_weight=0, activations_loss_fn=<function mse_loss at 0x7f57c4b07880>, attentions_weight=0, attentions_loss_fn=<function mse_loss at 0x7f57c4b07880>)
 - train_embeddings: True
 - learning_rate: 4e-05
 - train_batch_size: 4
@@ -58,26 +58,26 @@ The following hyperparameters were used during training:
 - num_epochs: 1.0
 
 ### Resource Usage
-Peak GPU Memory: 4.4311 GB
+Peak GPU Memory: 4.4338 GB
 
 ### Eval-Phase Metrics
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 30.2385 | 57.2728 |  |  |  |  | 18.1772 |
-| 0 | 0 | 55339.3672 | 57682.5742 | 6157.4399 | 21.4039 | 46.72 | 11.68 | 57080.2930 |
-| 500 | 0.0808 | 1008.8255 | 6131.8423 | 2271.7280 | 21.5213 | 46.466 | 11.616 | 44256.5508 |
-| 1000 | 0.1616 | 752.4122 | 4871.3159 | 2171.8879 | 21.5436 | 46.417 | 11.604 | 23939.5059 |
-| 1500 | 0.2424 | 635.3494 | 4207.1333 | 2004.7280 | 21.4673 | 46.582 | 11.646 | 16136.1289 |
-| 2000 | 0.3232 | 567.7023 | 3844.3479 | 1924.7280 | 21.4197 | 46.686 | 11.671 | 11753.7910 |
-| 2500 | 0.4040 | 517.8143 | 3658.1885 | 1864.2720 | 21.4756 | 46.564 | 11.641 | 7620.0234 |
-| 3000 | 0.4848 | 474.2581 | 3358.3323 | 1809.6720 | 21.4488 | 46.623 | 11.656 | 5299.9204 |
-| 3500 | 0.5657 | 430.1449 | 3037.0979 | 1761.0240 | 21.3865 | 46.759 | 11.69 | 4430.9653 |
-| 4000 | 0.6465 | 409.4490 | 2945.9871 | 1716.2080 | 21.4303 | 46.663 | 11.666 | 3593.3940 |
-| 4500 | 0.7273 | 376.3208 | 2802.5376 | 1672.4399 | 21.4086 | 46.71 | 11.678 | 3427.7061 |
-| 5000 | 0.8081 | 348.5558 | 2647.9026 | 1634.9520 | 21.3878 | 46.756 | 11.689 | 2210.5042 |
-| 5500 | 0.8889 | 337.0898 | 2634.4019 | 1601.7679 | 21.3816 | 46.769 | 11.692 | 1503.8546 |
-| 6000 | 0.9697 | 312.6012 | 2455.6636 | 1567.5601 | 21.565 | 46.371 | 11.593 | 1565.2198 |
-| 6187 | 0.9999 | 314.5309 | 2375.5381 | 1556.0640 | 21.4342 | 46.654 | 11.664 | 1510.4966 |
+| 0 | 0 | 55339.3672 | 57682.5742 | 6157.4399 | 21.587 | 46.324 | 11.581 | 57080.2930 |
+| 500 | 0.0808 | 1033.0840 | 6283.9331 | 2279.6641 | 21.6395 | 46.212 | 11.553 | 52104.6055 |
+| 1000 | 0.1616 | 765.2576 | 4892.3081 | 2097.8081 | 21.3235 | 46.897 | 11.724 | 26096.3535 |
+| 1500 | 0.2424 | 644.3925 | 4229.7388 | 2007.1281 | 21.672 | 46.142 | 11.536 | 19367.6953 |
+| 2000 | 0.3232 | 572.4389 | 3848.9590 | 1927.5200 | 21.3193 | 46.906 | 11.726 | 13467.1152 |
+| 2500 | 0.4040 | 522.0941 | 3661.2842 | 1866.1200 | 21.356 | 46.825 | 11.706 | 8405.3037 |
+| 3000 | 0.4848 | 479.3863 | 3402.9036 | 1813.2240 | 21.551 | 46.402 | 11.6 | 6118.0571 |
+| 3500 | 0.5657 | 438.9093 | 3085.1194 | 1765.5601 | 21.4458 | 46.629 | 11.657 | 5073.1313 |
+| 4000 | 0.6465 | 414.7451 | 2969.3420 | 1719.8400 | 21.3815 | 46.769 | 11.692 | 4076.1655 |
+| 4500 | 0.7273 | 384.2867 | 2787.7578 | 1679.6400 | 21.4967 | 46.519 | 11.63 | 3883.0295 |
+| 5000 | 0.8081 | 357.7559 | 2647.6235 | 1643.3920 | 21.5199 | 46.469 | 11.617 | 2635.5439 |
+| 5500 | 0.8889 | 344.0447 | 2642.6804 | 1612.0720 | 21.5612 | 46.38 | 11.595 | 1745.4163 |
+| 6000 | 0.9697 | 322.2757 | 2468.9441 | 1580.7920 | 21.3421 | 46.856 | 11.714 | 1794.2168 |
+| 6187 | 0.9999 | 322.4071 | 2411.2278 | 1569.2800 | 21.3907 | 46.749 | 11.687 | 1802.7419 |
 
 ### Framework versions
 - Distily 0.2.0

logs/distillation_objective=LinearObjective_logits_weight_100__logits_loss_fn__function_kl_divergence_loss_at_0x7f57c4b07910___activations_weight_0__activations_loss_fn__function_mse_loss_at_0x7f57c4b07880/events.out.tfevents.1723357975.715f24f8d8b8

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:813a2e1bc91e592c305fdb9169c7e9cffdc4630328377918412dad3bd0553bb3
+size 249