End of training

README.md

@@ -44,36 +44,36 @@ More information needed
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
 | **teacher eval** |  | 43.25 | 61.25 |  |  |  |  | 11.6875 | 19.125 |
-| 0 | 0 | 1735166787584.0 | 99505802313728.0 | 22.0900 | 26.414 | 94.647 | 11.85 | 4009754624.0 | 26663156973568.0 |
-| 2500 | 0.0404 | 708.0 | 5376.0 | 2.6130 | 26.4416 | 94.548 | 11.837 | 452.0 | 5792.0 |
-| 5000 | 0.0808 | 322.0 | 1352.0 | 1.9433 | 26.3874 | 94.742 | 11.862 | 258.0 | 207.0 |
-| 7500 | 0.1212 | 217.0 | 1136.0 | 1.6459 | 26.452 | 94.511 | 11.833 | 177.0 | 161.0 |
-| 10000 | 0.1616 | 169.0 | 648.0 | 1.4588 | 26.4487 | 94.522 | 11.834 | 145.0 | 163.0 |
-| 12500 | 0.2020 | 124.5 | 486.0 | 1.2095 | 26.4377 | 94.562 | 11.839 | 93.0 | 122.5 |
-| 15000 | 0.2424 | 105.5 | 406.0 | 1.0801 | 26.3618 | 94.834 | 11.873 | 84.5 | 129.0 |
-| 17500 | 0.2828 | 92.0 | 320.0 | 0.9853 | 26.4007 | 94.694 | 11.856 | 70.0 | 132.0 |
-| 20000 | 0.3232 | 81.5 | 290.0 | 0.9114 | 26.4024 | 94.688 | 11.855 | 69.0 | 115.5 |
-| 22500 | 0.3636 | 73.0 | 235.0 | 0.7887 | 26.3819 | 94.762 | 11.864 | 59.25 | 82.0 |
-| 25000 | 0.4040 | 67.5 | 200.0 | 0.7309 | 26.439 | 94.557 | 11.839 | 51.0 | 80.5 |
-| 27500 | 0.4444 | 65.5 | 211.0 | 0.7037 | 26.4467 | 94.53 | 11.835 | 50.75 | 114.0 |
-| 30000 | 0.4848 | 70.0 | 210.0 | 0.7139 | 26.3698 | 94.805 | 11.87 | 49.25 | 86.0 |
-| 32500 | 0.5253 | 63.0 | 194.0 | 0.6770 | 26.4139 | 94.647 | 11.85 | 47.5 | 78.0 |
-| 35000 | 0.5657 | 60.5 | 190.0 | 0.6340 | 26.3558 | 94.856 | 11.876 | 43.75 | 80.0 |
-| 37500 | 0.6061 | 60.25 | 183.0 | 0.6123 | 26.4019 | 94.69 | 11.855 | 42.5 | 59.75 |
-| 40000 | 0.6465 | 58.5 | 168.0 | 0.5889 | 26.4324 | 94.581 | 11.842 | 41.0 | 89.5 |
-| 42500 | 0.6869 | 57.25 | 162.0 | 0.5763 | 26.3983 | 94.703 | 11.857 | 41.0 | 61.5 |
-| 45000 | 0.7273 | 53.75 | 145.0 | 0.4873 | 26.3416 | 94.907 | 11.882 | 35.0 | 46.25 |
-| 47500 | 0.7677 | 52.0 | 134.0 | 0.4673 | 26.3948 | 94.715 | 11.858 | 32.75 | 41.5 |
-| 50000 | 0.8081 | 50.5 | 132.0 | 0.4519 | 26.3589 | 94.845 | 11.875 | 32.75 | 39.0 |
-| 52500 | 0.8485 | 50.25 | 129.0 | 0.4436 | 26.3703 | 94.803 | 11.869 | 32.5 | 37.5 |
-| 55000 | 0.8889 | 49.75 | 125.5 | 0.4278 | 26.4348 | 94.572 | 11.84 | 31.375 | 34.25 |
-| 57500 | 0.9293 | 49.25 | 123.5 | 0.4231 | 26.4211 | 94.621 | 11.847 | 31.125 | 33.25 |
-| 60000 | 0.9697 | 49.25 | 123.0 | 0.4195 | 26.4149 | 94.644 | 11.849 | 31.0 | 33.5 |
-| 61875 | 1.0 | 49.25 | 123.0 | 0.4193 | 26.3909 | 94.73 | 11.86 | 31.0 | 33.5 |
+| 0 | 0 | 2473901162496.0 | 170424302305280.0 | 30.7740 | 26.7214 | 93.558 | 11.713 | 4060086272.0 | 71468255805440.0 |
+| 2500 | 0.0404 | 1200.0 | 12160.0 | 9.8335 | 26.6807 | 93.701 | 11.731 | 776.0 | 13952.0 |
+| 5000 | 0.0808 | 412.0 | 2240.0 | 8.3973 | 26.664 | 93.759 | 11.739 | 290.0 | 434.0 |
+| 7500 | 0.1212 | 245.0 | 908.0 | 7.6617 | 26.6719 | 93.732 | 11.735 | 218.0 | 198.0 |
+| 10000 | 0.1616 | 183.0 | 672.0 | 7.2411 | 26.7416 | 93.487 | 11.705 | 165.0 | 202.0 |
+| 12500 | 0.2020 | 132.0 | 504.0 | 6.6894 | 26.7138 | 93.585 | 11.717 | 115.0 | 154.0 |
+| 15000 | 0.2424 | 113.0 | 436.0 | 6.4125 | 26.7091 | 93.601 | 11.719 | 89.5 | 139.0 |
+| 17500 | 0.2828 | 92.5 | 340.0 | 6.1942 | 26.6603 | 93.772 | 11.74 | 71.0 | 131.0 |
+| 20000 | 0.3232 | 75.0 | 276.0 | 5.9314 | 26.7037 | 93.62 | 11.721 | 65.0 | 109.0 |
+| 22500 | 0.3636 | 65.5 | 215.0 | 5.6582 | 26.6905 | 93.666 | 11.727 | 50.75 | 82.0 |
+| 25000 | 0.4040 | 63.0 | 196.0 | 5.5585 | 26.6163 | 93.927 | 11.76 | 43.75 | 85.5 |
+| 27500 | 0.4444 | 58.5 | 203.0 | 5.4882 | 26.6291 | 93.882 | 11.754 | 41.0 | 68.5 |
+| 30000 | 0.4848 | 58.75 | 196.0 | 5.4760 | 26.6873 | 93.678 | 11.728 | 41.0 | 62.75 |
+| 32500 | 0.5253 | 58.5 | 174.0 | 5.4524 | 26.667 | 93.749 | 11.737 | 39.5 | 62.25 |
+| 35000 | 0.5657 | 57.0 | 168.0 | 5.3756 | 26.6906 | 93.666 | 11.727 | 36.5 | 49.5 |
+| 37500 | 0.6061 | 57.25 | 158.0 | 5.3384 | 26.6664 | 93.751 | 11.738 | 37.5 | 49.5 |
+| 40000 | 0.6465 | 55.25 | 155.0 | 5.3176 | 26.673 | 93.728 | 11.735 | 34.75 | 58.75 |
+| 42500 | 0.6869 | 54.75 | 150.0 | 5.2917 | 26.7045 | 93.617 | 11.721 | 35.25 | 53.0 |
+| 45000 | 0.7273 | 50.5 | 132.0 | 5.1569 | 26.7113 | 93.593 | 11.718 | 30.125 | 46.5 |
+| 47500 | 0.7677 | 50.5 | 124.5 | 5.1275 | 26.704 | 93.619 | 11.721 | 29.5 | 36.5 |
+| 50000 | 0.8081 | 50.0 | 122.5 | 5.1100 | 26.6758 | 93.718 | 11.733 | 29.125 | 38.0 |
+| 52500 | 0.8485 | 48.75 | 119.5 | 5.0953 | 26.6639 | 93.76 | 11.739 | 28.875 | 35.5 |
+| 55000 | 0.8889 | 48.5 | 117.5 | 5.0750 | 26.718 | 93.57 | 11.715 | 28.25 | 35.75 |
+| 57500 | 0.9293 | 48.0 | 117.0 | 5.0700 | 26.7088 | 93.602 | 11.719 | 28.0 | 33.25 |
+| 60000 | 0.9697 | 48.25 | 116.5 | 5.0649 | 26.711 | 93.594 | 11.718 | 27.875 | 33.25 |
+| 61875 | 1.0 | 48.25 | 116.5 | 5.0644 | 26.6823 | 93.695 | 11.731 | 27.875 | 33.25 |
 
 # Resource Usage Comparison
 
-- VRAM Use: 7.7823 GB
+- VRAM Use: 7.7851 GB
 
 # Distillation (Teacher -> Student) Architecture Difference:
 
@@ -93,7 +93,7 @@ More information needed
 <br/>
 
 # Train Dataset
-Trained on 145,705,155 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 145,744,973 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
 - Num Samples: `247,500`
 - Subset: `20231101.en`
@@ -103,7 +103,7 @@ Trained on 145,705,155 tokens from the [wikimedia/wikipedia](https://huggingface
 # Training Objective
 
 ```
-DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=raw_mse, layer_mapper=layer-2))
+DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=cos, layer_mapper=layer-2))
 ```
 
 # Hyperparameters
@@ -120,9 +120,9 @@ The following hyperparameters were used during training:
 - lr_scheduler_type: `linear`
 - lr_scheduler_warmup_ratio: `0.5`
 - num_epochs: `1.0`
-- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=raw_mse, layer_mapper=layer-2))`
+- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=cos, layer_mapper=layer-2))`
 - train_embeddings: `True`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fc5b5101cf0>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fc5f0404a60>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `None`
 - student_model_config: `None`

logs/attn_loss_fn=cos, attn_weight=10.0, projector=orthogonal/events.out.tfevents.1724295072.e3f806ea38c9

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0690b170c3ffc3d65712256639e5c276331ec6fce8bc4df8ead2f6f7c311aaa8
+size 588