Hugging Face's logo

distily
/
distily_projector_experiment

TensorBoard
Safetensors
Distily
gpt2
bitnet
1.58b
Generated from Trainer
Model card Files Metrics Community
lapp0 commited on
Commit
4ddd051
·
verified ·
1 Parent(s): 10d5f82

End of training

Browse files
Files changed (2) hide show
  1. README.md +29 -29
  2. logs/attn_loss_fn=cos, attn_weight=10.0, projector=linear/events.out.tfevents.1724311707.e3f806ea38c9 +3 -0
README.md CHANGED
@@ -44,32 +44,32 @@ More information needed
44
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
45
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
46
  | **teacher eval** | | 43.25 | 61.25 | | | | | 11.6875 | 19.125 |
47
- | 0 | 0 | 2473901162496.0 | 170424302305280.0 | 22.0554 | 26.3026 | 95.048 | 11.9 | 4060086272.0 | 71468255805440.0 |
48
- | 2500 | 0.0404 | 744.0 | 5792.0 | 2.6067 | 26.3957 | 94.712 | 11.858 | 450.0 | 4160.0 |
49
- | 5000 | 0.0808 | 316.0 | 1408.0 | 1.9089 | 26.4305 | 94.588 | 11.842 | 236.0 | 300.0 |
50
- | 7500 | 0.1212 | 221.0 | 776.0 | 1.6174 | 26.4065 | 94.674 | 11.853 | 180.0 | 176.0 |
51
- | 10000 | 0.1616 | 165.0 | 620.0 | 1.4274 | 26.4063 | 94.674 | 11.853 | 144.0 | 149.0 |
52
- | 12500 | 0.2020 | 124.5 | 484.0 | 1.1896 | 26.4269 | 94.6 | 11.844 | 107.5 | 145.0 |
53
- | 15000 | 0.2424 | 108.0 | 452.0 | 1.0631 | 26.4405 | 94.552 | 11.838 | 92.5 | 110.0 |
54
- | 17500 | 0.2828 | 91.5 | 354.0 | 0.9650 | 26.417 | 94.636 | 11.848 | 75.5 | 119.0 |
55
- | 20000 | 0.3232 | 80.5 | 306.0 | 0.8889 | 26.4341 | 94.575 | 11.841 | 70.0 | 225.0 |
56
- | 22500 | 0.3636 | 73.0 | 233.0 | 0.7611 | 26.3815 | 94.763 | 11.864 | 58.0 | 83.5 |
57
- | 25000 | 0.4040 | 67.0 | 201.0 | 0.7173 | 26.4268 | 94.601 | 11.844 | 52.25 | 95.5 |
58
- | 27500 | 0.4444 | 64.0 | 212.0 | 0.6823 | 26.3976 | 94.705 | 11.857 | 49.0 | 141.0 |
59
- | 30000 | 0.4848 | 64.5 | 201.0 | 0.6783 | 26.337 | 94.924 | 11.884 | 51.5 | 86.0 |
60
- | 32500 | 0.5253 | 64.5 | 190.0 | 0.6657 | 26.3562 | 94.854 | 11.876 | 47.0 | 68.5 |
61
- | 35000 | 0.5657 | 60.75 | 179.0 | 0.6094 | 26.3895 | 94.735 | 11.861 | 42.5 | 59.25 |
62
- | 37500 | 0.6061 | 61.0 | 165.0 | 0.5963 | 26.4159 | 94.64 | 11.849 | 44.5 | 74.5 |
63
- | 40000 | 0.6465 | 58.75 | 171.0 | 0.5823 | 26.4219 | 94.619 | 11.846 | 39.0 | 93.0 |
64
- | 42500 | 0.6869 | 56.75 | 167.0 | 0.5589 | 26.4356 | 94.57 | 11.84 | 40.5 | 60.25 |
65
- | 45000 | 0.7273 | 52.75 | 140.0 | 0.4717 | 26.4084 | 94.667 | 11.852 | 34.25 | 47.75 |
66
- | 47500 | 0.7677 | 53.0 | 132.0 | 0.4510 | 26.4148 | 94.644 | 11.849 | 33.25 | 38.75 |
67
- | 50000 | 0.8081 | 50.75 | 133.0 | 0.4386 | 26.348 | 94.884 | 11.879 | 32.5 | 40.25 |
68
- | 52500 | 0.8485 | 50.0 | 129.0 | 0.4270 | 26.4189 | 94.629 | 11.848 | 32.25 | 38.25 |
69
- | 55000 | 0.8889 | 49.75 | 125.0 | 0.4124 | 26.4696 | 94.448 | 11.825 | 31.125 | 35.5 |
70
- | 57500 | 0.9293 | 48.75 | 125.5 | 0.4083 | 26.4221 | 94.618 | 11.846 | 30.625 | 33.5 |
71
- | 60000 | 0.9697 | 48.75 | 124.5 | 0.4048 | 26.3938 | 94.719 | 11.859 | 30.5 | 33.75 |
72
- | 61875 | 1.0 | 48.75 | 124.5 | 0.4044 | 26.3794 | 94.771 | 11.865 | 30.5 | 33.75 |
73
 
74
  # Resource Usage Comparison
75
 
@@ -103,7 +103,7 @@ Trained on 145,744,973 tokens from the [wikimedia/wikipedia](https://huggingface
103
  # Training Objective
104
 
105
  ```
106
- DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=raw_mse, layer_mapper=layer-2))
107
  ```
108
 
109
  # Hyperparameters
@@ -120,9 +120,9 @@ The following hyperparameters were used during training:
120
  - lr_scheduler_type: `linear`
121
  - lr_scheduler_warmup_ratio: `0.5`
122
  - num_epochs: `1.0`
123
- - distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=raw_mse, layer_mapper=layer-2))`
124
  - train_embeddings: `True`
125
- - lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fc55d09f820>`
126
  - student_model_name_or_path: `None`
127
  - student_config_name_or_path: `None`
128
  - student_model_config: `None`
 
44
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
45
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
46
  | **teacher eval** | | 43.25 | 61.25 | | | | | 11.6875 | 19.125 |
47
+ | 0 | 0 | 2473901162496.0 | 170424302305280.0 | 30.7740 | 26.7323 | 93.52 | 11.709 | 4060086272.0 | 71468255805440.0 |
48
+ | 2500 | 0.0404 | 1208.0 | 12224.0 | 9.8371 | 26.6698 | 93.739 | 11.736 | 776.0 | 13952.0 |
49
+ | 5000 | 0.0808 | 412.0 | 2240.0 | 8.3982 | 26.694 | 93.654 | 11.725 | 290.0 | 436.0 |
50
+ | 7500 | 0.1212 | 245.0 | 908.0 | 7.6617 | 26.6969 | 93.644 | 11.724 | 218.0 | 198.0 |
51
+ | 10000 | 0.1616 | 183.0 | 672.0 | 7.2408 | 26.7324 | 93.519 | 11.709 | 165.0 | 201.0 |
52
+ | 12500 | 0.2020 | 132.0 | 504.0 | 6.6895 | 26.7254 | 93.544 | 11.712 | 114.5 | 154.0 |
53
+ | 15000 | 0.2424 | 113.0 | 434.0 | 6.4134 | 26.6959 | 93.647 | 11.725 | 90.0 | 137.0 |
54
+ | 17500 | 0.2828 | 93.0 | 340.0 | 6.1940 | 26.7239 | 93.549 | 11.712 | 71.0 | 133.0 |
55
+ | 20000 | 0.3232 | 74.5 | 276.0 | 5.9286 | 26.7267 | 93.54 | 11.711 | 64.5 | 123.0 |
56
+ | 22500 | 0.3636 | 67.0 | 214.0 | 5.6598 | 26.7384 | 93.498 | 11.706 | 50.25 | 75.5 |
57
+ | 25000 | 0.4040 | 64.0 | 194.0 | 5.5584 | 26.6783 | 93.709 | 11.732 | 44.25 | 78.0 |
58
+ | 27500 | 0.4444 | 57.75 | 190.0 | 5.4847 | 26.7024 | 93.625 | 11.722 | 41.25 | 64.5 |
59
+ | 30000 | 0.4848 | 59.75 | 196.0 | 5.4728 | 26.6702 | 93.738 | 11.736 | 42.25 | 69.5 |
60
+ | 32500 | 0.5253 | 58.5 | 174.0 | 5.4507 | 26.712 | 93.591 | 11.718 | 40.5 | 66.0 |
61
+ | 35000 | 0.5657 | 57.0 | 172.0 | 5.3752 | 26.682 | 93.696 | 11.731 | 36.5 | 48.0 |
62
+ | 37500 | 0.6061 | 57.75 | 155.0 | 5.3424 | 26.7128 | 93.588 | 11.717 | 38.25 | 53.25 |
63
+ | 40000 | 0.6465 | 55.0 | 156.0 | 5.3208 | 26.7055 | 93.614 | 11.72 | 35.0 | 65.5 |
64
+ | 42500 | 0.6869 | 54.25 | 149.0 | 5.2910 | 26.6805 | 93.701 | 11.731 | 36.0 | 57.0 |
65
+ | 45000 | 0.7273 | 50.25 | 132.0 | 5.1566 | 26.7217 | 93.557 | 11.713 | 30.375 | 50.75 |
66
+ | 47500 | 0.7677 | 50.5 | 124.0 | 5.1251 | 26.7003 | 93.632 | 11.723 | 29.75 | 38.25 |
67
+ | 50000 | 0.8081 | 49.75 | 123.5 | 5.1102 | 26.6997 | 93.634 | 11.723 | 29.0 | 40.0 |
68
+ | 52500 | 0.8485 | 48.75 | 119.5 | 5.0954 | 26.6849 | 93.686 | 11.729 | 29.0 | 35.75 |
69
+ | 55000 | 0.8889 | 48.5 | 117.0 | 5.0744 | 26.7257 | 93.543 | 11.712 | 28.375 | 35.5 |
70
+ | 57500 | 0.9293 | 48.0 | 116.5 | 5.0686 | 26.7133 | 93.586 | 11.717 | 28.125 | 33.25 |
71
+ | 60000 | 0.9697 | 48.0 | 116.5 | 5.0646 | 26.6953 | 93.649 | 11.725 | 28.0 | 33.25 |
72
+ | 61875 | 1.0 | 48.0 | 117.0 | 5.0644 | 26.695 | 93.65 | 11.725 | 28.0 | 33.5 |
73
 
74
  # Resource Usage Comparison
75
 
 
103
  # Training Objective
104
 
105
  ```
106
+ DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=cos, layer_mapper=layer-2))
107
  ```
108
 
109
  # Hyperparameters
 
120
  - lr_scheduler_type: `linear`
121
  - lr_scheduler_warmup_ratio: `0.5`
122
  - num_epochs: `1.0`
123
+ - distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=cos, layer_mapper=layer-2))`
124
  - train_embeddings: `True`
125
+ - lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fc57c38a620>`
126
  - student_model_name_or_path: `None`
127
  - student_config_name_or_path: `None`
128
  - student_model_config: `None`
logs/attn_loss_fn=cos, attn_weight=10.0, projector=linear/events.out.tfevents.1724311707.e3f806ea38c9 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:68227e38d75f4947a67ffa07ab5d44dba79b7517983d083609cce2184f793425
3
+ size 588