Hugging Face's logo

distily
/
short_gpt2

TensorBoard
Safetensors
Distily
gpt2
Generated from Trainer
Model card Files Metrics Community
lapp0 commited on
Commit
23f827c
·
verified ·
1 Parent(s): 9f4559e

End of training

Browse files
Files changed (2) hide show
  1. README.md +210 -140
  2. logs/dataset_sample_size=1000000, lr_scheduler_type=cosine, warmup_ratio=0.5/events.out.tfevents.1724299704.f383272e719b +3 -0
README.md CHANGED
@@ -1,5 +1,7 @@
1
  ---
2
  base_model: distilbert/distilgpt2
 
 
3
  library_name: Distily
4
  license: apache-2.0
5
  tags:
@@ -9,163 +11,231 @@ model-index:
9
  results: []
10
  ---
11
 
12
- # short_gpt2
13
 
14
- This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).
15
 
16
- The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
17
-
18
- It achieves the following results on the evaluation set:
19
- - eval_enwikippl: 65.0
20
- - eval_frwikippl: 215.0
21
- - eval_zhwikippl: 104.5
22
- - eval_tinystoriesppl: 49.75
23
- - eval_loss: 0.4281
24
- - eval_runtime: 102.0824
25
- - eval_samples_per_second: 97.96
26
- - eval_steps_per_second: 12.245
27
 
28
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
29
  should probably proofread and complete it, then remove this comment.
30
 
31
- ## Model description
32
-
33
- More information needed
34
-
35
- ## Intended uses & limitations
36
 
37
  More information needed
38
 
39
- ## Training and evaluation data
40
 
41
  More information needed
42
  -->
43
 
44
- ## Training procedure
 
 
 
 
45
 
46
- ### Training hyperparameters
47
 
48
- The following hyperparameters were used during training:
49
- - distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=mse, layer_mapper=last, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=mse, layer_mapper=layer-2, projector=None))
50
- - train_embeddings: True
51
- - learning_rate: 0.0001
52
- - train_batch_size: 4
53
- - eval_batch_size: 8
54
- - seed: 42
55
- - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
56
- - lr_scheduler_type: constant
57
- - lr_scheduler_warmup_ratio: 0.2
58
- - num_epochs: 1.0
59
-
60
- ### Resource Usage
61
- Peak GPU Memory: 7.2012 GB
62
-
63
- ### Eval-Phase Metrics
64
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
65
- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
66
  | **teacher eval** | | 43.25 | 61.25 | | | | | 11.6875 | 19.125 |
67
- | 0 | 0 | 2611340115968.0 | 307863255777280.0 | 21.3930 | 101.7301 | 98.299 | 12.287 | 7214202880.0 | 36009005809664.0 |
68
- | 2500 | 0.0101 | 191.0 | 704.0 | 1.0465 | 101.8446 | 98.189 | 12.274 | 165.0 | 316.0 |
69
- | 5000 | 0.0202 | 131.0 | 492.0 | 0.8689 | 102.0266 | 98.014 | 12.252 | 111.5 | 148.0 |
70
- | 7500 | 0.0303 | 114.5 | 396.0 | 0.7492 | 101.8482 | 98.185 | 12.273 | 92.0 | 142.0 |
71
- | 10000 | 0.0404 | 97.5 | 380.0 | 0.6631 | 102.1266 | 97.918 | 12.24 | 76.5 | 136.0 |
72
- | 12500 | 0.0505 | 86.5 | 314.0 | 0.5968 | 101.9077 | 98.128 | 12.266 | 72.0 | 146.0 |
73
- | 15000 | 0.0606 | 80.5 | 302.0 | 0.5429 | 102.0007 | 98.039 | 12.255 | 66.0 | 135.0 |
74
- | 17500 | 0.0707 | 77.0 | 276.0 | 0.5161 | 101.9305 | 98.106 | 12.263 | 62.75 | 121.0 |
75
- | 20000 | 0.0808 | 74.5 | 262.0 | 0.5019 | 102.035 | 98.006 | 12.251 | 60.25 | 120.5 |
76
- | 22500 | 0.0909 | 68.5 | 266.0 | 0.4874 | 101.8648 | 98.169 | 12.271 | 59.0 | 160.0 |
77
- | 25000 | 0.1010 | 73.0 | 242.0 | 0.4754 | 102.1293 | 97.915 | 12.239 | 55.25 | 145.0 |
78
- | 27500 | 0.1111 | 70.0 | 243.0 | 0.4627 | 102.0199 | 98.02 | 12.253 | 56.75 | 100.0 |
79
- | 30000 | 0.1212 | 68.5 | 251.0 | 0.4621 | 102.0947 | 97.948 | 12.244 | 56.5 | 133.0 |
80
- | 32500 | 0.1313 | 68.5 | 252.0 | 0.4589 | 102.0148 | 98.025 | 12.253 | 52.0 | 139.0 |
81
- | 35000 | 0.1414 | 67.0 | 228.0 | 0.4628 | 101.8975 | 98.138 | 12.267 | 53.75 | 254.0 |
82
- | 37500 | 0.1515 | 68.5 | 227.0 | 0.4495 | 101.9205 | 98.116 | 12.264 | 53.0 | 130.0 |
83
- | 40000 | 0.1616 | 72.0 | 270.0 | 0.4502 | 101.8861 | 98.149 | 12.269 | 56.25 | 104.0 |
84
- | 42500 | 0.1717 | 68.0 | 238.0 | 0.4422 | 101.8871 | 98.148 | 12.268 | 54.25 | 167.0 |
85
- | 45000 | 0.1818 | 68.5 | 260.0 | 0.4498 | 101.9466 | 98.091 | 12.261 | 54.0 | 108.5 |
86
- | 47500 | 0.1919 | 69.0 | 229.0 | 0.4392 | 102.1911 | 97.856 | 12.232 | 49.25 | 113.0 |
87
- | 50000 | 0.2020 | 71.5 | 247.0 | 0.4473 | 104.3621 | 95.82 | 11.978 | 51.75 | 86.5 |
88
- | 52500 | 0.2121 | 72.5 | 233.0 | 0.4357 | 102.8199 | 97.257 | 12.157 | 53.25 | 147.0 |
89
- | 55000 | 0.2222 | 76.5 | 223.0 | 0.4321 | 102.1001 | 97.943 | 12.243 | 51.0 | 88.5 |
90
- | 57500 | 0.2323 | 75.5 | 238.0 | 0.4342 | 102.0258 | 98.014 | 12.252 | 54.75 | 115.0 |
91
- | 60000 | 0.2424 | 73.5 | 250.0 | 0.4374 | 101.9687 | 98.069 | 12.259 | 51.25 | 153.0 |
92
- | 62500 | 0.2525 | 67.0 | 225.0 | 0.4252 | 101.9203 | 98.116 | 12.264 | 50.25 | 145.0 |
93
- | 65000 | 0.2626 | 70.0 | 224.0 | 0.4304 | 101.9468 | 98.09 | 12.261 | 48.75 | 128.0 |
94
- | 67500 | 0.2727 | 67.5 | 208.0 | 0.4303 | 102.0608 | 97.981 | 12.248 | 55.25 | 166.0 |
95
- | 70000 | 0.2828 | 70.5 | 231.0 | 0.4263 | 101.9988 | 98.04 | 12.255 | 52.75 | 115.5 |
96
- | 72500 | 0.2929 | 65.5 | 230.0 | 0.4249 | 102.2665 | 97.784 | 12.223 | 54.25 | 128.0 |
97
- | 75000 | 0.3030 | 68.0 | 243.0 | 0.4279 | 102.0312 | 98.009 | 12.251 | 49.75 | 125.5 |
98
- | 77500 | 0.3131 | 67.5 | 222.0 | 0.4326 | 102.1256 | 97.919 | 12.24 | 52.0 | 121.5 |
99
- | 80000 | 0.3232 | 65.5 | 222.0 | 0.4254 | 101.9985 | 98.041 | 12.255 | 48.5 | 133.0 |
100
- | 82500 | 0.3333 | 68.0 | 230.0 | 0.4219 | 102.0083 | 98.031 | 12.254 | 52.0 | 111.5 |
101
- | 85000 | 0.3434 | 67.0 | 222.0 | 0.4243 | 102.057 | 97.984 | 12.248 | 48.25 | 109.5 |
102
- | 87500 | 0.3535 | 66.5 | 218.0 | 0.4240 | 101.9819 | 98.057 | 12.257 | 53.5 | 302.0 |
103
- | 90000 | 0.3636 | 66.5 | 229.0 | 0.4250 | 102.0841 | 97.958 | 12.245 | 50.0 | 118.0 |
104
- | 92500 | 0.3737 | 67.0 | 227.0 | 0.4239 | 102.0958 | 97.947 | 12.243 | 53.0 | 114.0 |
105
- | 95000 | 0.3838 | 67.5 | 240.0 | 0.4257 | 101.9889 | 98.05 | 12.256 | 50.75 | 110.0 |
106
- | 97500 | 0.3939 | 65.0 | 215.0 | 0.4281 | 102.0824 | 97.96 | 12.245 | 49.75 | 104.5 |
107
- | 100000 | 0.4040 | 67.5 | 230.0 | 0.4203 | 102.3463 | 97.707 | 12.213 | 50.5 | 115.0 |
108
- | 102500 | 0.4141 | 66.0 | 227.0 | 0.4239 | 102.8008 | 97.276 | 12.159 | 53.25 | 109.0 |
109
- | 105000 | 0.4242 | 66.5 | 219.0 | 0.4249 | 102.6156 | 97.451 | 12.181 | 51.75 | 159.0 |
110
- | 107500 | 0.4343 | 65.5 | 218.0 | 0.4209 | 102.6016 | 97.464 | 12.183 | 51.5 | 95.0 |
111
- | 110000 | 0.4444 | 66.5 | 227.0 | 0.4213 | 102.437 | 97.621 | 12.203 | 52.0 | 130.0 |
112
- | 112500 | 0.4545 | 67.5 | 211.0 | 0.4231 | 102.6961 | 97.375 | 12.172 | 49.5 | 145.0 |
113
- | 115000 | 0.4646 | 66.0 | 209.0 | 0.4215 | 102.1356 | 97.909 | 12.239 | 48.25 | 126.5 |
114
- | 117500 | 0.4747 | 66.5 | 228.0 | 0.4261 | 102.5136 | 97.548 | 12.194 | 48.25 | 104.0 |
115
- | 120000 | 0.4848 | 68.5 | 238.0 | 0.4239 | 102.325 | 97.728 | 12.216 | 50.5 | 212.0 |
116
- | 122500 | 0.4949 | 67.0 | 219.0 | 0.4203 | 102.8823 | 97.198 | 12.15 | 52.0 | 94.5 |
117
- | 125000 | 0.5051 | 66.5 | 249.0 | 0.4220 | 102.285 | 97.766 | 12.221 | 51.0 | 129.0 |
118
- | 127500 | 0.5152 | 65.0 | 226.0 | 0.4242 | 102.4487 | 97.61 | 12.201 | 49.0 | 76.5 |
119
- | 130000 | 0.5253 | 65.0 | 222.0 | 0.4206 | 102.615 | 97.452 | 12.181 | 51.5 | 106.0 |
120
- | 132500 | 0.5354 | 63.5 | 232.0 | 0.4195 | 102.0382 | 98.002 | 12.25 | 49.0 | 115.0 |
121
- | 135000 | 0.5455 | 65.0 | 239.0 | 0.4195 | 102.4661 | 97.593 | 12.199 | 50.75 | 83.5 |
122
- | 137500 | 0.5556 | 69.0 | 232.0 | 0.4227 | 102.0828 | 97.96 | 12.245 | 52.25 | 133.0 |
123
- | 140000 | 0.5657 | 66.0 | 206.0 | 0.4239 | 102.0497 | 97.991 | 12.249 | 55.0 | 148.0 |
124
- | 142500 | 0.5758 | 65.5 | 218.0 | 0.4256 | 102.0522 | 97.989 | 12.249 | 50.25 | 144.0 |
125
- | 145000 | 0.5859 | 65.0 | 227.0 | 0.4201 | 102.154 | 97.891 | 12.236 | 50.5 | 135.0 |
126
- | 147500 | 0.5960 | 65.5 | 211.0 | 0.4216 | 102.1033 | 97.94 | 12.243 | 49.75 | 92.5 |
127
- | 150000 | 0.6061 | 66.0 | 242.0 | 0.4288 | 102.1595 | 97.886 | 12.236 | 52.0 | 137.0 |
128
- | 152500 | 0.6162 | 67.0 | 229.0 | 0.4180 | 102.5134 | 97.548 | 12.194 | 49.25 | 111.0 |
129
- | 155000 | 0.6263 | 65.0 | 206.0 | 0.4224 | 102.3146 | 97.738 | 12.217 | 51.0 | 151.0 |
130
- | 157500 | 0.6364 | 66.0 | 220.0 | 0.4266 | 102.1949 | 97.852 | 12.232 | 48.75 | 107.5 |
131
- | 160000 | 0.6465 | 67.5 | 212.0 | 0.4226 | 102.1337 | 97.911 | 12.239 | 49.25 | 97.0 |
132
- | 162500 | 0.6566 | 67.0 | 212.0 | 0.4186 | 102.1028 | 97.94 | 12.243 | 50.0 | 89.0 |
133
- | 165000 | 0.6667 | 63.75 | 231.0 | 0.4159 | 101.9547 | 98.083 | 12.26 | 47.75 | 116.0 |
134
- | 167500 | 0.6768 | 67.5 | 227.0 | 0.4208 | 102.0173 | 98.023 | 12.253 | 51.0 | 203.0 |
135
- | 170000 | 0.6869 | 65.5 | 268.0 | 0.4194 | 101.9863 | 98.052 | 12.257 | 49.25 | 108.5 |
136
- | 172500 | 0.6970 | 66.5 | 208.0 | 0.4165 | 102.0041 | 98.035 | 12.254 | 49.75 | 175.0 |
137
- | 175000 | 0.7071 | 68.0 | 221.0 | 0.4189 | 102.0695 | 97.972 | 12.247 | 50.25 | 91.5 |
138
- | 177500 | 0.7172 | 66.0 | 211.0 | 0.4188 | 101.9141 | 98.122 | 12.265 | 48.75 | 124.0 |
139
- | 180000 | 0.7273 | 64.0 | 200.0 | 0.4169 | 102.2518 | 97.798 | 12.225 | 46.75 | 113.0 |
140
- | 182500 | 0.7374 | 64.0 | 204.0 | 0.4204 | 102.0976 | 97.946 | 12.243 | 49.5 | 140.0 |
141
- | 185000 | 0.7475 | 65.0 | 213.0 | 0.4152 | 102.3207 | 97.732 | 12.216 | 48.5 | 127.0 |
142
- | 187500 | 0.7576 | 65.0 | 206.0 | 0.4155 | 102.1198 | 97.924 | 12.241 | 49.25 | 108.0 |
143
- | 190000 | 0.7677 | 66.0 | 213.0 | 0.4182 | 102.192 | 97.855 | 12.232 | 49.25 | 130.0 |
144
- | 192500 | 0.7778 | 68.0 | 221.0 | 0.4160 | 102.0413 | 98.0 | 12.25 | 53.75 | 143.0 |
145
- | 195000 | 0.7879 | 66.5 | 225.0 | 0.4136 | 102.0553 | 97.986 | 12.248 | 53.0 | 164.0 |
146
- | 197500 | 0.7980 | 65.5 | 218.0 | 0.4160 | 101.9027 | 98.133 | 12.267 | 49.0 | 89.0 |
147
- | 200000 | 0.8081 | 66.0 | 225.0 | 0.4148 | 102.4437 | 97.615 | 12.202 | 48.0 | 105.5 |
148
- | 202500 | 0.8182 | 66.5 | 208.0 | 0.4189 | 102.0449 | 97.996 | 12.25 | 49.0 | 131.0 |
149
- | 205000 | 0.8283 | 66.0 | 217.0 | 0.4150 | 102.0719 | 97.97 | 12.246 | 51.5 | 186.0 |
150
- | 207500 | 0.8384 | 69.5 | 254.0 | 0.4214 | 102.0931 | 97.95 | 12.244 | 51.25 | 153.0 |
151
- | 210000 | 0.8485 | 67.0 | 216.0 | 0.4235 | 102.1471 | 97.898 | 12.237 | 49.75 | 121.5 |
152
- | 212500 | 0.8586 | 65.5 | 216.0 | 0.4125 | 102.1121 | 97.932 | 12.241 | 48.5 | 120.0 |
153
- | 215000 | 0.8687 | 65.5 | 225.0 | 0.4145 | 102.235 | 97.814 | 12.227 | 49.0 | 105.5 |
154
- | 217500 | 0.8788 | 69.0 | 264.0 | 0.4188 | 102.0807 | 97.962 | 12.245 | 49.5 | 209.0 |
155
- | 220000 | 0.8889 | 68.0 | 218.0 | 0.4157 | 102.1941 | 97.853 | 12.232 | 50.5 | 107.0 |
156
- | 222500 | 0.8990 | 65.5 | 212.0 | 0.4231 | 102.2725 | 97.778 | 12.222 | 51.5 | 201.0 |
157
- | 225000 | 0.9091 | 65.0 | 229.0 | 0.4150 | 102.3034 | 97.748 | 12.219 | 50.25 | 137.0 |
158
- | 227500 | 0.9192 | 65.0 | 218.0 | 0.4131 | 102.2051 | 97.842 | 12.23 | 48.75 | 131.0 |
159
- | 230000 | 0.9293 | 64.5 | 217.0 | 0.4264 | 102.0499 | 97.991 | 12.249 | 48.5 | 148.0 |
160
- | 232500 | 0.9394 | 68.0 | 232.0 | 0.4168 | 102.1883 | 97.859 | 12.232 | 49.0 | 107.0 |
161
- | 235000 | 0.9495 | 69.0 | 233.0 | 0.4199 | 102.0331 | 98.007 | 12.251 | 46.75 | 157.0 |
162
- | 237500 | 0.9596 | 65.0 | 212.0 | 0.4194 | 102.1218 | 97.922 | 12.24 | 52.25 | 150.0 |
163
- | 240000 | 0.9697 | 65.5 | 215.0 | 0.4144 | 102.0228 | 98.017 | 12.252 | 48.75 | 182.0 |
164
- | 242500 | 0.9798 | 65.0 | 209.0 | 0.4203 | 102.0484 | 97.993 | 12.249 | 49.5 | 95.5 |
165
- | 245000 | 0.9899 | 65.0 | 266.0 | 0.4097 | 102.0961 | 97.947 | 12.243 | 49.5 | 111.0 |
166
- | 247500 | 1.0 | 64.5 | 222.0 | 0.4159 | 102.373 | 97.682 | 12.21 | 49.75 | 81.5 |
167
-
168
- ### Framework versions
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
169
  - Distily 0.2.0
170
  - Transformers 4.44.0
171
  - Pytorch 2.3.0
 
1
  ---
2
  base_model: distilbert/distilgpt2
3
+ datasets:
4
+ - wikimedia/wikipedia
5
  library_name: Distily
6
  license: apache-2.0
7
  tags:
 
11
  results: []
12
  ---
13
 
 
14
 
15
+ # Summary
16
 
17
+ Distilled with [Distily](https://github.com/lapp0/distily) library
18
+ using teacher model [gpt2](https://huggingface.co/gpt2)
19
+ on dataset [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia).
 
 
 
 
 
 
 
 
20
 
21
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
22
  should probably proofread and complete it, then remove this comment.
23
 
24
+ # Model description
 
 
 
 
25
 
26
  More information needed
27
 
28
+ # Intended uses & limitations
29
 
30
  More information needed
31
  -->
32
 
33
+ # Model Architecture:
34
+ - **Architecture**: `GPT2LMHeadModel`
35
+ - **Total Parameters**: 81,912,576
36
+ - **Data Type (dtype)**: torch.bfloat16
37
+ - **Model Size**: 0.16 GB
38
 
 
39
 
40
+ # Evaluation Metrics Comparison
41
+
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
43
+ | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
44
  | **teacher eval** | | 43.25 | 61.25 | | | | | 11.6875 | 19.125 |
45
+ | 0 | 0 | 2018634629120.0 | 122045790683136.0 | 21.0022 | 102.1494 | 97.896 | 12.237 | 9999220736.0 | 43705587204096.0 |
46
+ | 2500 | 0.0101 | 299008.0 | 6422528.0 | 5.8065 | 101.9861 | 98.053 | 12.257 | 45824.0 | 14483456.0 |
47
+ | 5000 | 0.0202 | 6880.0 | 96256.0 | 3.3113 | 102.9516 | 97.133 | 12.142 | 4160.0 | 493568.0 |
48
+ | 7500 | 0.0303 | 1216.0 | 8096.0 | 2.1560 | 103.0236 | 97.065 | 12.133 | 692.0 | 42752.0 |
49
+ | 10000 | 0.0404 | 608.0 | 3664.0 | 1.7825 | 102.3752 | 97.68 | 12.21 | 388.0 | 888.0 |
50
+ | 12500 | 0.0505 | 358.0 | 1632.0 | 1.4664 | 102.1871 | 97.86 | 12.232 | 272.0 | 308.0 |
51
+ | 15000 | 0.0606 | 288.0 | 1176.0 | 1.3488 | 102.6007 | 97.465 | 12.183 | 228.0 | 260.0 |
52
+ | 17500 | 0.0707 | 255.0 | 1040.0 | 1.2932 | 102.1542 | 97.891 | 12.236 | 199.0 | 215.0 |
53
+ | 20000 | 0.0808 | 216.0 | 892.0 | 1.1570 | 102.1073 | 97.936 | 12.242 | 173.0 | 149.0 |
54
+ | 22500 | 0.0909 | 178.0 | 740.0 | 1.0350 | 102.0765 | 97.966 | 12.246 | 146.0 | 141.0 |
55
+ | 25000 | 0.1010 | 155.0 | 524.0 | 0.9676 | 102.1019 | 97.941 | 12.243 | 122.5 | 139.0 |
56
+ | 27500 | 0.1111 | 142.0 | 560.0 | 0.9230 | 102.0256 | 98.015 | 12.252 | 114.0 | 130.0 |
57
+ | 30000 | 0.1212 | 137.0 | 470.0 | 0.8998 | 102.3365 | 97.717 | 12.215 | 108.5 | 138.0 |
58
+ | 32500 | 0.1313 | 134.0 | 476.0 | 0.8740 | 102.3911 | 97.665 | 12.208 | 104.0 | 140.0 |
59
+ | 35000 | 0.1414 | 129.0 | 496.0 | 0.8657 | 102.2153 | 97.833 | 12.229 | 102.5 | 141.0 |
60
+ | 37500 | 0.1515 | 127.0 | 464.0 | 0.8513 | 102.0489 | 97.992 | 12.249 | 97.0 | 117.0 |
61
+ | 40000 | 0.1616 | 108.0 | 446.0 | 0.7522 | 102.9331 | 97.15 | 12.144 | 93.0 | 104.0 |
62
+ | 42500 | 0.1717 | 99.5 | 374.0 | 0.6850 | 103.1088 | 96.985 | 12.123 | 82.0 | 116.0 |
63
+ | 45000 | 0.1818 | 90.5 | 346.0 | 0.6316 | 102.7903 | 97.285 | 12.161 | 73.5 | 113.0 |
64
+ | 47500 | 0.1919 | 82.5 | 320.0 | 0.5960 | 102.5988 | 97.467 | 12.183 | 71.0 | 101.0 |
65
+ | 50000 | 0.2020 | 78.5 | 306.0 | 0.5676 | 102.5936 | 97.472 | 12.184 | 72.5 | 106.0 |
66
+ | 52500 | 0.2121 | 79.5 | 290.0 | 0.5424 | 102.5863 | 97.479 | 12.185 | 64.5 | 92.0 |
67
+ | 55000 | 0.2222 | 76.0 | 270.0 | 0.5280 | 102.6307 | 97.437 | 12.18 | 65.0 | 87.0 |
68
+ | 57500 | 0.2323 | 76.5 | 272.0 | 0.5278 | 101.9639 | 98.074 | 12.259 | 64.5 | 102.0 |
69
+ | 60000 | 0.2424 | 77.5 | 268.0 | 0.5286 | 102.0921 | 97.951 | 12.244 | 62.75 | 99.5 |
70
+ | 62500 | 0.2525 | 75.5 | 264.0 | 0.5204 | 102.0679 | 97.974 | 12.247 | 63.25 | 83.0 |
71
+ | 65000 | 0.2626 | 76.0 | 260.0 | 0.5176 | 102.1795 | 97.867 | 12.233 | 61.5 | 90.5 |
72
+ | 67500 | 0.2727 | 74.5 | 256.0 | 0.5112 | 102.5764 | 97.488 | 12.186 | 62.25 | 93.5 |
73
+ | 70000 | 0.2828 | 73.5 | 258.0 | 0.5128 | 101.9569 | 98.081 | 12.26 | 62.0 | 79.0 |
74
+ | 72500 | 0.2929 | 75.0 | 250.0 | 0.5053 | 101.9382 | 98.099 | 12.262 | 64.0 | 96.0 |
75
+ | 75000 | 0.3030 | 72.5 | 238.0 | 0.5068 | 102.0407 | 98.0 | 12.25 | 61.5 | 88.5 |
76
+ | 77500 | 0.3131 | 73.5 | 256.0 | 0.5085 | 102.0542 | 97.987 | 12.248 | 64.5 | 86.5 |
77
+ | 80000 | 0.3232 | 70.5 | 238.0 | 0.4699 | 102.4042 | 97.652 | 12.207 | 54.75 | 98.5 |
78
+ | 82500 | 0.3333 | 68.0 | 242.0 | 0.4574 | 102.2684 | 97.782 | 12.223 | 55.5 | 160.0 |
79
+ | 85000 | 0.3434 | 64.5 | 218.0 | 0.4490 | 102.3277 | 97.725 | 12.216 | 52.0 | 77.5 |
80
+ | 87500 | 0.3535 | 66.5 | 203.0 | 0.4394 | 102.1134 | 97.93 | 12.241 | 51.25 | 67.5 |
81
+ | 90000 | 0.3636 | 63.75 | 212.0 | 0.4310 | 102.0438 | 97.997 | 12.25 | 51.25 | 88.5 |
82
+ | 92500 | 0.3737 | 65.5 | 209.0 | 0.4262 | 101.9984 | 98.041 | 12.255 | 49.75 | 103.5 |
83
+ | 95000 | 0.3838 | 65.0 | 204.0 | 0.4274 | 102.0781 | 97.964 | 12.246 | 46.25 | 83.0 |
84
+ | 97500 | 0.3939 | 64.5 | 201.0 | 0.4192 | 102.0692 | 97.973 | 12.247 | 50.5 | 94.5 |
85
+ | 100000 | 0.4040 | 64.5 | 203.0 | 0.4207 | 102.1283 | 97.916 | 12.24 | 49.0 | 88.0 |
86
+ | 102500 | 0.4141 | 63.0 | 209.0 | 0.4184 | 102.224 | 97.824 | 12.228 | 48.0 | 125.0 |
87
+ | 105000 | 0.4242 | 62.75 | 193.0 | 0.4166 | 102.1918 | 97.855 | 12.232 | 46.0 | 76.0 |
88
+ | 107500 | 0.4343 | 62.75 | 197.0 | 0.4128 | 102.1719 | 97.874 | 12.234 | 47.0 | 113.0 |
89
+ | 110000 | 0.4444 | 64.5 | 191.0 | 0.4118 | 103.0992 | 96.994 | 12.124 | 49.0 | 82.0 |
90
+ | 112500 | 0.4545 | 65.0 | 213.0 | 0.4128 | 102.7296 | 97.343 | 12.168 | 47.0 | 111.5 |
91
+ | 115000 | 0.4646 | 68.5 | 207.0 | 0.4301 | 102.178 | 97.868 | 12.234 | 49.0 | 108.0 |
92
+ | 117500 | 0.4747 | 65.0 | 217.0 | 0.4372 | 102.2302 | 97.818 | 12.227 | 50.25 | 124.0 |
93
+ | 120000 | 0.4848 | 65.5 | 210.0 | 0.4351 | 102.2952 | 97.756 | 12.22 | 51.0 | 139.0 |
94
+ | 122500 | 0.4949 | 66.0 | 272.0 | 0.4352 | 102.1941 | 97.853 | 12.232 | 50.5 | 226.0 |
95
+ | 125000 | 0.5051 | 67.0 | 240.0 | 0.4387 | 101.978 | 98.06 | 12.258 | 49.0 | 71.0 |
96
+ | 127500 | 0.5152 | 66.5 | 224.0 | 0.4396 | 101.9014 | 98.134 | 12.267 | 49.75 | 100.0 |
97
+ | 130000 | 0.5253 | 65.5 | 227.0 | 0.4354 | 102.1244 | 97.92 | 12.24 | 50.75 | 146.0 |
98
+ | 132500 | 0.5354 | 66.0 | 209.0 | 0.4286 | 102.0218 | 98.018 | 12.252 | 52.25 | 101.5 |
99
+ | 135000 | 0.5455 | 64.5 | 220.0 | 0.4361 | 101.9074 | 98.128 | 12.266 | 51.25 | 181.0 |
100
+ | 137500 | 0.5556 | 66.5 | 223.0 | 0.4288 | 102.0744 | 97.968 | 12.246 | 49.0 | 103.0 |
101
+ | 140000 | 0.5657 | 66.5 | 232.0 | 0.4287 | 102.1162 | 97.928 | 12.241 | 49.25 | 127.5 |
102
+ | 142500 | 0.5758 | 66.5 | 220.0 | 0.4299 | 101.9461 | 98.091 | 12.261 | 49.5 | 88.5 |
103
+ | 145000 | 0.5859 | 65.5 | 217.0 | 0.4238 | 101.9572 | 98.08 | 12.26 | 48.75 | 177.0 |
104
+ | 147500 | 0.5960 | 64.0 | 205.0 | 0.4109 | 101.9497 | 98.088 | 12.261 | 48.75 | 128.0 |
105
+ | 150000 | 0.6061 | 63.5 | 224.0 | 0.4051 | 102.0205 | 98.02 | 12.252 | 48.5 | 117.5 |
106
+ | 152500 | 0.6162 | 63.25 | 202.0 | 0.4000 | 101.9318 | 98.105 | 12.263 | 47.5 | 160.0 |
107
+ | 155000 | 0.6263 | 63.75 | 195.0 | 0.4052 | 102.0203 | 98.02 | 12.252 | 48.75 | 100.0 |
108
+ | 157500 | 0.6364 | 63.75 | 212.0 | 0.4014 | 101.8935 | 98.142 | 12.268 | 49.25 | 113.0 |
109
+ | 160000 | 0.6465 | 62.75 | 198.0 | 0.3988 | 101.9178 | 98.118 | 12.265 | 44.5 | 132.0 |
110
+ | 162500 | 0.6566 | 64.5 | 192.0 | 0.3918 | 102.0303 | 98.01 | 12.251 | 45.5 | 100.0 |
111
+ | 165000 | 0.6667 | 62.5 | 202.0 | 0.3958 | 102.3627 | 97.692 | 12.211 | 47.75 | 88.5 |
112
+ | 167500 | 0.6768 | 62.5 | 191.0 | 0.3883 | 102.1537 | 97.892 | 12.236 | 44.75 | 80.5 |
113
+ | 170000 | 0.6869 | 63.5 | 195.0 | 0.3880 | 102.0728 | 97.969 | 12.246 | 51.0 | 91.5 |
114
+ | 172500 | 0.6970 | 60.75 | 201.0 | 0.3863 | 101.9235 | 98.113 | 12.264 | 47.5 | 90.5 |
115
+ | 175000 | 0.7071 | 61.5 | 189.0 | 0.3806 | 101.9376 | 98.099 | 12.262 | 46.5 | 82.5 |
116
+ | 177500 | 0.7172 | 58.75 | 171.0 | 0.3512 | 101.9844 | 98.054 | 12.257 | 42.75 | 66.0 |
117
+ | 180000 | 0.7273 | 55.5 | 161.0 | 0.3218 | 101.881 | 98.154 | 12.269 | 39.25 | 54.0 |
118
+ | 182500 | 0.7374 | 54.25 | 149.0 | 0.3148 | 101.9839 | 98.055 | 12.257 | 38.75 | 47.75 |
119
+ | 185000 | 0.7475 | 53.5 | 160.0 | 0.3133 | 101.9875 | 98.051 | 12.256 | 38.75 | 45.0 |
120
+ | 187500 | 0.7576 | 54.75 | 160.0 | 0.3114 | 101.9762 | 98.062 | 12.258 | 38.0 | 43.75 |
121
+ | 190000 | 0.7677 | 53.75 | 147.0 | 0.3075 | 101.9972 | 98.042 | 12.255 | 38.0 | 38.25 |
122
+ | 192500 | 0.7778 | 54.0 | 157.0 | 0.3057 | 101.9431 | 98.094 | 12.262 | 38.0 | 48.0 |
123
+ | 195000 | 0.7879 | 53.25 | 149.0 | 0.3058 | 101.9778 | 98.061 | 12.258 | 37.0 | 41.0 |
124
+ | 197500 | 0.7980 | 54.0 | 152.0 | 0.3032 | 102.0059 | 98.034 | 12.254 | 37.25 | 40.0 |
125
+ | 200000 | 0.8081 | 53.75 | 151.0 | 0.3033 | 102.0615 | 97.98 | 12.248 | 37.25 | 47.25 |
126
+ | 202500 | 0.8182 | 53.0 | 146.0 | 0.2957 | 102.0116 | 98.028 | 12.254 | 36.75 | 39.0 |
127
+ | 205000 | 0.8283 | 52.5 | 139.0 | 0.2903 | 102.1449 | 97.9 | 12.238 | 36.5 | 35.75 |
128
+ | 207500 | 0.8384 | 52.0 | 142.0 | 0.2894 | 102.0126 | 98.027 | 12.253 | 36.25 | 38.25 |
129
+ | 210000 | 0.8485 | 52.25 | 142.0 | 0.2883 | 102.0938 | 97.949 | 12.244 | 36.0 | 37.25 |
130
+ | 212500 | 0.8586 | 52.5 | 141.0 | 0.2874 | 101.9515 | 98.086 | 12.261 | 36.0 | 37.0 |
131
+ | 215000 | 0.8687 | 52.25 | 140.0 | 0.2873 | 101.9427 | 98.094 | 12.262 | 36.0 | 36.0 |
132
+ | 217500 | 0.8788 | 51.75 | 141.0 | 0.2863 | 102.0114 | 98.028 | 12.254 | 36.0 | 35.5 |
133
+ | 220000 | 0.8889 | 52.0 | 141.0 | 0.2854 | 102.0424 | 97.999 | 12.25 | 36.0 | 35.75 |
134
+ | 222500 | 0.8990 | 52.5 | 143.0 | 0.2853 | 102.0368 | 98.004 | 12.25 | 36.0 | 35.25 |
135
+ | 225000 | 0.9091 | 52.0 | 142.0 | 0.2849 | 102.115 | 97.929 | 12.241 | 35.75 | 35.0 |
136
+ | 227500 | 0.9192 | 52.0 | 141.0 | 0.2851 | 102.0455 | 97.996 | 12.249 | 36.0 | 35.25 |
137
+ | 230000 | 0.9293 | 52.0 | 141.0 | 0.2846 | 102.0273 | 98.013 | 12.252 | 35.75 | 35.25 |
138
+ | 232500 | 0.9394 | 52.0 | 141.0 | 0.2843 | 101.961 | 98.077 | 12.26 | 35.75 | 35.0 |
139
+ | 235000 | 0.9495 | 52.0 | 141.0 | 0.2844 | 102.0188 | 98.021 | 12.253 | 35.75 | 35.25 |
140
+ | 237500 | 0.9596 | 52.0 | 141.0 | 0.2845 | 102.0714 | 97.971 | 12.246 | 35.75 | 35.25 |
141
+ | 240000 | 0.9697 | 52.0 | 141.0 | 0.2844 | 102.0371 | 98.004 | 12.25 | 35.75 | 35.25 |
142
+ | 242500 | 0.9798 | 52.0 | 141.0 | 0.2844 | 102.0363 | 98.004 | 12.251 | 35.75 | 35.25 |
143
+ | 245000 | 0.9899 | 52.0 | 141.0 | 0.2844 | 102.0254 | 98.015 | 12.252 | 35.75 | 35.25 |
144
+ | 247500 | 1.0 | 52.0 | 141.0 | 0.2846 | 102.5728 | 97.492 | 12.186 | 35.75 | 35.25 |
145
+
146
+ # Resource Usage Comparison
147
+
148
+ - VRAM Use: 7.2012 GB
149
+
150
+ `# Distillation (Teacher -> Student) Architecture Difference:
151
+
152
+ - **Architecture**: `GPT2LMHeadModel` -> `GPT2LMHeadModel`
153
+ - **Total Parameters**: 124,439,808 -> 81,912,576
154
+ - **Data Type (dtype)**: 124439808 -> torch.bfloat16
155
+ - **Model Size**: 0.24 GB -> 0.16 GB
156
+
157
+ <details>
158
+ <summary>Module Diff Details</summary>
159
+
160
+ ```diff
161
+ --- teacher model modules
162
+ +++ student model modules
163
+ @@ -4,7 +4,7 @@
164
+ (wpe): Embedding(1024, 768)
165
+ (drop): Dropout(p=0.1, inplace=False)
166
+ (h): ModuleList(
167
+ - (0-11): 12 x GPT2Block(
168
+ + (0-5): 6 x GPT2Block(
169
+ (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
170
+ (attn): GPT2FlashAttention2(
171
+ (c_attn): Conv1D()
172
+
173
+ ```
174
+
175
+ </details>
176
+ <br/>
177
+
178
+ # Train Dataset
179
+ Trained on 521,350,663 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
180
+
181
+ - Num Samples: `990,000`
182
+ - Subset: `20231101.en`
183
+ - Split: `train`
184
+
185
+
186
+ # Training Objective
187
+
188
+ ```
189
+ DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl))
190
+ ```
191
+
192
+ # Hyperparameters
193
+ The following hyperparameters were used during training:
194
+
195
+ <details>
196
+ <summary>Expand</summary>
197
+
198
+ - learning_rate: `0.0001`
199
+ - train_batch_size: `4`
200
+ - eval_batch_size: `8`
201
+ - seed: `42`
202
+ - optimizer: `Adam with betas=(0.9,0.999) and epsilon=1e-08`
203
+ - lr_scheduler_type: `cosine`
204
+ - lr_scheduler_warmup_ratio: `0.5`
205
+ - num_epochs: `1.0`
206
+ - distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl))`
207
+ - train_embeddings: `True`
208
+ - lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fd9b01df220>`
209
+ - student_model_name_or_path: `None`
210
+ - student_config_name_or_path: `distilbert/distilgpt2`
211
+ - student_model_config: `None`
212
+ - reinitialize_weights: `None`
213
+ - copy_teacher_modules: `[('lm_head', False)]`
214
+ - student_model_as_bitnet: `False`
215
+ - student_model_compile: `False`
216
+ - dropout: `None`
217
+ - teacher_model_name_or_path: `gpt2`
218
+ - teacher_load_in_8bit: `False`
219
+ - teacher_load_in_4bit: `False`
220
+ - teacher_model_compile: `False`
221
+ - dataset_uri: `wikimedia/wikipedia`
222
+ - dataset_subset: `20231101.en`
223
+ - dataset_split: `train`
224
+ - dataset_column_name: `text`
225
+ - dataset_sample_size: `1000000`
226
+ - dataset_test_size: `0.01`
227
+ - gradient_accumulation_steps: `1`
228
+ - weight_decay: `0.0`
229
+ - max_grad_norm: `1.0`
230
+ - warmup_ratio: `0.5`
231
+ - warmup_steps: `0`
232
+ - gradient_checkpointing: `True`
233
+
234
+ </details>
235
+ <br/>
236
+
237
+
238
+ # Framework Versions
239
  - Distily 0.2.0
240
  - Transformers 4.44.0
241
  - Pytorch 2.3.0
logs/dataset_sample_size=1000000, lr_scheduler_type=cosine, warmup_ratio=0.5/events.out.tfevents.1724299704.f383272e719b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eb078a1e224df95b4a3313b682e0a047636b719ad696e0cdadedd7711e757272
3
+ size 588