Hugging Face's logo

ChosenQ
/
DeepSeek-R1-Distill-Qwen-1.5B-GRPO

Text Generation
Transformers
Safetensors
qwen2
Generated from Trainer
open-r1
trl
grpo
conversational
text-generation-inference
Model card Files Community
ChosenQ commited on
Commit
611917e
·
verified ·
1 Parent(s): d1c750a

Model save

Browse files
Files changed (5) hide show
  1. README.md +68 -0
  2. all_results.json +8 -0
  3. generation_config.json +9 -0
  4. train_results.json +8 -0
  5. trainer_state.json +518 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
3
+ library_name: transformers
4
+ model_name: DeepSeek-R1-Distill-Qwen-1.5B-GRPO
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - grpo
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for DeepSeek-R1-Distill-Qwen-1.5B-GRPO
13
+
14
+ This model is a fine-tuned version of [deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="ChosenQ/DeepSeek-R1-Distill-Qwen-1.5B-GRPO", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/chosenqiuch-kaust/huggingface/runs/rgh5vqsi)
31
+
32
+
33
+ This model was trained with GRPO, a method introduced in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300).
34
+
35
+ ### Framework versions
36
+
37
+ - TRL: 0.16.0.dev0
38
+ - Transformers: 4.49.0
39
+ - Pytorch: 2.5.1
40
+ - Datasets: 3.3.2
41
+ - Tokenizers: 0.21.0
42
+
43
+ ## Citations
44
+
45
+ Cite GRPO as:
46
+
47
+ ```bibtex
48
+ @article{zhihong2024deepseekmath,
49
+ title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
50
+ author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
51
+ year = 2024,
52
+ eprint = {arXiv:2402.03300},
53
+ }
54
+
55
+ ```
56
+
57
+ Cite TRL as:
58
+
59
+ ```bibtex
60
+ @misc{vonwerra2022trl,
61
+ title = {{TRL: Transformer Reinforcement Learning}},
62
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
63
+ year = 2020,
64
+ journal = {GitHub repository},
65
+ publisher = {GitHub},
66
+ howpublished = {\url{https://github.com/huggingface/trl}}
67
+ }
68
+ ```
all_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": 0.059163046507712674,
4
+ "train_runtime": 9090.9077,
5
+ "train_samples": 817,
6
+ "train_samples_per_second": 0.09,
7
+ "train_steps_per_second": 0.004
8
+ }
generation_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 151646,
4
+ "do_sample": true,
5
+ "eos_token_id": 151643,
6
+ "temperature": 0.6,
7
+ "top_p": 0.95,
8
+ "transformers_version": "4.49.0"
9
+ }
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": 0.059163046507712674,
4
+ "train_runtime": 9090.9077,
5
+ "train_samples": 817,
6
+ "train_samples_per_second": 0.09,
7
+ "train_steps_per_second": 0.004
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,518 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 0.9963369963369964,
5
+ "eval_steps": 500,
6
+ "global_step": 34,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "clip_ratio": 0.0,
13
+ "completion_length": 3534.5521545410156,
14
+ "epoch": 0.029304029304029304,
15
+ "grad_norm": 0.08098714798688889,
16
+ "kl": 0.0,
17
+ "learning_rate": 2.5e-07,
18
+ "loss": 0.0471,
19
+ "reward": 0.2721354260575026,
20
+ "reward_std": 0.2681832159869373,
21
+ "rewards/accuracy_reward": 0.16666666697710752,
22
+ "rewards/tag_count_reward": 0.10546875349245965,
23
+ "step": 1
24
+ },
25
+ {
26
+ "clip_ratio": 0.0,
27
+ "completion_length": 3753.3802490234375,
28
+ "epoch": 0.05860805860805861,
29
+ "grad_norm": 0.10224228352308273,
30
+ "kl": 0.0,
31
+ "learning_rate": 5e-07,
32
+ "loss": 0.065,
33
+ "reward": 0.1861979232635349,
34
+ "reward_std": 0.22471919422969222,
35
+ "rewards/accuracy_reward": 0.10937500325962901,
36
+ "rewards/tag_count_reward": 0.07682291802484542,
37
+ "step": 2
38
+ },
39
+ {
40
+ "clip_ratio": 0.0,
41
+ "completion_length": 3401.057403564453,
42
+ "epoch": 0.08791208791208792,
43
+ "grad_norm": 0.09934523701667786,
44
+ "kl": 3.714859485626221e-05,
45
+ "learning_rate": 7.5e-07,
46
+ "loss": 0.0568,
47
+ "reward": 0.2799479253590107,
48
+ "reward_std": 0.30056663788855076,
49
+ "rewards/accuracy_reward": 0.16145833674818277,
50
+ "rewards/tag_count_reward": 0.1184895858168602,
51
+ "step": 3
52
+ },
53
+ {
54
+ "clip_ratio": 0.0,
55
+ "completion_length": 3252.7084045410156,
56
+ "epoch": 0.11721611721611722,
57
+ "grad_norm": 0.08650524169206619,
58
+ "kl": 3.603845834732056e-05,
59
+ "learning_rate": 1e-06,
60
+ "loss": 0.0621,
61
+ "reward": 0.36328125500585884,
62
+ "reward_std": 0.2626346012111753,
63
+ "rewards/accuracy_reward": 0.2447916716337204,
64
+ "rewards/tag_count_reward": 0.11848958709742874,
65
+ "step": 4
66
+ },
67
+ {
68
+ "clip_ratio": 0.0,
69
+ "completion_length": 3478.4375915527344,
70
+ "epoch": 0.14652014652014653,
71
+ "grad_norm": 0.08601976931095123,
72
+ "kl": 3.30507755279541e-05,
73
+ "learning_rate": 9.975348529157229e-07,
74
+ "loss": 0.0597,
75
+ "reward": 0.34114584454800934,
76
+ "reward_std": 0.2457886270713061,
77
+ "rewards/accuracy_reward": 0.23958333721384406,
78
+ "rewards/tag_count_reward": 0.10156250407453626,
79
+ "step": 5
80
+ },
81
+ {
82
+ "clip_ratio": 0.0,
83
+ "completion_length": 3602.2396850585938,
84
+ "epoch": 0.17582417582417584,
85
+ "grad_norm": 0.09895998239517212,
86
+ "kl": 3.193318843841553e-05,
87
+ "learning_rate": 9.901664203302124e-07,
88
+ "loss": 0.0823,
89
+ "reward": 0.24348959093913436,
90
+ "reward_std": 0.2597248964011669,
91
+ "rewards/accuracy_reward": 0.15625000884756446,
92
+ "rewards/tag_count_reward": 0.08723958628252149,
93
+ "step": 6
94
+ },
95
+ {
96
+ "clip_ratio": 0.0,
97
+ "completion_length": 3152.7813415527344,
98
+ "epoch": 0.20512820512820512,
99
+ "grad_norm": 0.10855786502361298,
100
+ "kl": 2.0995736122131348e-05,
101
+ "learning_rate": 9.779754323328192e-07,
102
+ "loss": 0.0637,
103
+ "reward": 0.4752604365348816,
104
+ "reward_std": 0.3382994793355465,
105
+ "rewards/accuracy_reward": 0.34375000931322575,
106
+ "rewards/tag_count_reward": 0.1315104216337204,
107
+ "step": 7
108
+ },
109
+ {
110
+ "clip_ratio": 0.0,
111
+ "completion_length": 3384.385498046875,
112
+ "epoch": 0.23443223443223443,
113
+ "grad_norm": 0.08194123208522797,
114
+ "kl": 1.2666918337345123e-05,
115
+ "learning_rate": 9.610954559391704e-07,
116
+ "loss": 0.04,
117
+ "reward": 0.296875006519258,
118
+ "reward_std": 0.24468757305294275,
119
+ "rewards/accuracy_reward": 0.19270833814516664,
120
+ "rewards/tag_count_reward": 0.10416667116805911,
121
+ "step": 8
122
+ },
123
+ {
124
+ "clip_ratio": 0.0,
125
+ "completion_length": 3482.776123046875,
126
+ "epoch": 0.26373626373626374,
127
+ "grad_norm": 0.11475110799074173,
128
+ "kl": 1.0907649993896484e-05,
129
+ "learning_rate": 9.397114317029974e-07,
130
+ "loss": 0.1002,
131
+ "reward": 0.2395833432674408,
132
+ "reward_std": 0.2327865227125585,
133
+ "rewards/accuracy_reward": 0.1510416711680591,
134
+ "rewards/tag_count_reward": 0.0885416695382446,
135
+ "step": 9
136
+ },
137
+ {
138
+ "clip_ratio": 0.0,
139
+ "completion_length": 3433.7032165527344,
140
+ "epoch": 0.29304029304029305,
141
+ "grad_norm": 0.09596528857946396,
142
+ "kl": 2.337619662284851e-06,
143
+ "learning_rate": 9.140576474687263e-07,
144
+ "loss": 0.0698,
145
+ "reward": 0.34895834513008595,
146
+ "reward_std": 0.2847986426204443,
147
+ "rewards/accuracy_reward": 0.23958333861082792,
148
+ "rewards/tag_count_reward": 0.10937500442378223,
149
+ "step": 10
150
+ },
151
+ {
152
+ "clip_ratio": 0.0,
153
+ "completion_length": 3547.1927795410156,
154
+ "epoch": 0.32234432234432236,
155
+ "grad_norm": 0.09714619070291519,
156
+ "kl": 1.394655555486679e-07,
157
+ "learning_rate": 8.844151714648274e-07,
158
+ "loss": 0.0661,
159
+ "reward": 0.3085937525611371,
160
+ "reward_std": 0.2829530159942806,
161
+ "rewards/accuracy_reward": 0.2083333358168602,
162
+ "rewards/tag_count_reward": 0.10026041860692203,
163
+ "step": 11
164
+ },
165
+ {
166
+ "clip_ratio": 0.0,
167
+ "completion_length": 3510.7396545410156,
168
+ "epoch": 0.3516483516483517,
169
+ "grad_norm": 0.09303876012563705,
170
+ "kl": 3.088265657424927e-05,
171
+ "learning_rate": 8.511087728614862e-07,
172
+ "loss": 0.0517,
173
+ "reward": 0.27604167291428894,
174
+ "reward_std": 0.21292753401212394,
175
+ "rewards/accuracy_reward": 0.18750000931322575,
176
+ "rewards/tag_count_reward": 0.08854166825767606,
177
+ "step": 12
178
+ },
179
+ {
180
+ "clip_ratio": 0.0,
181
+ "completion_length": 3502.9532165527344,
182
+ "epoch": 0.38095238095238093,
183
+ "grad_norm": 0.10329490154981613,
184
+ "kl": 0.00013720989227294922,
185
+ "learning_rate": 8.145033635316128e-07,
186
+ "loss": 0.0755,
187
+ "reward": 0.2122395852347836,
188
+ "reward_std": 0.1669893644284457,
189
+ "rewards/accuracy_reward": 0.11979167070239782,
190
+ "rewards/tag_count_reward": 0.09244791802484542,
191
+ "step": 13
192
+ },
193
+ {
194
+ "clip_ratio": 0.0,
195
+ "completion_length": 3517.135498046875,
196
+ "epoch": 0.41025641025641024,
197
+ "grad_norm": 0.11326687037944794,
198
+ "kl": 0.00015026330947875977,
199
+ "learning_rate": 7.75e-07,
200
+ "loss": 0.0726,
201
+ "reward": 0.3020833428017795,
202
+ "reward_std": 0.2878086185082793,
203
+ "rewards/accuracy_reward": 0.19791667303070426,
204
+ "rewards/tag_count_reward": 0.10416666837409139,
205
+ "step": 14
206
+ },
207
+ {
208
+ "clip_ratio": 0.0,
209
+ "completion_length": 3760.557373046875,
210
+ "epoch": 0.43956043956043955,
211
+ "grad_norm": 0.06679689884185791,
212
+ "kl": 0.00015845894813537598,
213
+ "learning_rate": 7.330314893841101e-07,
214
+ "loss": 0.0487,
215
+ "reward": 0.1692708395421505,
216
+ "reward_std": 0.19603674625977874,
217
+ "rewards/accuracy_reward": 0.10937500465661287,
218
+ "rewards/tag_count_reward": 0.05989583348855376,
219
+ "step": 15
220
+ },
221
+ {
222
+ "clip_ratio": 0.0,
223
+ "completion_length": 3544.5365600585938,
224
+ "epoch": 0.46886446886446886,
225
+ "grad_norm": 0.08167487382888794,
226
+ "kl": 0.00022110342979431152,
227
+ "learning_rate": 6.890576474687263e-07,
228
+ "loss": 0.0531,
229
+ "reward": 0.2864583423361182,
230
+ "reward_std": 0.2276854459196329,
231
+ "rewards/accuracy_reward": 0.19270833721384406,
232
+ "rewards/tag_count_reward": 0.09375000314321369,
233
+ "step": 16
234
+ },
235
+ {
236
+ "clip_ratio": 0.0,
237
+ "completion_length": 3696.0521850585938,
238
+ "epoch": 0.4981684981684982,
239
+ "grad_norm": 0.11336914449930191,
240
+ "kl": 0.0003203749656677246,
241
+ "learning_rate": 6.435602608679916e-07,
242
+ "loss": 0.0661,
243
+ "reward": 0.2500000058207661,
244
+ "reward_std": 0.22962875291705132,
245
+ "rewards/accuracy_reward": 0.1666666679084301,
246
+ "rewards/tag_count_reward": 0.08333333558402956,
247
+ "step": 17
248
+ },
249
+ {
250
+ "clip_ratio": 0.0,
251
+ "completion_length": 3608.9584045410156,
252
+ "epoch": 0.5274725274725275,
253
+ "grad_norm": 0.08060825616121292,
254
+ "kl": 0.0004144906997680664,
255
+ "learning_rate": 5.97037808470444e-07,
256
+ "loss": 0.0568,
257
+ "reward": 0.3007812607102096,
258
+ "reward_std": 0.2574492711573839,
259
+ "rewards/accuracy_reward": 0.2135416711680591,
260
+ "rewards/tag_count_reward": 0.08723958604969084,
261
+ "step": 18
262
+ },
263
+ {
264
+ "clip_ratio": 0.0,
265
+ "completion_length": 3601.6927795410156,
266
+ "epoch": 0.5567765567765568,
267
+ "grad_norm": 0.07288842648267746,
268
+ "kl": 0.0003364086151123047,
269
+ "learning_rate": 5.5e-07,
270
+ "loss": 0.0319,
271
+ "reward": 0.376302097691223,
272
+ "reward_std": 0.2284244946204126,
273
+ "rewards/accuracy_reward": 0.281250003259629,
274
+ "rewards/tag_count_reward": 0.09505208558402956,
275
+ "step": 19
276
+ },
277
+ {
278
+ "clip_ratio": 0.0,
279
+ "completion_length": 3625.0208740234375,
280
+ "epoch": 0.5860805860805861,
281
+ "grad_norm": 0.07371893525123596,
282
+ "kl": 0.0006115436553955078,
283
+ "learning_rate": 5.02962191529556e-07,
284
+ "loss": 0.0335,
285
+ "reward": 0.2552083439659327,
286
+ "reward_std": 0.25146263325586915,
287
+ "rewards/accuracy_reward": 0.17187500791624188,
288
+ "rewards/tag_count_reward": 0.08333333511836827,
289
+ "step": 20
290
+ },
291
+ {
292
+ "clip_ratio": 0.0,
293
+ "completion_length": 3542.1354370117188,
294
+ "epoch": 0.6153846153846154,
295
+ "grad_norm": 0.10749542713165283,
296
+ "kl": 0.00035321712493896484,
297
+ "learning_rate": 4.5643973913200837e-07,
298
+ "loss": 0.0995,
299
+ "reward": 0.373697929084301,
300
+ "reward_std": 0.31743163615465164,
301
+ "rewards/accuracy_reward": 0.281250006519258,
302
+ "rewards/tag_count_reward": 0.09244791977107525,
303
+ "step": 21
304
+ },
305
+ {
306
+ "clip_ratio": 0.0,
307
+ "completion_length": 3414.6927795410156,
308
+ "epoch": 0.6446886446886447,
309
+ "grad_norm": 0.08405382186174393,
310
+ "kl": 0.00036454200744628906,
311
+ "learning_rate": 4.1094235253127374e-07,
312
+ "loss": 0.075,
313
+ "reward": 0.37760418374091387,
314
+ "reward_std": 0.2682782169431448,
315
+ "rewards/accuracy_reward": 0.2708333353511989,
316
+ "rewards/tag_count_reward": 0.1067708374466747,
317
+ "step": 22
318
+ },
319
+ {
320
+ "clip_ratio": 0.0,
321
+ "completion_length": 3678.791748046875,
322
+ "epoch": 0.673992673992674,
323
+ "grad_norm": 0.07961313426494598,
324
+ "kl": 0.0004169940948486328,
325
+ "learning_rate": 3.6696851061588994e-07,
326
+ "loss": 0.0502,
327
+ "reward": 0.2786458395421505,
328
+ "reward_std": 0.24647203274071217,
329
+ "rewards/accuracy_reward": 0.1979166711680591,
330
+ "rewards/tag_count_reward": 0.08072916697710752,
331
+ "step": 23
332
+ },
333
+ {
334
+ "clip_ratio": 0.0,
335
+ "completion_length": 3838.8334045410156,
336
+ "epoch": 0.7032967032967034,
337
+ "grad_norm": 0.06324659287929535,
338
+ "kl": 0.0005202293395996094,
339
+ "learning_rate": 3.250000000000001e-07,
340
+ "loss": 0.0313,
341
+ "reward": 0.13541667081881315,
342
+ "reward_std": 0.16657467489130795,
343
+ "rewards/accuracy_reward": 0.08854167023673654,
344
+ "rewards/tag_count_reward": 0.04687500209547579,
345
+ "step": 24
346
+ },
347
+ {
348
+ "clip_ratio": 0.0,
349
+ "completion_length": 3507.5677490234375,
350
+ "epoch": 0.7326007326007326,
351
+ "grad_norm": 0.085300974547863,
352
+ "kl": 0.0008640289306640625,
353
+ "learning_rate": 2.854966364683872e-07,
354
+ "loss": 0.0561,
355
+ "reward": 0.29427084082271904,
356
+ "reward_std": 0.19792527868412435,
357
+ "rewards/accuracy_reward": 0.20833333488553762,
358
+ "rewards/tag_count_reward": 0.0859375026775524,
359
+ "step": 25
360
+ },
361
+ {
362
+ "clip_ratio": 0.0,
363
+ "completion_length": 3683.151123046875,
364
+ "epoch": 0.7619047619047619,
365
+ "grad_norm": 0.06933189183473587,
366
+ "kl": 0.0010666847229003906,
367
+ "learning_rate": 2.488912271385139e-07,
368
+ "loss": 0.0326,
369
+ "reward": 0.272135422565043,
370
+ "reward_std": 0.2521828315220773,
371
+ "rewards/accuracy_reward": 0.1927083353511989,
372
+ "rewards/tag_count_reward": 0.0794270858168602,
373
+ "step": 26
374
+ },
375
+ {
376
+ "clip_ratio": 0.0,
377
+ "completion_length": 3190.5000610351562,
378
+ "epoch": 0.7912087912087912,
379
+ "grad_norm": 0.12118662893772125,
380
+ "kl": 0.0013322830200195312,
381
+ "learning_rate": 2.1558482853517253e-07,
382
+ "loss": 0.0751,
383
+ "reward": 0.40755209885537624,
384
+ "reward_std": 0.31804133765399456,
385
+ "rewards/accuracy_reward": 0.2604166753590107,
386
+ "rewards/tag_count_reward": 0.1471354179084301,
387
+ "step": 27
388
+ },
389
+ {
390
+ "clip_ratio": 0.0,
391
+ "completion_length": 3750.2240600585938,
392
+ "epoch": 0.8205128205128205,
393
+ "grad_norm": 0.07822524011135101,
394
+ "kl": 0.0015530586242675781,
395
+ "learning_rate": 1.8594235253127372e-07,
396
+ "loss": 0.0456,
397
+ "reward": 0.2773437611758709,
398
+ "reward_std": 0.25566962361335754,
399
+ "rewards/accuracy_reward": 0.1927083395421505,
400
+ "rewards/tag_count_reward": 0.08463541837409139,
401
+ "step": 28
402
+ },
403
+ {
404
+ "clip_ratio": 0.0,
405
+ "completion_length": 3394.760498046875,
406
+ "epoch": 0.8498168498168498,
407
+ "grad_norm": 0.12853899598121643,
408
+ "kl": 0.0016031265258789062,
409
+ "learning_rate": 1.6028856829700258e-07,
410
+ "loss": 0.0882,
411
+ "reward": 0.43750001303851604,
412
+ "reward_std": 0.33361043129116297,
413
+ "rewards/accuracy_reward": 0.31770833767950535,
414
+ "rewards/tag_count_reward": 0.1197916716337204,
415
+ "step": 29
416
+ },
417
+ {
418
+ "clip_ratio": 0.0,
419
+ "completion_length": 3737.7344360351562,
420
+ "epoch": 0.8791208791208791,
421
+ "grad_norm": 0.07847777009010315,
422
+ "kl": 0.0011386871337890625,
423
+ "learning_rate": 1.3890454406082956e-07,
424
+ "loss": 0.0195,
425
+ "reward": 0.24479167815297842,
426
+ "reward_std": 0.23484968207776546,
427
+ "rewards/accuracy_reward": 0.17187500931322575,
428
+ "rewards/tag_count_reward": 0.07291666860692203,
429
+ "step": 30
430
+ },
431
+ {
432
+ "clip_ratio": 0.0,
433
+ "completion_length": 3828.666778564453,
434
+ "epoch": 0.9084249084249084,
435
+ "grad_norm": 0.06969457864761353,
436
+ "kl": 0.0012054443359375,
437
+ "learning_rate": 1.220245676671809e-07,
438
+ "loss": 0.0396,
439
+ "reward": 0.17578125256113708,
440
+ "reward_std": 0.19791242526844144,
441
+ "rewards/accuracy_reward": 0.12500000558793545,
442
+ "rewards/tag_count_reward": 0.050781250931322575,
443
+ "step": 31
444
+ },
445
+ {
446
+ "clip_ratio": 0.0,
447
+ "completion_length": 3739.2969360351562,
448
+ "epoch": 0.9377289377289377,
449
+ "grad_norm": 0.08980078995227814,
450
+ "kl": 0.001861572265625,
451
+ "learning_rate": 1.0983357966978745e-07,
452
+ "loss": 0.0435,
453
+ "reward": 0.28906251152511686,
454
+ "reward_std": 0.2909349126275629,
455
+ "rewards/accuracy_reward": 0.20312500838190317,
456
+ "rewards/tag_count_reward": 0.08593750221189111,
457
+ "step": 32
458
+ },
459
+ {
460
+ "clip_ratio": 0.0,
461
+ "completion_length": 3494.8907470703125,
462
+ "epoch": 0.967032967032967,
463
+ "grad_norm": 0.16050584614276886,
464
+ "kl": 0.00262451171875,
465
+ "learning_rate": 1.0246514708427701e-07,
466
+ "loss": 0.1098,
467
+ "reward": 0.281250006868504,
468
+ "reward_std": 0.2926723880227655,
469
+ "rewards/accuracy_reward": 0.17187500465661287,
470
+ "rewards/tag_count_reward": 0.10937500221189111,
471
+ "step": 33
472
+ },
473
+ {
474
+ "clip_ratio": 0.0,
475
+ "completion_length": 3986.875,
476
+ "epoch": 0.9963369963369964,
477
+ "grad_norm": 0.08669467270374298,
478
+ "kl": 0.0013146400451660156,
479
+ "learning_rate": 1e-07,
480
+ "loss": 0.0431,
481
+ "reward": 0.2903645932674408,
482
+ "reward_std": 0.3158208169043064,
483
+ "rewards/accuracy_reward": 0.20312500838190317,
484
+ "rewards/tag_count_reward": 0.0872395858168602,
485
+ "step": 34
486
+ },
487
+ {
488
+ "epoch": 0.9963369963369964,
489
+ "step": 34,
490
+ "total_flos": 0.0,
491
+ "train_loss": 0.059163046507712674,
492
+ "train_runtime": 9090.9077,
493
+ "train_samples_per_second": 0.09,
494
+ "train_steps_per_second": 0.004
495
+ }
496
+ ],
497
+ "logging_steps": 1,
498
+ "max_steps": 34,
499
+ "num_input_tokens_seen": 0,
500
+ "num_train_epochs": 1,
501
+ "save_steps": 100,
502
+ "stateful_callbacks": {
503
+ "TrainerControl": {
504
+ "args": {
505
+ "should_epoch_stop": false,
506
+ "should_evaluate": false,
507
+ "should_log": false,
508
+ "should_save": true,
509
+ "should_training_stop": true
510
+ },
511
+ "attributes": {}
512
+ }
513
+ },
514
+ "total_flos": 0.0,
515
+ "train_batch_size": 8,
516
+ "trial_name": null,
517
+ "trial_params": null
518
+ }