Hugging Face's logo

ghost613
/
gemma2_on_korean_summary

PEFT
Safetensors
Generated from Trainer
Model card Files Community
ghost613 commited on
Commit
e2028b6
·
verified ·
1 Parent(s): 2b51d4b

gemma2_on_korean_summary

Browse files
Files changed (4) hide show
  1. README.md +52 -153
  2. adapter_config.json +7 -6
  3. adapter_model.safetensors +2 -2
  4. training_args.bin +2 -2
README.md CHANGED
@@ -12,12 +12,11 @@ model-index:
12
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
  should probably proofread and complete it, then remove this comment. -->
14
 
15
- [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/ghost_in_the_lab/Korean-fine-tune-models/runs/fc7yrozn)
16
  # gemma2_on_korean_summary
17
 
18
  This model is a fine-tuned version of [beomi/gemma-ko-2b](https://huggingface.co/beomi/gemma-ko-2b) on the None dataset.
19
  It achieves the following results on the evaluation set:
20
- - Loss: 0.9709
21
 
22
  ## Model description
23
 
@@ -37,167 +36,67 @@ More information needed
37
 
38
  The following hyperparameters were used during training:
39
  - learning_rate: 5e-05
40
- - train_batch_size: 1
41
- - eval_batch_size: 1
42
  - seed: 42
43
  - gradient_accumulation_steps: 5
44
- - total_train_batch_size: 5
45
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
46
  - lr_scheduler_type: linear
47
  - lr_scheduler_warmup_steps: 50
48
- - training_steps: 1400
49
  - mixed_precision_training: Native AMP
50
 
51
  ### Training results
52
 
53
- | Training Loss | Epoch | Step | Validation Loss |
54
- |:-------------:|:------:|:----:|:---------------:|
55
- | 1.6915 | 0.0658 | 10 | 1.6222 |
56
- | 1.6475 | 0.1316 | 20 | 1.5357 |
57
- | 1.4535 | 0.1974 | 30 | 1.4325 |
58
- | 1.414 | 0.2632 | 40 | 1.3366 |
59
- | 1.3122 | 0.3289 | 50 | 1.2576 |
60
- | 1.2768 | 0.3947 | 60 | 1.2126 |
61
- | 1.174 | 0.4605 | 70 | 1.1838 |
62
- | 1.1816 | 0.5263 | 80 | 1.1575 |
63
- | 1.1163 | 0.5921 | 90 | 1.1330 |
64
- | 1.1197 | 0.6579 | 100 | 1.1116 |
65
- | 1.1635 | 0.7237 | 110 | 1.0954 |
66
- | 1.0971 | 0.7895 | 120 | 1.0730 |
67
- | 1.0985 | 0.8553 | 130 | 1.0568 |
68
- | 1.0725 | 0.9211 | 140 | 1.0414 |
69
- | 1.0483 | 0.9868 | 150 | 1.0302 |
70
- | 0.9666 | 1.0526 | 160 | 1.0203 |
71
- | 0.9754 | 1.1184 | 170 | 1.0096 |
72
- | 0.9688 | 1.1842 | 180 | 0.9976 |
73
- | 0.9673 | 1.25 | 190 | 0.9874 |
74
- | 0.9636 | 1.3158 | 200 | 0.9805 |
75
- | 0.9482 | 1.3816 | 210 | 0.9744 |
76
- | 0.9231 | 1.4474 | 220 | 0.9657 |
77
- | 0.9208 | 1.5132 | 230 | 0.9555 |
78
- | 0.9321 | 1.5789 | 240 | 0.9488 |
79
- | 0.9362 | 1.6447 | 250 | 0.9431 |
80
- | 0.939 | 1.7105 | 260 | 0.9393 |
81
- | 0.919 | 1.7763 | 270 | 0.9342 |
82
- | 0.9277 | 1.8421 | 280 | 0.9312 |
83
- | 0.8955 | 1.9079 | 290 | 0.9263 |
84
- | 0.8679 | 1.9737 | 300 | 0.9211 |
85
- | 0.8545 | 2.0395 | 310 | 0.9251 |
86
- | 0.7897 | 2.1053 | 320 | 0.9200 |
87
- | 0.7835 | 2.1711 | 330 | 0.9198 |
88
- | 0.8139 | 2.2368 | 340 | 0.9105 |
89
- | 0.7861 | 2.3026 | 350 | 0.9089 |
90
- | 0.7752 | 2.3684 | 360 | 0.9075 |
91
- | 0.7919 | 2.4342 | 370 | 0.8985 |
92
- | 0.7433 | 2.5 | 380 | 0.9038 |
93
- | 0.7905 | 2.5658 | 390 | 0.8955 |
94
- | 0.7763 | 2.6316 | 400 | 0.8930 |
95
- | 0.792 | 2.6974 | 410 | 0.8869 |
96
- | 0.7854 | 2.7632 | 420 | 0.8834 |
97
- | 0.7978 | 2.8289 | 430 | 0.8770 |
98
- | 0.7864 | 2.8947 | 440 | 0.8780 |
99
- | 0.8007 | 2.9605 | 450 | 0.8730 |
100
- | 0.7686 | 3.0263 | 460 | 0.8760 |
101
- | 0.6573 | 3.0921 | 470 | 0.8888 |
102
- | 0.7183 | 3.1579 | 480 | 0.8833 |
103
- | 0.6644 | 3.2237 | 490 | 0.8864 |
104
- | 0.6648 | 3.2895 | 500 | 0.8834 |
105
- | 0.6763 | 3.3553 | 510 | 0.8814 |
106
- | 0.6844 | 3.4211 | 520 | 0.8824 |
107
- | 0.6796 | 3.4868 | 530 | 0.8769 |
108
- | 0.6748 | 3.5526 | 540 | 0.8708 |
109
- | 0.6899 | 3.6184 | 550 | 0.8688 |
110
- | 0.6866 | 3.6842 | 560 | 0.8747 |
111
- | 0.7 | 3.75 | 570 | 0.8645 |
112
- | 0.6896 | 3.8158 | 580 | 0.8703 |
113
- | 0.7176 | 3.8816 | 590 | 0.8628 |
114
- | 0.6517 | 3.9474 | 600 | 0.8655 |
115
- | 0.6941 | 4.0132 | 610 | 0.8641 |
116
- | 0.5873 | 4.0789 | 620 | 0.8889 |
117
- | 0.6208 | 4.1447 | 630 | 0.8772 |
118
- | 0.6067 | 4.2105 | 640 | 0.8844 |
119
- | 0.5892 | 4.2763 | 650 | 0.8798 |
120
- | 0.6059 | 4.3421 | 660 | 0.8830 |
121
- | 0.6129 | 4.4079 | 670 | 0.8810 |
122
- | 0.6104 | 4.4737 | 680 | 0.8799 |
123
- | 0.5846 | 4.5395 | 690 | 0.8763 |
124
- | 0.6141 | 4.6053 | 700 | 0.8807 |
125
- | 0.6467 | 4.6711 | 710 | 0.8766 |
126
- | 0.634 | 4.7368 | 720 | 0.8774 |
127
- | 0.5976 | 4.8026 | 730 | 0.8680 |
128
- | 0.5638 | 4.8684 | 740 | 0.8742 |
129
- | 0.6067 | 4.9342 | 750 | 0.8733 |
130
- | 0.6219 | 5.0 | 760 | 0.8644 |
131
- | 0.5169 | 5.0658 | 770 | 0.8969 |
132
- | 0.5726 | 5.1316 | 780 | 0.9012 |
133
- | 0.5483 | 5.1974 | 790 | 0.9031 |
134
- | 0.5197 | 5.2632 | 800 | 0.8976 |
135
- | 0.5479 | 5.3289 | 810 | 0.8963 |
136
- | 0.5631 | 5.3947 | 820 | 0.8962 |
137
- | 0.5687 | 5.4605 | 830 | 0.9009 |
138
- | 0.4825 | 5.5263 | 840 | 0.8982 |
139
- | 0.5305 | 5.5921 | 850 | 0.8937 |
140
- | 0.5743 | 5.6579 | 860 | 0.8945 |
141
- | 0.5293 | 5.7237 | 870 | 0.8951 |
142
- | 0.5169 | 5.7895 | 880 | 0.9034 |
143
- | 0.5585 | 5.8553 | 890 | 0.8894 |
144
- | 0.5373 | 5.9211 | 900 | 0.8936 |
145
- | 0.5524 | 5.9868 | 910 | 0.8876 |
146
- | 0.4815 | 6.0526 | 920 | 0.9187 |
147
- | 0.47 | 6.1184 | 930 | 0.9200 |
148
- | 0.4694 | 6.1842 | 940 | 0.9204 |
149
- | 0.5035 | 6.25 | 950 | 0.9246 |
150
- | 0.4852 | 6.3158 | 960 | 0.9232 |
151
- | 0.5266 | 6.3816 | 970 | 0.9257 |
152
- | 0.4907 | 6.4474 | 980 | 0.9232 |
153
- | 0.5139 | 6.5132 | 990 | 0.9135 |
154
- | 0.464 | 6.5789 | 1000 | 0.9207 |
155
- | 0.5172 | 6.6447 | 1010 | 0.9128 |
156
- | 0.4948 | 6.7105 | 1020 | 0.9244 |
157
- | 0.4606 | 6.7763 | 1030 | 0.9171 |
158
- | 0.491 | 6.8421 | 1040 | 0.9187 |
159
- | 0.4641 | 6.9079 | 1050 | 0.9157 |
160
- | 0.4684 | 6.9737 | 1060 | 0.9115 |
161
- | 0.4625 | 7.0395 | 1070 | 0.9299 |
162
- | 0.4324 | 7.1053 | 1080 | 0.9454 |
163
- | 0.4143 | 7.1711 | 1090 | 0.9446 |
164
- | 0.4357 | 7.2368 | 1100 | 0.9447 |
165
- | 0.4471 | 7.3026 | 1110 | 0.9459 |
166
- | 0.4691 | 7.3684 | 1120 | 0.9441 |
167
- | 0.4556 | 7.4342 | 1130 | 0.9471 |
168
- | 0.4296 | 7.5 | 1140 | 0.9406 |
169
- | 0.4323 | 7.5658 | 1150 | 0.9439 |
170
- | 0.4243 | 7.6316 | 1160 | 0.9430 |
171
- | 0.4583 | 7.6974 | 1170 | 0.9435 |
172
- | 0.4346 | 7.7632 | 1180 | 0.9405 |
173
- | 0.4747 | 7.8289 | 1190 | 0.9406 |
174
- | 0.4443 | 7.8947 | 1200 | 0.9405 |
175
- | 0.4418 | 7.9605 | 1210 | 0.9424 |
176
- | 0.3878 | 8.0263 | 1220 | 0.9464 |
177
- | 0.4014 | 8.0921 | 1230 | 0.9721 |
178
- | 0.4183 | 8.1579 | 1240 | 0.9647 |
179
- | 0.4103 | 8.2237 | 1250 | 0.9672 |
180
- | 0.3951 | 8.2895 | 1260 | 0.9702 |
181
- | 0.4488 | 8.3553 | 1270 | 0.9648 |
182
- | 0.41 | 8.4211 | 1280 | 0.9653 |
183
- | 0.3726 | 8.4868 | 1290 | 0.9668 |
184
- | 0.395 | 8.5526 | 1300 | 0.9644 |
185
- | 0.4141 | 8.6184 | 1310 | 0.9637 |
186
- | 0.3774 | 8.6842 | 1320 | 0.9659 |
187
- | 0.4003 | 8.75 | 1330 | 0.9653 |
188
- | 0.3841 | 8.8158 | 1340 | 0.9671 |
189
- | 0.4202 | 8.8816 | 1350 | 0.9688 |
190
- | 0.4226 | 8.9474 | 1360 | 0.9684 |
191
- | 0.3914 | 9.0132 | 1370 | 0.9673 |
192
- | 0.4105 | 9.0789 | 1380 | 0.9687 |
193
- | 0.4021 | 9.1447 | 1390 | 0.9702 |
194
- | 0.4008 | 9.2105 | 1400 | 0.9709 |
195
 
196
 
197
  ### Framework versions
198
 
199
- - PEFT 0.7.1
200
- - Transformers 4.41.0.dev0
201
- - Pytorch 2.1.2
202
- - Datasets 2.15.0
203
- - Tokenizers 0.19.1
 
12
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
  should probably proofread and complete it, then remove this comment. -->
14
 
 
15
  # gemma2_on_korean_summary
16
 
17
  This model is a fine-tuned version of [beomi/gemma-ko-2b](https://huggingface.co/beomi/gemma-ko-2b) on the None dataset.
18
  It achieves the following results on the evaluation set:
19
+ - Loss: 0.9622
20
 
21
  ## Model description
22
 
 
36
 
37
  The following hyperparameters were used during training:
38
  - learning_rate: 5e-05
39
+ - train_batch_size: 2
40
+ - eval_batch_size: 2
41
  - seed: 42
42
  - gradient_accumulation_steps: 5
43
+ - total_train_batch_size: 10
44
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
  - lr_scheduler_type: linear
46
  - lr_scheduler_warmup_steps: 50
47
+ - training_steps: 800
48
  - mixed_precision_training: Native AMP
49
 
50
  ### Training results
51
 
52
+ | Training Loss | Epoch | Step | Validation Loss |
53
+ |:-------------:|:-----:|:----:|:---------------:|
54
+ | 1.6606 | 0.26 | 20 | 1.5454 |
55
+ | 1.4381 | 0.53 | 40 | 1.3247 |
56
+ | 1.2548 | 0.79 | 60 | 1.1921 |
57
+ | 1.1574 | 1.05 | 80 | 1.1227 |
58
+ | 1.0968 | 1.32 | 100 | 1.0727 |
59
+ | 1.0485 | 1.58 | 120 | 1.0316 |
60
+ | 1.0258 | 1.84 | 140 | 1.0019 |
61
+ | 0.9582 | 2.11 | 160 | 0.9785 |
62
+ | 0.906 | 2.37 | 180 | 0.9575 |
63
+ | 0.8837 | 2.63 | 200 | 0.9429 |
64
+ | 0.8763 | 2.89 | 220 | 0.9247 |
65
+ | 0.8295 | 3.16 | 240 | 0.9213 |
66
+ | 0.7799 | 3.42 | 260 | 0.9122 |
67
+ | 0.7742 | 3.68 | 280 | 0.8992 |
68
+ | 0.7708 | 3.95 | 300 | 0.8918 |
69
+ | 0.7196 | 4.21 | 320 | 0.8952 |
70
+ | 0.6908 | 4.47 | 340 | 0.8917 |
71
+ | 0.6977 | 4.74 | 360 | 0.8841 |
72
+ | 0.6789 | 5.0 | 380 | 0.8764 |
73
+ | 0.6198 | 5.26 | 400 | 0.9003 |
74
+ | 0.6203 | 5.53 | 420 | 0.9030 |
75
+ | 0.6169 | 5.79 | 440 | 0.8913 |
76
+ | 0.6111 | 6.05 | 460 | 0.8935 |
77
+ | 0.564 | 6.32 | 480 | 0.9096 |
78
+ | 0.5819 | 6.58 | 500 | 0.9027 |
79
+ | 0.5673 | 6.84 | 520 | 0.8997 |
80
+ | 0.5382 | 7.11 | 540 | 0.9181 |
81
+ | 0.5228 | 7.37 | 560 | 0.9197 |
82
+ | 0.5197 | 7.63 | 580 | 0.9254 |
83
+ | 0.5319 | 7.89 | 600 | 0.9158 |
84
+ | 0.4954 | 8.16 | 620 | 0.9394 |
85
+ | 0.4954 | 8.42 | 640 | 0.9345 |
86
+ | 0.4719 | 8.68 | 660 | 0.9388 |
87
+ | 0.4866 | 8.95 | 680 | 0.9390 |
88
+ | 0.4739 | 9.21 | 700 | 0.9528 |
89
+ | 0.4591 | 9.47 | 720 | 0.9534 |
90
+ | 0.4478 | 9.74 | 740 | 0.9542 |
91
+ | 0.4547 | 10.0 | 760 | 0.9539 |
92
+ | 0.4379 | 10.26 | 780 | 0.9617 |
93
+ | 0.4348 | 10.53 | 800 | 0.9622 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
94
 
95
 
96
  ### Framework versions
97
 
98
+ - PEFT 0.8.2
99
+ - Transformers 4.38.2
100
+ - Pytorch 2.1.2+cu121
101
+ - Datasets 2.17.0
102
+ - Tokenizers 0.15.0
adapter_config.json CHANGED
@@ -15,17 +15,18 @@
15
  "megatron_core": "megatron.core",
16
  "modules_to_save": null,
17
  "peft_type": "LORA",
18
- "r": 16,
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
22
  "up_proj",
 
23
  "q_proj",
24
- "o_proj",
25
- "v_proj",
26
  "gate_proj",
27
- "k_proj",
28
- "down_proj"
 
29
  ],
30
- "task_type": "CAUSAL_LM"
 
31
  }
 
15
  "megatron_core": "megatron.core",
16
  "modules_to_save": null,
17
  "peft_type": "LORA",
18
+ "r": 32,
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
22
  "up_proj",
23
+ "k_proj",
24
  "q_proj",
 
 
25
  "gate_proj",
26
+ "down_proj",
27
+ "v_proj",
28
+ "o_proj"
29
  ],
30
+ "task_type": "CAUSAL_LM",
31
+ "use_rslora": false
32
  }
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2314cf6b52c9b7c617802a576429a203fc1ad4de245b8c6d967c01eaa2b648b3
3
- size 78480072
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a9ce050e4e4496519431daaf68b3ed0aebfbb4002facbbaf7ae1da64ad590a53
3
+ size 156926880
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f1c7323d160e89f2a6ac6956b9d72d538e7f5f682f87c23706f644fa0f8fd29d
3
- size 5112
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b8b0f18188621a32025e61f3adc60296fd9fe2c790c228b5889a71cfafbc7876
3
+ size 4920