Update README.md
Browse files
README.md
CHANGED
|
@@ -174,30 +174,30 @@ Need to install lm-eval from source: https://github.com/EleutherAI/lm-evaluation
|
|
| 174 |
|
| 175 |
## baseline
|
| 176 |
```Shell
|
| 177 |
-
lm_eval --model hf --model_args pretrained=
|
| 178 |
```
|
| 179 |
|
| 180 |
## int8 dynamic activation and int4 weight quantization (8da4w)
|
| 181 |
```Shell
|
| 182 |
-
lm_eval --model hf --model_args pretrained=
|
| 183 |
```
|
| 184 |
|
| 185 |
| Benchmark | | |
|
| 186 |
|----------------------------------|----------------|---------------------------|
|
| 187 |
-
| | Qwen3-4B | Qwen3-4B-8da4w
|
| 188 |
| **Popular aggregated benchmark** | | |
|
| 189 |
-
| mmlu
|
| 190 |
-
| mmlu_pro
|
| 191 |
-
| bbh
|
| 192 |
| **Reasoning** | | |
|
| 193 |
-
| gpqa_main_zeroshot |
|
| 194 |
-
| mgsm_en_cot_en
|
| 195 |
| **Multilingual** | | |
|
| 196 |
-
| m_mmlu
|
| 197 |
| **Math** | | |
|
| 198 |
-
| gsm8k
|
| 199 |
-
| leaderboard_math_hard
|
| 200 |
-
| **Overall** |
|
| 201 |
|
| 202 |
|
| 203 |
# Exporting to ExecuTorch
|
|
|
|
| 174 |
|
| 175 |
## baseline
|
| 176 |
```Shell
|
| 177 |
+
lm_eval --model hf --model_args pretrained=Qwen3/Qwen3-4B --tasks hellaswag --device cuda:0 --batch_size auto
|
| 178 |
```
|
| 179 |
|
| 180 |
## int8 dynamic activation and int4 weight quantization (8da4w)
|
| 181 |
```Shell
|
| 182 |
+
lm_eval --model hf --model_args pretrained=TODO:ADD LINK --tasks hellaswag --device cuda:0 --batch_size auto
|
| 183 |
```
|
| 184 |
|
| 185 |
| Benchmark | | |
|
| 186 |
|----------------------------------|----------------|---------------------------|
|
| 187 |
+
| | Qwen3-4B | Qwen3-4B-8da4w |
|
| 188 |
| **Popular aggregated benchmark** | | |
|
| 189 |
+
| mmlu | 68.38 | 66.74 |
|
| 190 |
+
| mmlu_pro | 49.71 | 46.73 |
|
| 191 |
+
| bbh | 74.86 | 67.47 |
|
| 192 |
| **Reasoning** | | |
|
| 193 |
+
| gpqa_main_zeroshot | 33.93 | 31.03 |
|
| 194 |
+
| mgsm_en_cot_en | 30.40 | 29.20 |
|
| 195 |
| **Multilingual** | | |
|
| 196 |
+
| m_mmlu | 50.41 | 47.13 |
|
| 197 |
| **Math** | | |
|
| 198 |
+
| gsm8k | 84.76 | 82.87 |
|
| 199 |
+
| leaderboard_math_hard | 62.83 | 53.30 |
|
| 200 |
+
| **Overall** | 56.91 | 53.06 |
|
| 201 |
|
| 202 |
|
| 203 |
# Exporting to ExecuTorch
|