Update README.md
Browse files
README.md
CHANGED
|
@@ -156,7 +156,17 @@ lm_eval --model hf --model_args pretrained=microsoft/Phi-4-mini-instruct --tasks
|
|
| 156 |
|
| 157 |
## int8 dynamic activation and int4 weight quantization (8da4w)
|
| 158 |
```
|
| 159 |
-
lm_eval
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 160 |
```
|
| 161 |
|
| 162 |
| Benchmark | | |
|
|
@@ -187,7 +197,7 @@ We can run the quantized model on a mobile phone using [ExecuTorch](https://gith
|
|
| 187 |
Once ExecuTorch is [set-up](https://pytorch.org/executorch/main/getting-started.html), exporting and running the model on device is a breeze.
|
| 188 |
|
| 189 |
We first convert the quantized checkpoint to one ExecuTorch's LLM export script expects by renaming some of the checkpoint keys.
|
| 190 |
-
The following script does this for you.
|
| 191 |
```
|
| 192 |
python -m executorch.examples.models.phi_4_mini.convert_weights pytorch_model.bin phi4-mini-8da4w-converted.bin
|
| 193 |
```
|
|
|
|
| 156 |
|
| 157 |
## int8 dynamic activation and int4 weight quantization (8da4w)
|
| 158 |
```
|
| 159 |
+
import lm_eval
|
| 160 |
+
from lm_eval import evaluator
|
| 161 |
+
from lm_eval.utils import (
|
| 162 |
+
make_table,
|
| 163 |
+
)
|
| 164 |
+
|
| 165 |
+
lm_eval_model = lm_eval.models.huggingface.HFLM(pretrained=quantized_model, batch_size=64)
|
| 166 |
+
results = evaluator.simple_evaluate(
|
| 167 |
+
lm_eval_model, tasks=["hellaswag"], device="cuda:0", batch_size="auto"
|
| 168 |
+
)
|
| 169 |
+
print(make_table(results))
|
| 170 |
```
|
| 171 |
|
| 172 |
| Benchmark | | |
|
|
|
|
| 197 |
Once ExecuTorch is [set-up](https://pytorch.org/executorch/main/getting-started.html), exporting and running the model on device is a breeze.
|
| 198 |
|
| 199 |
We first convert the quantized checkpoint to one ExecuTorch's LLM export script expects by renaming some of the checkpoint keys.
|
| 200 |
+
The following script does this for you. We have uploaded phi4-mini-8da4w-converted.bin here for convenience.
|
| 201 |
```
|
| 202 |
python -m executorch.examples.models.phi_4_mini.convert_weights pytorch_model.bin phi4-mini-8da4w-converted.bin
|
| 203 |
```
|