File size: 793 Bytes
c03e4f3 cf05ca1 c03e4f3 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
---
tags:
- fp8
base_model: mgoin/Nemotron-4-340B-Base-hf
---
```
lm_eval --model vllm --model_args pretrained=/home/mgoin/code/Nemotron-4-340B-Base-hf-FP8,tensor_parallel_size=8,distributed_executor_backend="ray",max_model_len=4096 --tasks gsm8k --num_fewshot 5 --batch_size auto
vllm (pretrained=/home/mgoin/code/Nemotron-4-340B-Base-hf-FP8,tensor_parallel_size=8,distributed_executor_backend=ray,max_model_len=4096), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.2949|± |0.0126|
| | |strict-match | 5|exact_match|↑ |0.1600|± |0.0101|
``` |