Update README.md

README.md

@@ -118,17 +118,133 @@ oneshot(
 
 ## Evaluation
 
-The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard), OpenLLM Leaderboard [V2](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/) and on [HumanEval](https://github.com/neuralmagic/evalplus), using the following commands:
+The model was evaluated using [mistral-evals](https://github.com/neuralmagic/mistral-evals) for vision-related tasks and using [lm_evaluation_harness](https://github.com/neuralmagic/lm-evaluation-harness) for select text-based benchmarks. The evaluations were conducted using the following commands:
 
 <details>
 <summary>Evaluation Commands</summary>
+  
+### Vision Tasks
+- vqav2
+- docvqa
+- mathvista
+- mmmu
+- chartqa
+
+```
+vllm serve neuralmagic/pixtral-12b-quantized.w8a8 --tensor_parallel_size 1 --max_model_len 25000 --trust_remote_code --max_num_seqs 8 --gpu_memory_utilization 0.9 --dtype float16 --limit_mm_per_prompt image=7
+
+python -m eval.run eval_vllm \
+        --model_name neuralmagic/pixtral-12b-quantized.w4a16 \
+        --url http://0.0.0.0:8000 \
+        --output_dir ~/tmp
+        --eval_name <vision_task_name>
+```
 
+### Text-based Tasks
+#### MMLU
+  
 ```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/pixtral-12b-quantized.w4a16 ",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+  --tasks mmlu \
+  --num_fewshot 5
+  --batch_size auto \
+  --output_path output_dir \
+
 ```
 
+#### HumanEval
+
+##### Generation
+```
+python3 codegen/generate.py \
+  --model neuralmagic/pixtral-12b-quantized.w4a16 \
+  --bs 16 \
+  --temperature 0.2 \
+  --n_samples 50 \
+  --root "." \
+  --dataset humaneval
+```
+##### Sanitization
+```
+python3 evalplus/sanitize.py \
+  humaneval/neuralmagic/pixtral-12b-quantized.w4a16_vllm_temp_0.2
+```
+##### Evaluation
+```
+evalplus.evaluate \
+  --dataset humaneval \
+  --samples humaneval/neuralmagic/pixtral-12b-quantized.w4a16_vllm_temp_0.2-sanitized
+```
 </details>
 
-### Accuracy
+## Accuracy
+
+<table border="1">
+  <thead>
+    <tr>
+      <th>Category</th>
+      <th>Metric</th>
+      <th>mgoin/pixtral-12b</th>
+      <th>neuralmagic/pixtral-12b-FP8-Dynamic</th>
+      <th>Recovery (%)</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td rowspan="6"><b>Vision</b></td>
+      <td>MMMU (val, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td>
+      <td>48.00</td>
+      <td>50.11</td>
+      <td>104.40%</td>
+    </tr>
+    <tr>
+      <td>VQAv2 (val)<br><i>vqa_match</i></td>
+      <td>78.71</td>
+      <td>78.44</td>
+      <td>99.66%</td>
+    </tr>
+    <tr>
+      <td>DocVQA (val)<br><i>anls</i></td>
+      <td>89.47</td>
+      <td>89.20</td>
+      <td>99.70%</td>
+    </tr>
+    <tr>
+      <td>ChartQA (test, CoT)<br><i>anywhere_in_answer_relaxed_correctness</i></td>
+      <td>81.68</td>
+      <td>81.76</td>
+      <td>100.10%</td>
+    </tr>
+    <tr>
+      <td>Mathvista (testmini, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td>
+      <td>56.50</td>
+      <td>58.70</td>
+      <td>103.89%</td>
+    </tr>
+    <tr>
+      <td><b>Average Score</b></td>
+      <td><b>70.07</b></td>
+      <td><b>71.24</b></td>
+      <td><b>101.67%</b></td>
+    </tr>
+    <tr>
+      <td rowspan="2"><b>Text</b></td>
+      <td>HumanEval <br><i>pass@1</i></td>
+      <td>68.40</td>
+      <td>69.50</td>
+      <td>101.61%</td>
+    </tr>
+    <tr>
+      <td>MMLU (5-shot)</td>
+      <td>71.40</td>
+      <td>69.50</td>
+      <td>97.34%</td>
+    </tr>
+  </tbody>
+</table>
+
 
 ## Inference Performance