Update README.md

README.md

@@ -183,17 +183,121 @@ oneshot(
 
 ## Evaluation
 
-The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard), OpenLLM Leaderboard [V2](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/) and on [HumanEval](https://github.com/neuralmagic/evalplus), using the following commands:
+The model was evaluated using [mistral-evals](https://github.com/neuralmagic/mistral-evals) for vision-related tasks and using [lm_evaluation_harness](https://github.com/neuralmagic/lm-evaluation-harness) for select text-based benchmarks. The evaluations were conducted using the following commands:
 
 <details>
 <summary>Evaluation Commands</summary>
+  
+### Vision Tasks
+- vqav2
+- docvqa
+- mathvista
+- mmmu
+- chartqa
+
+```
+vllm serve neuralmagic/pixtral-12b-quantized.w8a8 --tensor_parallel_size 1 --max_model_len 25000 --trust_remote_code --max_num_seqs 8 --gpu_memory_utilization 0.9 --dtype float16 --limit_mm_per_prompt image=7
 
+python -m eval.run eval_vllm \
+        --model_name neuralmagic/pixtral-12b-quantized.w8a8 \
+        --url http://0.0.0.0:8000 \
+        --output_dir ~/tmp \
+        --eval_name <vision_task_name>
 ```
+
+### Text-based Tasks
+#### MMLU
+  
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+  --tasks mmlu \
+  --num_fewshot 5 \
+  --batch_size auto \
+  --output_path output_dir
+
+```
+
+#### MGSM
+
 ```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \
+  --tasks mgsm_cot_native \
+  --num_fewshot 0 \
+  --batch_size auto \
+  --output_path output_dir
 
+```
 </details>
 
 ### Accuracy
+<table>
+  <thead>
+    <tr>
+      <th>Category</th>
+      <th>Metric</th>
+      <th>Qwen/Qwen2.5-VL-3B-Instruct</th>
+      <th>neuralmagic/Qwen2.5-VL-3B-Instruct-quantized.w4a16</th>
+      <th>Recovery (%)</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td rowspan="6"><b>Vision</b></td>
+      <td>MMMU (val, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td>
+      <td>44.56</td>
+      <td>45.67</td>
+      <td>102.49%</td>
+    </tr>
+    <tr>
+      <td>VQAv2 (val)<br><i>vqa_match</i></td>
+      <td>75.94</td>
+      <td>75.55</td>
+      <td>99.49%</td>
+    </tr>
+    <tr>
+      <td>DocVQA (val)<br><i>anls</i></td>
+      <td>92.53</td>
+      <td>92.32</td>
+      <td>99.77%</td>
+    </tr>
+    <tr>
+      <td>ChartQA (test, CoT)<br><i>anywhere_in_answer_relaxed_correctness</i></td>
+      <td>81.20</td>
+      <td>78.80</td>
+      <td>97.04%</td>
+    </tr>
+    <tr>
+      <td>Mathvista (testmini, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td>
+      <td>54.15</td>
+      <td>53.85</td>
+      <td>99.45%</td>
+    </tr>
+    <tr>
+      <td><b>Average Score</b></td>
+      <td><b>69.28</b></td>
+      <td><b>69.24</b></td>
+      <td><b>99.94%</b></td>
+    </tr>
+    <tr>
+      <td rowspan="2"><b>Text</b></td>
+      <td>MGSM (CoT)</td>
+      <td>52.49</td>
+      <td>50.42</td>
+      <td>96.05%</td>
+    </tr>
+    <tr>
+      <td>MMLU (5-shot)</td>
+      <td>65.32</td>
+      <td>64.83</td>
+      <td>99.25%</td>
+    </tr>
+  </tbody>
+</table>
+
 
 ## Inference Performance