Update README.md

README.md

@@ -119,18 +119,124 @@ oneshot(
 
 ## Evaluation
 
-The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard), OpenLLM Leaderboard [V2](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/) and on [HumanEval](https://github.com/neuralmagic/evalplus), using the following commands:
+The model was evaluated using [mistral-evals](https://github.com/neuralmagic/mistral-evals) for vision-related tasks and using [lm_evaluation_harness](https://github.com/neuralmagic/lm-evaluation-harness) for select text-based benchmarks. The evaluations were conducted using the following commands:
 
 <details>
 <summary>Evaluation Commands</summary>
+  
+### Vision Tasks
+- vqav2
+- docvqa
+- mathvista
+- mmmu
+- chartqa
+
+```
+vllm serve neuralmagic/pixtral-12b-quantized.w8a8 --tensor_parallel_size 1 --max_model_len 25000 --trust_remote_code --max_num_seqs 8 --gpu_memory_utilization 0.9 --dtype float16 --limit_mm_per_prompt image=7
+
+python -m eval.run eval_vllm \
+        --model_name neuralmagic/pixtral-12b-quantized.w8a8 \
+        --url http://0.0.0.0:8000 \
+        --output_dir ~/tmp \
+        --eval_name <vision_task_name>
+```
+
+### Text-based Tasks
+#### MMLU
+  
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+  --tasks mmlu \
+  --num_fewshot 5 \
+  --batch_size auto \
+  --output_path output_dir
 
 ```
+
+#### MGSM
+
 ```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \
+  --tasks mgsm_cot_native \
+  --num_fewshot 0 \
+  --batch_size auto \
+  --output_path output_dir
 
+```
 </details>
 
+
 ### Accuracy
 
+<table>
+  <thead>
+    <tr>
+      <th>Category</th>
+      <th>Metric</th>
+      <th>Qwen/Qwen2.5-VL-7B-Instruct</th>
+      <th>neuralmagic/Qwen2.5-VL-7B-Instruct-FP8-Dynamic</th>
+      <th>Recovery (%)</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td rowspan="6"><b>Vision</b></td>
+      <td>MMMU (val, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td>
+      <td>52.00</td>
+      <td>52.55</td>
+      <td>101.06%</td>
+    </tr>
+    <tr>
+      <td>VQAv2 (val)<br><i>vqa_match</i></td>
+      <td>75.59</td>
+      <td>75.79</td>
+      <td>100.26%</td>
+    </tr>
+    <tr>
+      <td>DocVQA (val)<br><i>anls</i></td>
+      <td>94.27</td>
+      <td>94.27</td>
+      <td>100.00%</td>
+    </tr>
+    <tr>
+      <td>ChartQA (test, CoT)<br><i>anywhere_in_answer_relaxed_correctness</i></td>
+      <td>86.44</td>
+      <td>86.80</td>
+      <td>100.42%</td>
+    </tr>
+    <tr>
+      <td>Mathvista (testmini, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td>
+      <td>69.47</td>
+      <td>71.07</td>
+      <td>102.31%</td>
+    </tr>
+    <tr>
+      <td><b>Average Score</b></td>
+      <td><b>75.95</b></td>
+      <td><b>76.50</b></td>
+      <td><b>100.73%</b></td>
+    </tr>
+    <tr>
+      <td rowspan="2"><b>Text</b></td>
+      <td>MGSM (CoT)</td>
+      <td>58.72</td>
+      <td>55.34</td>
+      <td>94.24%</td>
+    </tr>
+    <tr>
+      <td>MMLU (5-shot)</td>
+      <td>71.09</td>
+      <td>70.98</td>
+      <td>99.85%</td>
+    </tr>
+  </tbody>
+</table>
+
+
 ## Inference Performance