Update README.md
Browse files
README.md
CHANGED
@@ -654,7 +654,7 @@ lm_eval \
|
|
654 |
## Inference Performance
|
655 |
|
656 |
|
657 |
-
This model achieves up to 1.
|
658 |
The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.7.2, and [GuideLLM](https://github.com/neuralmagic/guidellm).
|
659 |
|
660 |
<details>
|
@@ -806,21 +806,21 @@ The following performance benchmarks were conducted with [vLLM](https://docs.vll
|
|
806 |
<tr>
|
807 |
<td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w8a8</td>
|
808 |
<td>1.70</td>
|
809 |
-
<td>
|
810 |
<td>766</td>
|
811 |
-
<td>
|
812 |
<td>1142</td>
|
813 |
-
<td>
|
814 |
<td>1348</td>
|
815 |
</tr>
|
816 |
<tr>
|
817 |
<td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w4a16</td>
|
818 |
<td>1.48</td>
|
819 |
-
<td>
|
820 |
<td>552</td>
|
821 |
-
<td>
|
822 |
<td>1010</td>
|
823 |
-
<td>
|
824 |
<td>1360</td>
|
825 |
</tr>
|
826 |
<tr>
|
@@ -837,21 +837,21 @@ The following performance benchmarks were conducted with [vLLM](https://docs.vll
|
|
837 |
<tr>
|
838 |
<td>neuralmagic/Pixtral-Large-Instruct-2411-hf-FP8-Dynamic</td>
|
839 |
<td>1.61</td>
|
840 |
-
<td>
|
841 |
<td>905</td>
|
842 |
-
<td>
|
843 |
<td>1406</td>
|
844 |
-
<td>
|
845 |
<td>1759</td>
|
846 |
</tr>
|
847 |
<tr>
|
848 |
<td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w4a16</td>
|
849 |
<td>1.33</td>
|
850 |
-
<td>
|
851 |
<td>761</td>
|
852 |
-
<td>
|
853 |
<td>1228</td>
|
854 |
-
<td>
|
855 |
<td>1480</td>
|
856 |
</tr>
|
857 |
</tbody>
|
|
|
654 |
## Inference Performance
|
655 |
|
656 |
|
657 |
+
This model achieves up to 1.9x speedup in single-stream deployment and up to 1.78x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario.
|
658 |
The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.7.2, and [GuideLLM](https://github.com/neuralmagic/guidellm).
|
659 |
|
660 |
<details>
|
|
|
806 |
<tr>
|
807 |
<td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w8a8</td>
|
808 |
<td>1.70</td>
|
809 |
+
<td>0.8</td>
|
810 |
<td>766</td>
|
811 |
+
<td>1.1</td>
|
812 |
<td>1142</td>
|
813 |
+
<td>1.3</td>
|
814 |
<td>1348</td>
|
815 |
</tr>
|
816 |
<tr>
|
817 |
<td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w4a16</td>
|
818 |
<td>1.48</td>
|
819 |
+
<td>0.5</td>
|
820 |
<td>552</td>
|
821 |
+
<td>1.0</td>
|
822 |
<td>1010</td>
|
823 |
+
<td>1.4</td>
|
824 |
<td>1360</td>
|
825 |
</tr>
|
826 |
<tr>
|
|
|
837 |
<tr>
|
838 |
<td>neuralmagic/Pixtral-Large-Instruct-2411-hf-FP8-Dynamic</td>
|
839 |
<td>1.61</td>
|
840 |
+
<td>1.7</td>
|
841 |
<td>905</td>
|
842 |
+
<td>2.6</td>
|
843 |
<td>1406</td>
|
844 |
+
<td>3.2</td>
|
845 |
<td>1759</td>
|
846 |
</tr>
|
847 |
<tr>
|
848 |
<td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w4a16</td>
|
849 |
<td>1.33</td>
|
850 |
+
<td>1.4</td>
|
851 |
<td>761</td>
|
852 |
+
<td>2.2</td>
|
853 |
<td>1228</td>
|
854 |
+
<td>2.7</td>
|
855 |
<td>1480</td>
|
856 |
</tr>
|
857 |
</tbody>
|