RedHatAI
/

Qwen2.5-VL-72B-Instruct-quantized.w8a8

Image-Text-to-Text

text-generation-inference

8-bit precision

compressed-tensors

Model card Files Files and versions Community

shubhrapandit commited on Mar 3

Commit

f005e30

·

verified ·

1 Parent(s): 9d691ab

Update README.md

Files changed (1) hide show

README.md +28 -28

README.md CHANGED Viewed

@@ -400,10 +400,10 @@ The following performance benchmarks were conducted with [vLLM](https://docs.vll
       <td>5.66</td>
       <td>4.3</td>
       <td>252</td>
-      <td>4.3</td>
-      <td>252</td>
-      <td>1.0</td>
-      <td>1065</td>
     </tr>
   </tbody>
 </table>
@@ -451,22 +451,22 @@ The following performance benchmarks were conducted with [vLLM](https://docs.vll
     <tr>
       <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w8a8</td>
       <td>1.80</td>
-      <td>1.2</td>
-      <td>578</td>
-      <td>4.0</td>
-      <td>2040</td>
-      <td>4.6</td>
-      <td>2266</td>
     </tr>
     <tr>
       <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
       <td>2.75</td>
-      <td>2.8</td>
-      <td>1364</td>
-      <td>12.8</td>
-      <td>6352</td>
-      <td>16.4</td>
-      <td>8148</td>
     </tr>
     <tr>
       <th rowspan="3" valign="top">H100x4</th>
@@ -482,22 +482,22 @@ The following performance benchmarks were conducted with [vLLM](https://docs.vll
     <tr>
       <td>neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic</td>
       <td>1.73</td>
-      <td>1.8</td>
-      <td>479</td>
-      <td>4.4</td>
-      <td>1203</td>
-      <td>4.8</td>
-      <td>1296</td>
     </tr>
     <tr>
       <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
       <td>8.27</td>
-      <td>13.2</td>
-      <td>3652</td>
-      <td>13.2</td>
-      <td>3652</td>
-      <td>99.2</td>
-      <td>27108</td>
     </tr>
   </tbody>
 </table>

       <td>5.66</td>
       <td>4.3</td>
       <td>252</td>
+      <td>4.4</td>
+      <td>251</td>
+      <td>4.2</td>
+      <td>259</td>
     </tr>
   </tbody>
 </table>
     <tr>
       <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w8a8</td>
       <td>1.80</td>
+      <td>0.6</td>
+      <td>289</td>
+      <td>2.0</td>
+      <td>1020</td>
+      <td>2.3</td>
+      <td>1133</td>
     </tr>
     <tr>
       <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
       <td>2.75</td>
+      <td>0.7</td>
+      <td>341</td>
+      <td>3.2</td>
+      <td>1588</td>
+      <td>4.1</td>
+      <td>2037</td>
     </tr>
     <tr>
       <th rowspan="3" valign="top">H100x4</th>
     <tr>
       <td>neuralmagic/Qwen2.5-VL-72B-Instruct-FP8-Dynamic</td>
       <td>1.73</td>
+      <td>0.9</td>
+      <td>247</td>
+      <td>2.2</td>
+      <td>621</td>
+      <td>2.4</td>
+      <td>669</td>
     </tr>
     <tr>
       <td>neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16</td>
       <td>8.27</td>
+      <td>3.3</td>
+      <td>913</td>
+      <td>3.3</td>
+      <td>898</td>
+      <td>3.6</td>
+      <td>991</td>
     </tr>
   </tbody>
 </table>