Update README.md

README.md

@@ -270,3 +270,240 @@ lm_eval \
     </tr>
   </tbody>
 </table>
+
+
+## Inference Performance
+
+
+This model achieves up to 1.4x speedup in single-stream deployment and up to 1.8x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario.
+The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.6.7.2, and [GuideLLM](https://github.com/neuralmagic/guidellm).
+
+<details>
+<summary>Benchmarking Command</summary>
+
+```
+guidellm --model neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w4a16 --target "http://localhost:8000/v1" --data-type emulated --data "prompt_tokens=<prompt_tokens>,generated_tokens=<generated_tokens>" --max seconds 360 --backend aiohttp_server
+```
+</details>
+
+### Single-stream performance (measured with vLLM version 0.7.2)
+<table>
+  <thead>
+    <tr>
+      <th></th>
+      <th></th>
+      <th></th>
+      <th></th>
+      <th style="text-align: center;" colspan="2" >Instruction Following<br>256 / 128</th>
+      <th style="text-align: center;" colspan="2" >Multi-turn Chat<br>512 / 256</th>
+      <th style="text-align: center;" colspan="2" >Docstring Generation<br>768 / 128</th>
+      <th style="text-align: center;" colspan="2" >RAG<br>1024 / 128</th>
+      <th style="text-align: center;" colspan="2" >Code Completion<br>256 / 1024</th>
+      <th style="text-align: center;" colspan="2" >Code Fixing<br>1024 / 1024</th>
+      <th style="text-align: center;" colspan="2" >Large Summarization<br>4096 / 512</th>
+      <th style="text-align: center;" colspan="2" >Large RAG<br>10240 / 1536</th>
+    </tr>
+    <tr>
+      <th>GPU class</th>
+      <th>Number of GPUs</th>
+      <th>Model</th>
+      <th>Average cost reduction</th>
+      <th>Latency (s)</th>
+      <th>QPD</th>
+      <th>Latency (s)</th>
+      <th>QPD</th>
+      <th>Latency (s)</th>
+      <th>QPD</th>
+      <th>Latency (s)</th>
+      <th>QPD</th>
+      <th>Latency (s)</th>
+      <th>QPD</th>
+      <th>Latency (s)</th>
+      <th>QPD</th>
+      <th>Latency (s)</th>
+      <th>QPD</th>
+      <th>Latency (s)</th>
+      <th>QPD</th>
+    </tr>
+  </thead>
+  <tbody style="text-align: center" >
+    <tr>
+      <th rowspan="3" valign="top">H100</th>
+      <td>2</td>
+      <th>deepseek-ai/DeepSeek-R1-Distill-Llama-70B</th>
+      <td>---</td>
+      <td>3.8</td>
+      <td>149</td>
+      <td>7.6</td>
+      <td>74</td>
+      <td>3.9</td>
+      <td>146</td>
+      <td>3.9</td>
+      <td>144</td>
+      <td>30.0</td>
+      <td>19</td>
+      <td>30.4</td>
+      <td>19</td>
+      <td>16.1</td>
+      <td>35</td>
+      <td>56.5</td>
+      <td>10</td>
+    </tr>
+    <tr>
+      <td>2</td>
+      <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-FP8-dynamic</th>
+      <td>1.39</td>
+      <td>2.7</td>
+      <td>210</td>
+      <td>5.3</td>
+      <td>106</td>
+      <td>2.7</td>
+      <td>207</td>
+      <td>2.8</td>
+      <td>203</td>
+      <td>21.1</td>
+      <td>27</td>
+      <td>21.4</td>
+      <td>26</td>
+      <td>11.5</td>
+      <td>49</td>
+      <td>47.2</td>
+      <td>12</td>
+    </tr>
+    <tr>
+      <td>1</td>
+      <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w4a16</th>
+      <td>1.83</td>
+      <td>4.0</td>
+      <td>277</td>
+      <td>7.9</td>
+      <td>138</td>
+      <td>4.1</td>
+      <td>266</td>
+      <td>4.2</td>
+      <td>262</td>
+      <td>31.2</td>
+      <td>35</td>
+      <td>31.8</td>
+      <td>34</td>
+      <td>17.8</td>
+      <td>61</td>
+      <td>61.4</td>
+      <td>18</td>
+    </tr>
+  </tbody>
+</table>
+
+**Use case profiles: prompt tokens / generation tokens
+
+**QPD: Queries per dollar, based on on-demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu-cloud) (observed on 2/18/2025).
+
+
+### Multi-stream asynchronous performance (measured with vLLM version 0.7.2)
+<table>
+  <thead>
+    <tr>
+      <th></th>
+      <th></th>
+      <th></th>
+      <th style="text-align: center;" colspan="2" >Instruction Following<br>256 / 128</th>
+      <th style="text-align: center;" colspan="2" >Multi-turn Chat<br>512 / 256</th>
+      <th style="text-align: center;" colspan="2" >Docstring Generation<br>768 / 128</th>
+      <th style="text-align: center;" colspan="2" >RAG<br>1024 / 128</th>
+      <th style="text-align: center;" colspan="2" >Code Completion<br>256 / 1024</th>
+      <th style="text-align: center;" colspan="2" >Code Fixing<br>1024 / 1024</th>
+      <th style="text-align: center;" colspan="2" >Large Summarization<br>4096 / 512</th>
+      <th style="text-align: center;" colspan="2" >Large RAG<br>10240 / 1536</th>
+    </tr>
+    <tr>
+      <th>Hardware</th>
+      <th>Model</th>
+      <th>Average cost reduction</th>
+      <th>Maximum throughput (QPS)</th>
+      <th>QPD</th>
+      <th>Maximum throughput (QPS)</th>
+      <th>QPD</th>
+      <th>Maximum throughput (QPS)</th>
+      <th>QPD</th>
+      <th>Maximum throughput (QPS)</th>
+      <th>QPD</th>
+      <th>Maximum throughput (QPS)</th>
+      <th>QPD</th>
+      <th>Maximum throughput (QPS)</th>
+      <th>QPD</th>
+      <th>Maximum throughput (QPS)</th>
+      <th>QPD</th>
+      <th>Maximum throughput (QPS)</th>
+      <th>QPD</th>
+    </tr>
+  </thead>
+  <tbody style="text-align: center" >
+    <tr>
+      <th rowspan="3" valign="top">H100x4</th>
+      <th>deepseek-ai/DeepSeek-R1-Distill-Llama-70B</th>
+      <td>---</td>
+      <td>14.04</td>
+      <td>2113</td>
+      <td>10.85</td>
+      <td>1634</td>
+      <td>12.25</td>
+      <td>1844</td>
+      <td>9.93</td>
+      <td>1494</td>
+      <td>3.68</td>
+      <td>554</td>
+      <td>2.82</td>
+      <td>425</td>
+      <td>1.81</td>
+      <td>273</td>
+      <td>0.35</td>
+      <td>52</td>
+    </tr>
+    <tr>
+      <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-FP8-dynamic</th>
+      <td>1.78</td>
+      <td>41.44</td>
+      <td>6236</td>
+      <td>19.64</td>
+      <td>2956</td>
+      <td>21.03</td>
+      <td>3166</td>
+      <td>16.72</td>
+      <td>2516</td>
+      <td>6.01</td>
+      <td>904</td>
+      <td>4.46</td>
+      <td>672</td>
+      <td>2.55</td>
+      <td>383</td>
+      <td>0.49</td>
+      <td>74</td>
+    </tr>
+    <tr>
+      <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w4a16</th>
+      <td>1.45</td>
+      <td>36.61</td>
+      <td>5509</td>
+      <td>15.12</td>
+      <td>2275</td>
+      <td>16.24</td>
+      <td>2443</td>
+      <td>13.22</td>
+      <td>1990</td>
+      <td>5.48</td>
+      <td>825</td>
+      <td>3.01</td>
+      <td>453</td>
+      <td>2.07</td>
+      <td>312</td>
+      <td>0.43</td>
+      <td>64</td>
+    </tr>
+  </tbody>
+</table>
+
+**Use case profiles: prompt tokens / generation tokens
+
+**QPS: Queries per second.
+
+**QPD: Queries per dollar, based on on-demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu-cloud) (observed on 2/18/2025).