RedHatAI
/

DeepSeek-R1-Distill-Llama-8B-FP8-dynamic

@@ -137,39 +137,146 @@ lm_eval \
 ### Accuracy
-#### OpenLLM Leaderboard V1 evaluation scores
-| Metric                                   | deepseek-ai/DeepSeek-R1-Distill-Llama-8B             | neuralmagic-ent/DeepSeek-R1-Distill-Llama-8B-FP8-Dynamic |
-|-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
-| ARC-Challenge (Acc-Norm, 25-shot)       |          45.05                   |               44.88                         |
-| GSM8K (Strict-Match, 5-shot)            |          62.77                   |              61.49                          |
-| HellaSwag (Acc-Norm, 10-shot)           |          76.78                   |             76.68                     |
-| MMLU (Acc, 5-shot)                      |          55.65                  |               55.82                      |
-| TruthfulQA (MC2, 0-shot)                |          50.55                   |             49.92                      |
-| Winogrande (Acc, 5-shot)                |          68.51                   |             67.72                           |
-| **Average Score**                       | **59.88**                        | **59.42**                                   |
-| **Recovery (%)**                            | **100.00**                       | **99.22**                                   |
-#### OpenLLM Leaderboard V2 evaluation scores
-| Metric                                                   | deepseek-ai/DeepSeek-R1-Distill-Llama-8B             | neuralmagic-ent/DeepSeek-R1-Distill-Llama-8B-FP8-Dynamic |
-|---------------------------------------------------------|:---------------------------------:|:-------------------------------------------:|
-| IFEval (Inst-and-Prompt Level Strict Acc, 0-shot)       |         38.34                    |          38.22                              |
-| BBH (Acc-Norm, 3-shot)                                  |          38.19                   |               38.32                         |
-| GPQA (Acc-Norm, 0-shot)                                 |          28.87                    |                   27.56                      |
-| MUSR (Acc-Norm, 0-shot)                                 |          33.31                    |                33.71                       |
-| MMLU-Pro (Acc, 5-shot)                                  |          20.10                   |         21.39                              |
-| **Average Score**                                       | **26.47**                        | **26.53**                                   |
-| **Recovery (%)**                                            | **100.00**                       | **100.24**                                   |
-#### Coding evaluation scores
-| Metric                                                   | deepseek-ai/DeepSeek-R1-Distill-Llama-8B             | neuralmagic-ent/DeepSeek-R1-Distill-Llama-8B-FP8-Dynamic |
-|---------------------------------------------------------|:---------------------------------:|:-------------------------------------------:|
-| HumanEval pass@1                                         |         49.90                    |             51.20
-| HumanEval pass@10                                        |         68.90                    |             68.20                           |
-| HumanEval+ pass@1                                        |         44.10                    |             46.60                           |
-| HumanEval+ pass@10                                       |         62.90                    |              62.70                          |
-| **Average Score**                                       | **56.45**                        | **57.17**                                   |
-| **Recovery (%)**                                            | **100.00**                       | **101.27**                                   |

 ### Accuracy
+<table>
+  <thead>
+    <tr>
+      <th>Category</th>
+      <th>Metric</th>
+      <th>deepseek-ai/DeepSeek-R1-Distill-Llama-8B</th>
+      <th>neuralmagic-ent/DeepSeek-R1-Distill-Llama-8B-FP8-Dynamic</th>
+      <th>Recovery</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td rowspan="3"><b>Reasoning</b></td>
+      <td>AIME 2024 (pass@1)</td>
+      <td>50.00</td>
+      <td>30.00</td>
+      <td>60.0%</td>
+    </tr>
+    <tr>
+      <td>MATH-500 (pass@1)</td>
+      <td>87.60</td>
+      <td>84.60</td>
+      <td>96.6%</td>
+    </tr>
+    <tr>
+      <td>GPQA Diamond (pass@1)</td>
+      <td>44.95</td>
+      <td>43.94</td>
+      <td>97.8%</td>
+    </tr>
+    <tr>
+      <td rowspan="7"><b>OpenLLM V1</b></td>
+      <td>ARC-Challenge (Acc-Norm, 25-shot)</td>
+      <td>45.05</td>
+      <td>44.88</td>
+      <td>99.6%</td>
+    </tr>
+    <tr>
+      <td>GSM8K (Strict-Match, 5-shot)</td>
+      <td>62.77</td>
+      <td>61.49</td>
+      <td>98.0</td>
+    </tr>
+    <tr>
+      <td>HellaSwag (Acc-Norm, 10-shot)</td>
+      <td>76.78</td>
+      <td>76.68</td>
+      <td>99.9%</td>
+    </tr>
+    <tr>
+      <td>MMLU (Acc, 5-shot)</td>
+      <td>55.65</td>
+      <td>55.82</td>
+      <td>100.3%</td>
+    </tr>
+    <tr>
+      <td>TruthfulQA (MC2, 0-shot)</td>
+      <td>50.55</td>
+      <td>49.92</td>
+      <td>98.8%</td>
+    </tr>
+    <tr>
+      <td>Winogrande (Acc, 5-shot)</td>
+      <td>68.51</td>
+      <td>67.72</td>
+      <td>98.8%</td>
+    </tr>
+    <tr>
+      <td><b>Average Score</b></td>
+      <td><b>58.88</b></td>
+      <td><b>59.42</b></td>
+      <td><b>99.2</b></td>
+    </tr>
+    <tr>
+      <td rowspan="7"><b>OpenLLM V2</b></td>
+      <td>IFEval (Inst Level Strict Acc, 0-shot)</td>
+      <td>38.34</td>
+      <td>38.22</td>
+      <td>99.7%</td>
+    </tr>
+    <tr>
+      <td>BBH (Acc-Norm, 3-shot)</td>
+      <td>38.19</td>
+      <td>38.32</td>
+      <td>100.3%</td>
+    </tr>
+    <tr>
+      <td>Math-Hard (Exact-Match, 4-shot)</td>
+      <td>0.00</td>
+      <td>0.00</td>
+      <td>---</td>
+    </tr>
+    <tr>
+      <td>GPQA (Acc-Norm, 0-shot)</td>
+      <td>28.87</td>
+      <td>27.56</td>
+      <td>95.5%</td>
+    </tr>
+    <tr>
+      <td>MUSR (Acc-Norm, 0-shot)</td>
+      <td>33.31</td>
+      <td>33.71</td>
+      <td>101.2%</td>
+    </tr>
+    <tr>
+      <td>MMLU-Pro (Acc, 5-shot)</td>
+      <td>20.10</td>
+      <td>21.39</td>
+      <td>106.4%</td>
+    </tr>
+    <tr>
+      <td><b>Average Score</b></td>
+      <td><b>26.47</b></td>
+      <td><b>26.53</b></td>
+      <td><b>100.2%</b></td>
+    </tr>
+    <tr>
+      <td rowspan="4"><b>Coding</b></td>
+      <td>HumanEval (pass@1)</td>
+      <td>49.90</td>
+      <td>51.20</td>
+      <td><b>102.6%</b></td>
+    </tr>
+    <tr>
+      <td>HumanEval (pass@10)</td>
+      <td>68.90</td>
+      <td>68.20</td>
+      <td>99.0%</td>
+    </tr>
+    <tr>
+      <td>HumanEval+ (pass@10)</td>
+      <td>44.10</td>
+      <td>46.60</td>
+      <td>105.7%</td>
+    </tr>
+    <tr>
+      <td>HumanEval+ (pass@10)</td>
+      <td>62.90</td>
+      <td>62.70</td>
+      <td>99.7%</td>
+    </tr>
+  </tbody>
+</table>