Update README.md
Browse files
README.md
CHANGED
|
@@ -10,4 +10,20 @@ Pruned SmolLM2-1.7B model - total parameters 1.47B.
|
|
| 10 |
|
| 11 |
Intermediate step, requires further training - try it yourself.
|
| 12 |
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
|
| 11 |
Intermediate step, requires further training - try it yourself.
|
| 12 |
|
| 13 |
+
Eval results using SmolLM evaluation scripts (**LightEval**):
|
| 14 |
+
|
| 15 |
+
| Task |Version| Metric |Value | |Stderr|
|
| 16 |
+
|-----------------------|------:|--------|-----:|---|-----:|
|
| 17 |
+
|all | |acc_norm|0.4555|± |0.0114|
|
| 18 |
+
| | |qem |0.0431|± |0.0022|
|
| 19 |
+
|custom:arc:_average:0 | |acc_norm|0.5021|± |0.0120|
|
| 20 |
+
|custom:arc:challenge:0 | 0|acc_norm|0.3686|± |0.0141|
|
| 21 |
+
|custom:arc:easy:0 | 0|acc_norm|0.6355|± |0.0099|
|
| 22 |
+
|custom:commonsense_qa:0| 0|acc_norm|0.3333|± |0.0135|
|
| 23 |
+
|custom:gsm8k:5 | 0|qem |0.0076|± |0.0024|
|
| 24 |
+
|custom:hellaswag:0 | 0|acc_norm|0.5568|± |0.0050|
|
| 25 |
+
|custom:mmlu_pro:0 | 0|acc_norm|0.1287|± |0.0031|
|
| 26 |
+
|custom:openbook_qa:0 | 0|acc_norm|0.3660|± |0.0216|
|
| 27 |
+
|custom:piqa:0 | 0|acc_norm|0.7187|± |0.0105|
|
| 28 |
+
|custom:trivia_qa:0 | 0|qem |0.0787|± |0.0020|
|
| 29 |
+
|custom:winogrande:0 | 0|acc_norm|0.5367|± |0.0140|
|