|
|
--- |
|
|
license: cc-by-4.0 |
|
|
datasets: |
|
|
- allenai/c4 |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- accuracy |
|
|
base_model: |
|
|
- deepseek-ai/DeepSeek-R1-Distill-Llama-70B |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
|
|
|
|
|
|
# Overview |
|
|
This document presents the evaluation results of `DeepSeek-R1-Distill-Llama-70B`, a **4-bit quantized model using GPTQ**, evaluated with the **Language Model Evaluation Harness** on the **ARC-Challenge** benchmark. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📊 Evaluation Summary |
|
|
|
|
|
| **Metric** | **Value** | **Description** | **8bit** | |
|
|
|----------------------|-----------|-----------------|-----------| |
|
|
| **Accuracy (acc,none)** | `21.2%` | Raw accuracy - percentage of correct answers. | `21.2%` | |
|
|
| **Standard Error (acc_stderr,none)** | `1.19%` | Uncertainty in the accuracy estimate. | `1.2%` | |
|
|
| **Normalized Accuracy (acc_norm,none)** | `25.4%` | Accuracy after dataset-specific normalization. | `25.2%` | |
|
|
| **Standard Error (acc_norm_stderr,none)** | `1.27%` | Uncertainty for normalized accuracy. | `1.3%` | |
|
|
|
|
|
📌 **Interpretation:** |
|
|
- The model correctly answered **21.2% of the questions**. |
|
|
- After **normalization**, the accuracy slightly improves to **25.4%**. |
|
|
- The **standard error (~1.27%)** indicates a small margin of uncertainty. |
|
|
|
|
|
--- |
|
|
|
|
|
## ⚙️ Model Configuration |
|
|
|
|
|
- **Model:** `DeepSeek-R1-Distill-Llama-70B` |
|
|
- **Parameters:** `70 billion` |
|
|
- **Quantization:** `4-bit GPTQ` |
|
|
- **Source:** Hugging Face (`hf`) |
|
|
- **Precision:** `torch.float16` |
|
|
- **Hardware:** `NVIDIA A100 80GB PCIe` |
|
|
- **CUDA Version:** `12.4` |
|
|
- **PyTorch Version:** `2.6.0+cu124` |
|
|
- **Batch Size:** `1` |
|
|
- **Evaluation Time:** `365.89 seconds (~6 minutes)` |
|
|
|
|
|
📌 **Interpretation:** |
|
|
- The evaluation was performed on a **high-performance GPU (A100 80GB)**. |
|
|
- The model is significantly larger than the previous 8B version, with **GPTQ 4-bit quantization reducing memory footprint**. |
|
|
- A **single-sample batch size** was used, which might slow evaluation speed. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📂 Dataset Information |
|
|
|
|
|
- **Dataset:** `AI2 ARC-Challenge` |
|
|
- **Task Type:** `Multiple Choice` |
|
|
- **Number of Samples Evaluated:** `1,172` |
|
|
- **Few-shot Examples Used:** `0` (Zero-shot setting) |
|
|
|
|
|
📌 **Interpretation:** |
|
|
- This benchmark assesses **grade-school-level scientific reasoning**. |
|
|
- Since **no few-shot examples** were provided, the model was evaluated in a **pure zero-shot setting**. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📈 Performance Insights |
|
|
|
|
|
- The `"higher_is_better"` flag confirms that **higher accuracy is preferred**. |
|
|
- The model's **raw accuracy (21.2%)** is significantly lower compared to state-of-the-art models (**60–80%** on ARC-Challenge). |
|
|
- **Quantization Impact:** The **4-bit GPTQ quantization** reduces memory usage but may also impact accuracy slightly. |
|
|
- **Zero-shot Limitation:** Performance could improve with **few-shot prompting** (providing examples before testing). |
|
|
|
|
|
--- |
|
|
|
|
|
|
|
|
## 📊 Detailed Evaluation on MMLU Challenges |
|
|
|
|
|
|
|
|
| **Metric** | **Value** | **Description** | |
|
|
|----------------------|-----------|-----------------| |
|
|
| **MMLU** | `37.88%` | Averaged over MMLU-Stem, MMLU-Social-Sciences, MMLU-Humanities, MMLU-ther | |
|
|
| **MMLU-Humanities** | `31.83%` | Averaged over MMLU-Formal-Logic, MMLU-Prehistory, MMLU-World-Religions, MMLU-Philosophy, MMLU-High-School-World-History, MMLU-Professional-Law, MMLU-High-School-US-History, MMLU-Logical-Fallacies, MMLU-International-Law, MMLU-High-School-European-History, MMLU-Moral-Disputes, MMLU-Moral-Scenarios, MMLU-Jurisprudence | |
|
|
| **MMLU-Social-Sciences** | `45.43%` | Averaged over MMLU-Public-Relations, MMLU-Sociology, MMLU-Security-Studies, MMLU-High-School-Government-and-Politics, MMLU-High-School-Psychology, MMLU-Human-Sexuality, MMLU-US-Foreign-Policy, MMLU-High-School-Microeconomics, MMLU-Econometrics, MMLU-High-School-Macroeconomics, MMLU-High-School-Geography, MMLU-Professional-Psychology | |
|
|
| **MMLU-Stem** | `33.01%` | Averaged over MMLU-Conceptual-Physics, MMLU-High-School-Chemistry, MMLU-College-Biology, MMLU-College-Chemistry, MMLU-Machine-Learning, MMLU-Elementary-Mathematics, MMLU-Abstract-Algebra, MMLU-Astronomy, MMLU-High-School-Statistics, MMLU-Anatomy, MMLU-College-Mathematics, MMLU-Computer-Security, MMLU-College-Computer-Science, MMLU-Electrical-Engineering, MMLU-College-Physics, MMLU-High-School-Computer-Science, MMLU-High-School-Physics, MMLU-High-School-Biology, MMLU-High-School-Mathematics | |
|
|
| **MMLU-Other** | `44.48%` | Averaged over MMLU-Medical-Genetics, MMLU-Global-Facts, MMLU-Marketing, MMLU-College-Medicine, MMLU-Human-Aging, MMLU-Virology, MMLU-Business-Ethics, MMLU-Clinical-Knowledge, MMLU-Professional-Medicine, MMLU-Nutrition, MMLU-Miscellaneous, MMLU-Professional-Accounting, MMLU-Management | |
|
|
|
|
|
|
|
|
|
|
|
📌 Let us know if you need further analysis or model tuning! 🚀 |
