File size: 14,000 Bytes
a209986 98544a4 a209986 98544a4 a209986 58ada82 a209986 9898ce0 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 58ada82 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 98544a4 a209986 58ada82 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 |
---
library_name: transformers
license: llama3.2
base_model: meta-llama/Llama-3.2-3B
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: code-knowledge-eval
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# **Llama-3.2-3B-Code-Knowledge-Value-Eval**
This model is a fine-tuned version of [meta-llama/Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B) on the [kimsan0622/code-knowledge-eval](https://huggingface.co/datasets/kimsan0622/code-knowledge-eval) dataset.
It achieves the following results on the evaluation set:
- Loss: 0.9977
- Accuracy: 0.5552
## **Model Description**
The model trained on the **Code Knowledge Value Evaluation Dataset** is designed to assess the educational and knowledge value of code snippets. It leverages patterns and contextual information from a large collection of open-source code, sourced from the `bigcode/the-stack` repository. By analyzing these code samples, the model can evaluate their utility in teaching coding concepts, solving problems, and improving developer education.
The model focuses on understanding the structure, syntax, and logic of various programming languages, enabling it to provide insights into the learning potential and technical depth of different code samples. The dataset used for training consists of 22,786 samples for training, 4,555 for validation, and 18,232 for testing, ensuring that the model is both robust and well-generalized across different coding contexts.
## **Intended Uses & Limitations**
### **Intended Uses**:
1. **Automated Code Review**: The model can be applied in automated systems to assess the knowledge value of code during code review processes. It can help identify areas where code could be optimized for better readability, maintainability, and educational impact.
2. **Educational Feedback**: For instructors and educational platforms, the model can offer feedback on the effectiveness of code samples used in teaching, helping to improve curriculum materials and select code that best conveys core programming concepts.
3. **Curriculum Development**: The model can aid in designing coding courses or instructional materials by suggesting code examples that have higher educational value, supporting a more effective learning experience.
4. **Technical Skill Assessment**: Organizations or platforms can use the model to assess the complexity and educational value of code submissions in coding challenges or exams.
### **Limitations**:
1. **Narrow Scope in Knowledge Evaluation**: The model is specialized in evaluating code from an educational standpoint, focusing primarily on learning potential rather than production-level code quality (e.g., performance optimization or security).
2. **Language and Domain Limitations**: Since the dataset is sourced from `bigcode/the-stack`, it may not cover all programming languages or specialized domains. The model may perform less effectively in underrepresented languages or niche coding styles not well-represented in the dataset.
3. **Not Suitable for All Educational Levels**: While the model is designed to evaluate code for educational purposes, its outputs may be better suited for certain levels (e.g., beginner or intermediate coding), and its recommendations might not fully cater to advanced or highly specialized learners.
## How to use this model?
```python
import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification
# Define the model name or path for loading the tokenizer and model
model_name_or_path = "kimsan0622/Llama-3.2-3B-Code-Knowledge-Value-Eval"
# Load the tokenizer from the pre-trained model
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
# Load the pre-trained model for sequence classification and map it to the first CUDA device
model = AutoModelForSequenceClassification.from_pretrained(
model_name_or_path,
device_map="cuda:0",
)
# Example code snippet to be evaluated
code = [
"""
import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification
# Define the model name or path for loading the tokenizer and model
model_name_or_path = "kimsan0622/Llama-3.2-1B-Code-Knowledge-Value-Eval"
# Load the tokenizer from the pre-trained model
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
# Load the pre-trained model for sequence classification and map it to the first CUDA device
model = AutoModelForSequenceClassification.from_pretrained(
model_name_or_path,
device_map="cuda:0",
)
# Example code snippet to be evaluated
code = ["code 1"]
# Tokenize the input code, setting max length, padding, and truncation
batch = tokenizer(code, max_length=1024, padding=True, truncation=True, return_tensors="pt")
# Perform inference with the model, without computing gradients (for faster inference)
with torch.no_grad():
# Pass the input IDs and attention mask to the model, using the CUDA device
res = model(
input_ids=batch["input_ids"].to("cuda:0"),
attention_mask=batch["attention_mask"].to("cuda:0"),
)
# Move the logits to the CPU, convert them to a numpy array
preds = res.logits.cpu().numpy()
# Get the predicted class by taking the argmax of the logits across the classification axis
preds = np.argmax(preds, axis=1).tolist()
"""
]
# Tokenize the input code, setting max length, padding, and truncation
batch = tokenizer(code, max_length=1024, padding=True, truncation=True, return_tensors="pt")
# Perform inference with the model, without computing gradients (for faster inference)
with torch.no_grad():
# Pass the input IDs and attention mask to the model, using the CUDA device
res = model(
input_ids=batch["input_ids"].to("cuda:0"),
attention_mask=batch["attention_mask"].to("cuda:0"),
)
# Move the logits to the CPU, convert them to a numpy array
preds = res.logits.cpu().numpy()
# Get the predicted class by taking the argmax of the logits across the classification axis
preds = np.argmax(preds, axis=1).tolist()
print(preds)
```
### 8 Bit quantization
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig
# Define the model name or path for loading the model
model_name_or_path = "kimsan0622/Llama-3.2-3B-Code-Knowledge-Value-Eval"
# Configure the model to load in 8-bit precision for memory efficiency
bnb_config = BitsAndBytesConfig(load_in_8bit=True)
# Load the pre-trained model for sequence classification with quantization for 8-bit precision
# This helps reduce memory usage, particularly for large models, and map it to the first CUDA device
model = AutoModelForSequenceClassification.from_pretrained(
model_name_or_path,
quantization_config=bnb_config, # Apply 8-bit quantization
device_map="cuda:0", # Map the model to the first CUDA device
)
```
### 4 Bit quntization
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig
import torch
# Define the model name or path for loading the model
model_name_or_path = "kimsan0622/Llama-3.2-3B-Code-Knowledge-Value-Eval"
# Define configuration parameters for 4-bit quantization
bnb_config_params = {
"bnb_4bit_quant_type": "fp4", # Use FP4 for 4-bit quantization type
"bnb_4bit_compute_dtype": torch.bfloat16, # Use bfloat16 for computation to balance performance and precision
"bnb_4bit_use_double_quant": False, # Disable double quantization, which is typically used to further reduce precision
"bnb_4bit_quant_storage": torch.bfloat16, # Store quantized values in bfloat16 format
}
# Configure the model to load in 4-bit precision for memory and performance optimization
bnb_config = BitsAndBytesConfig(load_in_4bit=True, **bnb_config_params)
# Load the pre-trained model for sequence classification with 4-bit quantization settings
# This reduces memory usage while still maintaining reasonable accuracy, mapping the model to the first CUDA device
model = AutoModelForSequenceClassification.from_pretrained(
model_name_or_path,
quantization_config=bnb_config, # Apply 4-bit quantization configuration
device_map="cuda:0", # Map the model to the first CUDA device
)
```
## Training and evaluation data
[kimsan0622/code-knowledge-eval](https://huggingface.co/datasets/kimsan0622/code-knowledge-eval)
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 8
- total_train_batch_size: 128
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5.0
### Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:------:|:----:|:---------------:|:--------:|
| 1.1115 | 0.9993 | 178 | 1.0997 | 0.4900 |
| 0.9391 | 1.9986 | 356 | 0.9977 | 0.5552 |
| 0.6717 | 2.9979 | 534 | 1.1785 | 0.5431 |
| 0.2989 | 3.9972 | 712 | 1.8204 | 0.5405 |
| 0.133 | 4.9965 | 890 | 2.5260 | 0.5482 |
### Framework versions
- Transformers 4.44.2
- Pytorch 2.3.0
- Datasets 2.20.0
- Tokenizers 0.19.1
## Test set results
### Confusion matrix
| **y_true** |**pred_0**|**pred_1**|**pred_2**|**pred_3**|**pred_4**|**pred_5**|
|:----------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| 0 | 940 | 268 | 107 | 57 | 5 | 0 |
| 1 | 316 | 344 | 338 | 221 | 24 | 1 |
| 2 | 132 | 251 | 510 | 738 | 135 | 8 |
| 3 | 43 | 128 | 458 | 2674 | 1565 | 57 |
| 4 | 8 | 15 | 74 | 1248 | 4229 | 750 |
| 5 | 0 | 2 | 0 | 29 | 1094 | 1463 |
### Classification report
| | **precision** | **recall** | **f1-score** | **support** |
|:-------------:|:-------------:|:----------:|:------------:|:-----------:|
| 0 | 0.65 | 0.68 | 0.67 | 1377 |
| 1 | 0.34 | 0.28 | 0.31 | 1244 |
| 2 | 0.34 | 0.29 | 0.31 | 1774 |
| 3 | 0.54 | 0.54 | 0.54 | 4925 |
| 4 | 0.60 | 0.67 | 0.63 | 6324 |
| 5 | 0.64 | 0.57 | 0.60 | 2588 |
| **accuracy** | | | 0.56 | 18232 |
| **macro avg**| 0.52 | 0.50 | 0.51 | 18232 |
| **weighted avg** | 0.55 | 0.56 | 0.55 | 18232 |
## 8 Bit quantization model
### Confusion matrix
| **y_true** |**pred_0**|**pred_1**|**pred_2**|**pred_3**|**pred_4**|**pred_5**|
|-------|-------|-------|-------|-------|-------|-------|
| 0 | 933 | 272 | 111 | 56 | 5 | 0 |
| 1 | 320 | 333 | 340 | 224 | 25 | 2 |
| 2 | 129 | 244 | 502 | 752 | 139 | 8 |
| 3 | 43 | 119 | 463 | 2627 | 1604 | 69 |
| 4 | 7 | 15 | 67 | 1227 | 4191 | 817 |
| 5 | 0 | 2 | 0 | 30 | 1034 | 1522 |
### Classification report
| **y_true** | **precision** | **recall** | **f1-score** | **support** |
|:-------------:|:-------------:|:----------:|:------------:|:-----------:|
| 0 | 0.65 | 0.68 | 0.66 | 1377 |
| 1 | 0.34 | 0.27 | 0.30 | 1244 |
| 2 | 0.34 | 0.28 | 0.31 | 1774 |
| 3 | 0.53 | 0.53 | 0.53 | 4925 |
| 4 | 0.60 | 0.66 | 0.63 | 6324 |
| 5 | 0.63 | 0.59 | 0.61 | 2588 |
| **accuracy** | | | 0.55 | 18232 |
| **macro avg**| 0.52 | 0.50 | 0.51 | 18232 |
| **weighted avg** | 0.55 | 0.55 | 0.55 | 18232 |
## 4 Bit quantization model
### Confusion matrix
| **y_true** |**pred_0**|**pred_1**|**pred_2**|**pred_3**|**pred_4**|**pred_5**|
|-------|-------|-------|-------|-------|-------|-------|
| 0 | 695 | 581 | 42 | 55 | 4 | 0 |
| 1 | 151 | 662 | 190 | 215 | 24 | 2 |
| 2 | 53 | 485 | 353 | 716 | 159 | 8 |
| 3 | 20 | 277 | 335 | 2446 | 1765 | 82 |
| 4 | 4 | 31 | 60 | 1104 | 4211 | 914 |
| 5 | 0 | 2 | 0 | 24 | 934 | 1628 |
### Classification report
| | **precision** | **recall** | **f1-score** | **support** |
|:-------------:|:-------------:|:----------:|:------------:|:-----------:|
| 0 | 0.75 | 0.50 | 0.60 | 1377 |
| 1 | 0.32 | 0.53 | 0.40 | 1244 |
| 2 | 0.36 | 0.20 | 0.26 | 1774 |
| 3 | 0.54 | 0.50 | 0.52 | 4925 |
| 4 | 0.59 | 0.67 | 0.63 | 6324 |
| 5 | 0.62 | 0.63 | 0.62 | 2588 |
| **accuracy** | | | 0.55 | 18232 |
| **macro avg**| 0.53 | 0.50 | 0.51 | 18232 |
| **weighted avg** | 0.55 | 0.55 | 0.54 | 18232 |
|