Hugging Face's logo

patched-codes
/
patched-coder-34b

Text Generation
Transformers
PyTorch
Safetensors
llama
Eval Results
text-generation-inference
Model card Files Community
patched-coder-34b
File size: 3,366 Bytes 
768fb2e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
 
---
inference: false
datasets:
- bigcode/commitpackft
model-index:
  - name: patched-coder-34b
    results:
      - task:
          type: text-generation
        dataset:
          type: openai_humaneval
          name: HumanEval
        metrics:
          - name: pass@1
            type: pass@1
            value: 53.567
            verified: false
      - task:
          type: text-generation
        dataset:
          type: bigcode/humanevalpack
          name: HumanEvalFix Python
        metrics:
          - name: pass@1
            type: pass@1
            value: 41.341
            verified: false
      - task:
          type: text-generation
        dataset:
          type: patched-codes/static-analysis-eval
          name: Static Analysis Eval
        metrics:
          - name: pass@1
            type: pass@1
            value: 51.316
            verified: false
---
# Model Card for patched-coder-34b


This is an instruction fine-tuned model focussed on the task of patching code. Patching may include fixing bugs, remediating security vulnerabilities, 
doing API migrations and other kinds of code matainence.

## Model Details

### Model Description

- **Developed by:** [codelion](https://huggingface.co/codelion)
- **Model type:** Code Llama
- **Finetuned from model:** [CodeLlama-34b-Python](https://huggingface.co/codellama/CodeLlama-34b-Python-hf)


## How to Get Started with the Model

Make sure to install Transformers from the main git branch:

```bash
pip install git+https://github.com/huggingface/transformers.git
```

## How to Prompt the Model

This model accepts the alpaca instruction format.

For example: 

```
### Instruction:
{instruction}

### Input:
{input}

### Response:
...
```

## Bias, Risks, and Limitations

This model has undergone very limited testing. Additional safety testing should be performed before any real-world deployments.

## Training Details

- **GPU:** A100 80 GB
- **Time:** ~8 hrs

### Training Data

The model was fine-tuned on [commitpackft](https://huggingface.co/datasets/bigcode/commitpackft), an open dataset consisting of commits.
We started with the commits for the `python` langauge from the dataset and then filtered all the commits that were related to fixing bugs. 

### Training Procedure 

Instruction fine-tuning to follow instructions in natural langauge related to code. We load the quantized base model in 4 bits 
and then use QLoRA for Parameter-Efficient Fine-Tuning (PEFT) with Flash Attention. The model was trained for 2 epochs.

#### Training Hyperparameters

**Training regime:** 

The following `bitsandbytes` quantization config was used during training:
  - quant_method: bitsandbytes
  - load_in_8bit: False
  - load_in_4bit: True
  - llm_int8_threshold: 6.0
  - llm_int8_skip_modules: None
  - llm_int8_enable_fp32_cpu_offload: False
  - llm_int8_has_fp16_weight: False
  - bnb_4bit_quant_type: nf4
  - bnb_4bit_use_double_quant: True
  - bnb_4bit_compute_dtype: bfloat16

## Evaluation

We evaluate the model on `HumanEval` and `HumanEvalPack` benchmarks using 
[Code Generation LM Evaluation Harness](https://github.com/bigcode-project/bigcode-evaluation-harness).

We also evaluate the model for vulnerability remediation using the `Static Analysis Eval` benchmark available [here](https://huggingface.co/datasets/patched-codes/static-analysis-eval).

### Results