README.md

---
library_name: peft
base_model: deepseek-ai/deepseek-coder-6.7b-instruct
license: mit
language:
- en
---

# Model Card for Model ID
Fine-tuned version of `deepseek-coder-6.7b-instruct` aiming to improve vulnerability detection in solidity smart contracts and provide informative explanations on what the vulnerabilities are, and how to solve them.


## Model Details

### Model Description

Given the following prompt below:
```
Below are one or more Solidity codeblocks. The codeblocks might contain vulnerable code.
If there is a vulnerability please provide a description of the vulnearblity in terms of the code that is responsible for it.
Describe how an attacker would be able to take advantage of the vulnerability so the explanation is even more clear.

Output only the description of the vulnerability and the attacking vector. No additional information is needed.

If there is no vulnerability output "There is no vulnearbility".

Codeblocks:
{}
```

When 1 or more codeblocks are provided to the model using this prompt, the model will output:
1. Wether there is a vulnerability or not.
2. What the vulnerability is.
3. How an attacker would take advantage of the detected vulnerability.

Afterwards, the above output can be chained to produce a solution - the context has the code, the vulnerability and the attacking vector so deducing a solution becomes a more straight-forward task.
Additionally, the same fine-tuned model can be used for the solution recommendation as the fine-tuning is low-rank (LoRA) and a lot of the model ability is preserved. 


- **Developed by:** [Kristian Apostolov]
- **Shared by:** [Kristian Apostolov]
- **Model type:** [Decoder]
- **Language(s) (NLP):** [English]
- **License:** [MIT]
- **Finetuned from model:** [deepseek-ai/deepseek-coder-6.7b-instruct]

### Model Sources [optional]

- **Repository:** [https://huggingface.co/msc-smart-contract-auditing/deepseek-coder-6.7b-vulnerability-detection]

## Uses

Provide code from a smart contract for a preliminary audit.

### Direct Use

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

Malicious entity could detect 0-day vulnerability and take advantage of it.

## Bias, Risks, and Limitations

The training data could be improved. Audits sometimes describe vulnerabilities which are not necessarily contained in the code itself, but are a part of a larger context. 

### Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. 

## How to Get Started with the Model

Use the code below to get started with the model.

```python
model_name = 'msc-smart-contract-auditing/deepseek-coder-6.7b-vulnerability'
tokenizer = AutoTokenizer.from_pretrained( # For some reason the tokenizer didn't safe properly
    "deepseek-ai/deepseek-coder-6.7b-instruct", 
    trust_remote_code=True, 
    force_download=True,
)

prompt = \
"""
Below are one or more Solidity codeblocks. The codeblocks might contain vulnerable code.
If there is a vulnerability please provide a description of the vulnearblity in terms of the code that is responsible for it.
Describe how an attacker would be able to take advantage of the vulnerability so the explanation is even more clear.

Output only the description of the vulnerability and the attacking vector. No additional information is needed.

If there is no vulnerability output "There is no vulnearbility".

Codeblocks:
{}

"""

codeblocks = "Your code here"

messages = [
    { 'role': 'user', 'content': prompt.format(codeblocks) }
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, do_sample=True, top_k=25, top_p=0.95, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)
description = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True) 

print(description)
```
## Training Details

### Training Data

https://huggingface.co/datasets/msc-smart-contract-auditing/audits-with-reasons

### Training Procedure

lora_config = LoraConfig(
    r=16,  # rank
    lora_alpha=32,  # scaling factor
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",],
    lora_dropout=0.05,  # dropout rate for LoRA layers
)

model = get_peft_model(model, lora_config)

from transformers import Trainer, TrainingArguments

trainer = Trainer(
    model=model,
    args=TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        num_train_epochs = 1,
        learning_rate = 2e-4,
        fp16 = True,
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = "outputs",
    ),
    train_dataset=train_prompts,
    eval_dataset=test_prompts,
)

#### Training Hyperparameters

- **Training regime:** fp16 mixed precision

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

https://huggingface.co/datasets/msc-smart-contract-auditing/audits-with-reasons

#### Factors

[More Information Needed]

#### Metrics

[More Information Needed]

### Results

[More Information Needed]

#### Summary



## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]

## Technical Specifications [optional]

### Model Architecture and Objective

[More Information Needed]

### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

[More Information Needed]

## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Model Card Authors [optional]

[More Information Needed]

## Model Card Contact

[More Information Needed]
### Framework versions

- PEFT 0.11.1