|
--- |
|
library_name: peft |
|
base_model: deepseek-ai/deepseek-coder-6.7b-instruct |
|
license: mit |
|
language: |
|
- en |
|
--- |
|
|
|
# Model Card for Model ID |
|
Fine-tuned version of `deepseek-coder-6.7b-instruct` aiming to improve vulnerability detection in solidity smart contracts and provide informative explanations on what the vulnerabilities are, and how to solve them. |
|
|
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
Given the following prompt below: |
|
``` |
|
Below are one or more Solidity codeblocks. The codeblocks might contain vulnerable code. |
|
If there is a vulnerability please provide a description of the vulnearblity in terms of the code that is responsible for it. |
|
Describe how an attacker would be able to take advantage of the vulnerability so the explanation is even more clear. |
|
|
|
Output only the description of the vulnerability and the attacking vector. No additional information is needed. |
|
|
|
If there is no vulnerability output "There is no vulnearbility". |
|
|
|
Codeblocks: |
|
{} |
|
``` |
|
|
|
When 1 or more codeblocks are provided to the model using this prompt, the model will output: |
|
1. Wether there is a vulnerability or not. |
|
2. What the vulnerability is. |
|
3. How an attacker would take advantage of the detected vulnerability. |
|
|
|
Afterwards, the above output can be chained to produce a solution - the context has the code, the vulnerability and the attacking vector so deducing a solution becomes a more straight-forward task. |
|
Additionally, the same fine-tuned model can be used for the solution recommendation as the fine-tuning is low-rank (LoRA) and a lot of the model ability is preserved. |
|
|
|
|
|
- **Developed by:** [Kristian Apostolov] |
|
- **Shared by:** [Kristian Apostolov] |
|
- **Model type:** [Decoder] |
|
- **Language(s) (NLP):** [English] |
|
- **License:** [MIT] |
|
- **Finetuned from model:** [deepseek-ai/deepseek-coder-6.7b-instruct] |
|
|
|
### Model Sources [optional] |
|
|
|
- **Repository:** [https://huggingface.co/msc-smart-contract-auditing/deepseek-coder-6.7b-vulnerability-detection] |
|
|
|
## Uses |
|
|
|
Provide code from a smart contract for a preliminary audit. |
|
|
|
### Direct Use |
|
|
|
[More Information Needed] |
|
|
|
### Out-of-Scope Use |
|
|
|
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. --> |
|
|
|
Malicious entity could detect 0-day vulnerability and take advantage of it. |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
The training data could be improved. Audits sometimes describe vulnerabilities which are not necessarily contained in the code itself, but are a part of a larger context. |
|
|
|
### Recommendations |
|
|
|
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
```python |
|
model_name = 'msc-smart-contract-auditing/deepseek-coder-6.7b-vulnerability' |
|
tokenizer = AutoTokenizer.from_pretrained( # For some reason the tokenizer didn't safe properly |
|
"deepseek-ai/deepseek-coder-6.7b-instruct", |
|
trust_remote_code=True, |
|
force_download=True, |
|
) |
|
|
|
prompt = \ |
|
""" |
|
Below are one or more Solidity codeblocks. The codeblocks might contain vulnerable code. |
|
If there is a vulnerability please provide a description of the vulnearblity in terms of the code that is responsible for it. |
|
Describe how an attacker would be able to take advantage of the vulnerability so the explanation is even more clear. |
|
|
|
Output only the description of the vulnerability and the attacking vector. No additional information is needed. |
|
|
|
If there is no vulnerability output "There is no vulnearbility". |
|
|
|
Codeblocks: |
|
{} |
|
|
|
""" |
|
|
|
codeblocks = "Your code here" |
|
|
|
messages = [ |
|
{ 'role': 'user', 'content': prompt.format(codeblocks) } |
|
] |
|
|
|
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device) |
|
outputs = model.generate(inputs, max_new_tokens=512, do_sample=True, top_k=25, top_p=0.95, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id) |
|
description = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True) |
|
|
|
print(description) |
|
``` |
|
## Training Details |
|
|
|
### Training Data |
|
|
|
https://huggingface.co/datasets/msc-smart-contract-auditing/audits-with-reasons |
|
|
|
### Training Procedure |
|
|
|
lora_config = LoraConfig( |
|
r=16, # rank |
|
lora_alpha=32, # scaling factor |
|
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",], |
|
lora_dropout=0.05, # dropout rate for LoRA layers |
|
) |
|
|
|
model = get_peft_model(model, lora_config) |
|
|
|
from transformers import Trainer, TrainingArguments |
|
|
|
trainer = Trainer( |
|
model=model, |
|
args=TrainingArguments( |
|
per_device_train_batch_size = 2, |
|
gradient_accumulation_steps = 4, |
|
warmup_steps = 5, |
|
num_train_epochs = 1, |
|
learning_rate = 2e-4, |
|
fp16 = True, |
|
logging_steps = 1, |
|
optim = "adamw_8bit", |
|
weight_decay = 0.01, |
|
lr_scheduler_type = "linear", |
|
seed = 3407, |
|
output_dir = "outputs", |
|
), |
|
train_dataset=train_prompts, |
|
eval_dataset=test_prompts, |
|
) |
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** fp16 mixed precision |
|
|
|
## Evaluation |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
https://huggingface.co/datasets/msc-smart-contract-auditing/audits-with-reasons |
|
|
|
#### Factors |
|
|
|
[More Information Needed] |
|
|
|
#### Metrics |
|
|
|
[More Information Needed] |
|
|
|
### Results |
|
|
|
[More Information Needed] |
|
|
|
#### Summary |
|
|
|
|
|
|
|
## Environmental Impact |
|
|
|
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> |
|
|
|
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
|
|
|
- **Hardware Type:** [More Information Needed] |
|
- **Hours used:** [More Information Needed] |
|
- **Cloud Provider:** [More Information Needed] |
|
- **Compute Region:** [More Information Needed] |
|
- **Carbon Emitted:** [More Information Needed] |
|
|
|
## Technical Specifications [optional] |
|
|
|
### Model Architecture and Objective |
|
|
|
[More Information Needed] |
|
|
|
### Compute Infrastructure |
|
|
|
[More Information Needed] |
|
|
|
#### Hardware |
|
|
|
[More Information Needed] |
|
|
|
#### Software |
|
|
|
[More Information Needed] |
|
|
|
## Citation [optional] |
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
**BibTeX:** |
|
|
|
[More Information Needed] |
|
|
|
**APA:** |
|
|
|
[More Information Needed] |
|
|
|
## Glossary [optional] |
|
|
|
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. --> |
|
|
|
[More Information Needed] |
|
|
|
## More Information [optional] |
|
|
|
[More Information Needed] |
|
|
|
## Model Card Authors [optional] |
|
|
|
[More Information Needed] |
|
|
|
## Model Card Contact |
|
|
|
[More Information Needed] |
|
### Framework versions |
|
|
|
- PEFT 0.11.1 |
