library_name: peft
base_model: deepseek-ai/deepseek-coder-6.7b-instruct
license: mit
language:
- en
Model Card for Model ID
Fine-tuned version of deepseek-coder-6.7b-instruct aiming to improve vulnerability detection in solidity smart contracts and provide informative explanations on what the vulnerabilities are, and how to solve them.
Model Details
Model Description
Given the following prompt below:
Below are one or more Solidity codeblocks. The codeblocks might contain vulnerable code.
If there is a vulnerability please provide a description of the vulnearblity in terms of the code that is responsible for it.
Describe how an attacker would be able to take advantage of the vulnerability so the explanation is even more clear.
Output only the description of the vulnerability and the attacking vector. No additional information is needed.
If there is no vulnerability output "There is no vulnearbility".
Codeblocks:
{}
When 1 or more codeblocks are provided to the model using this prompt, the model will output:
- Wether there is a vulnerability or not.
- What the vulnerability is.
- How an attacker would take advantage of the detected vulnerability.
Afterwards, the above output can be chained to produce a solution - the context has the code, the vulnerability and the attacking vector so deducing a solution becomes a more straight-forward task. Additionally, the same fine-tuned model can be used for the solution recommendation as the fine-tuning is low-rank (LoRA) and a lot of the model ability is preserved.
- Developed by: [Kristian Apostolov]
- Shared by: [Kristian Apostolov]
- Model type: [Decoder]
- Language(s) (NLP): [English]
- License: [MIT]
- Finetuned from model: [deepseek-ai/deepseek-coder-6.7b-instruct]
Model Sources [optional]
- Repository: [https://huggingface.co/msc-smart-contract-auditing/deepseek-coder-6.7b-vulnerability-detection]
Uses
Provide code from a smart contract for a preliminary audit.
Direct Use
[More Information Needed]
Out-of-Scope Use
Malicious entity could detect 0-day vulnerability and take advantage of it.
Bias, Risks, and Limitations
The training data could be improved. Audits sometimes describe vulnerabilities which are not necessarily contained in the code itself, but are a part of a larger context.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
How to Get Started with the Model
Use the code below to get started with the model.
model_name = 'msc-smart-contract-auditing/deepseek-coder-6.7b-vulnerability'
tokenizer = AutoTokenizer.from_pretrained( # For some reason the tokenizer didn't safe properly
"deepseek-ai/deepseek-coder-6.7b-instruct",
trust_remote_code=True,
force_download=True,
)
prompt = \
"""
Below are one or more Solidity codeblocks. The codeblocks might contain vulnerable code.
If there is a vulnerability please provide a description of the vulnearblity in terms of the code that is responsible for it.
Describe how an attacker would be able to take advantage of the vulnerability so the explanation is even more clear.
Output only the description of the vulnerability and the attacking vector. No additional information is needed.
If there is no vulnerability output "There is no vulnearbility".
Codeblocks:
{}
"""
codeblocks = "Your code here"
messages = [
{ 'role': 'user', 'content': prompt.format(codeblocks) }
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, do_sample=True, top_k=25, top_p=0.95, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)
description = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True)
print(description)
Training Details
Training Data
https://huggingface.co/datasets/msc-smart-contract-auditing/audits-with-reasons
Training Procedure
lora_config = LoraConfig( r=16, # rank lora_alpha=32, # scaling factor target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",], lora_dropout=0.05, # dropout rate for LoRA layers )
model = get_peft_model(model, lora_config)
from transformers import Trainer, TrainingArguments
trainer = Trainer( model=model, args=TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, num_train_epochs = 1, learning_rate = 2e-4, fp16 = True, logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", ), train_dataset=train_prompts, eval_dataset=test_prompts, )
Training Hyperparameters
- Training regime: fp16 mixed precision
Evaluation
Testing Data, Factors & Metrics
Testing Data
https://huggingface.co/datasets/msc-smart-contract-auditing/audits-with-reasons
Factors
[More Information Needed]
Metrics
[More Information Needed]
Results
[More Information Needed]
Summary
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: [More Information Needed]
- Hours used: [More Information Needed]
- Cloud Provider: [More Information Needed]
- Compute Region: [More Information Needed]
- Carbon Emitted: [More Information Needed]
Technical Specifications [optional]
Model Architecture and Objective
[More Information Needed]
Compute Infrastructure
[More Information Needed]
Hardware
[More Information Needed]
Software
[More Information Needed]
Citation [optional]
BibTeX:
[More Information Needed]
APA:
[More Information Needed]
Glossary [optional]
[More Information Needed]
More Information [optional]
[More Information Needed]
Model Card Authors [optional]
[More Information Needed]
Model Card Contact
[More Information Needed]
Framework versions
- PEFT 0.11.1