|
--- |
|
library_name: transformers |
|
tags: |
|
- code |
|
license: mit |
|
datasets: |
|
- ArtifactAI/arxiv_python_research_code |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
A parameter-efficient finetune (using LoRA) of DeepSeek Coder 1.3B finetuned on Python code. |
|
|
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
A finetune of DeepSeek Coder 1.3B finetuned on 1000 examples of Python code from the ArtifactAI/arxiv_python_research_code dataset. |
|
|
|
- **Model type:** Text Generation |
|
- **Language(s) (NLP):** English, Python |
|
- **Finetuned from model:** deepseek-ai/deepseek-coder-1.3b-base |
|
|
|
### Model Sources [optional] |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** https://github.com/kevin-v96/python-codecomplete-lm |
|
|
|
## Uses |
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
To generate Python code |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
``` |
|
from transformers import pipeline |
|
|
|
model_name = "MadMarx37/deepseek-coder-1.3b-python-peft" |
|
|
|
def generate_output(input): |
|
# Run text generation pipeline with our next model |
|
pipe = pipeline(task="text-generation", model=model_name, tokenizer=model_name, max_length=max_length) |
|
result = pipe(input) |
|
print(result[0]['generated_text']) |
|
``` |
|
|
|
## Training Details |
|
|
|
#### Training Hyperparameters |
|
|
|
- Training regime: fp16 mixed-precision with original model loaded in 4bits with bitsandbytes <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision --> |
|
- learning_rate = 2e-3 |
|
- lr_scheduler_type = 'cosine_with_restarts' |
|
- max_grad_norm = 0.001 |
|
- weight_decay = 0.001 |
|
- num_train_epochs = 15 |
|
- eval_strategy = "steps" |
|
- eval_steps = 25 |
|
|
|
#### Speeds, Sizes, Times [optional] |
|
|
|
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. --> |
|
|
|
1.3B parameters. Training time of ~2 hours on an RTX3080. |
|
|
|
## Evaluation |
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
<!-- This should link to a Dataset Card if possible. --> |
|
|
|
https://huggingface.co/datasets/ArtifactAI/arxiv_python_research_code |
|
|
|
#### Metrics |
|
|
|
<!-- These are the evaluation metrics being used, ideally with a description of why. --> |
|
|
|
Standard training and eval loss from the HF SFTTrainer. |
|
|
|
### Results |
|
|
|
Training Loss: 0.074100 |
|
Validation Loss: 0.022271 |
|
|
|
#### Summary |
|
|
|
The training had some unstability in the gradient norms, but the overall trend in both training and validation loss |
|
were downward, and validation loss has almost plateaud, which is ideally where we want our model. The code generation on the same |
|
prompts that we tested the original model on also seem better with the finetuned model. A good way to make the model better, if |
|
we wanted to increase the finetuning data, would be to also increase the epochs. |
|
|
|
The training run metrics can be seen here: |
|
https://wandb.ai/kevinv3796/python-autocomplete-deepseek/reports/Supervised-Finetuning-run-for-DeepSeek-Coder-1-3B-on-Python-Code--Vmlldzo3NzQ4NjY0?accessToken=bo0rlzp0yj9vxf1xe3fybfv6rbgl97w5kkab478t8f5unbwltdczy63ba9o9kwjp |
|
|
