

# CodeParrot 🦜

CodeParrot 🦜 is a GPT-2 model (1.5B parameters) trained to generate Python code. After the initial training and release of v1.0 we trained the model some more and released v1.1 (see below for details).

## Usage

You can load the CodeParrot model and tokenizer directly in `transformers`:

```Python
from transformers import AutoTokenizer, AutoModelWithLMHead
  
tokenizer = AutoTokenizer.from_pretrained("codeparrot/codeparrot")
model = AutoModelWithLMHead.from_pretrained("codeparrot/codeparrot")

inputs = tokenizer("def hello_world():", return_tensors="pt")
outputs = model(**inputs)
```

or with a `pipeline`:

```Python
from transformers import pipeline

pipe = pipeline("text-generation", model="codeparrot/codeparrot")
outputs = pipe("def hello_world():")
```

## Training

The model was trained on the cleaned [CodeParrot 🦜 dataset](https://huggingface.co/datasets/codeparrot/codeparrot-clean) in two steps. After the initial training (v1.0) the model was trained for another 30k steps resulting in v1.1 and you find the settings in the following table:

|Config| v1.0| v1.1|
|------|------------------|--------------------|
|Batch size| 512 | 512 |
|Context size| 1024 | 1024 |
|Training steps| 50'000| 30'000
|Gradient accumulation| 16| 16 |
|Gradient checkpointing| True| True |
|Learning rate| 2e-4 | 5e-5 |
|Weight decay | 0.1 | 0.1 |
|Warmup steps| 750 | 750 |
|Schedule| Cosine | Cosine |

The training was executed on 16 x A100 (40GB) GPUs. This setting amounts to roughly 26 + 15 billion tokens.

## Performance

We evaluated the model on OpenAI's [HumanEval](https://huggingface.co/datasets/openai_humaneval) benchmark which consists of programming challenges:

| Metric | v1.0 | v1.1 |
|--------|-----|-----|
|pass@1 | 3.58% | 3.99% |
|pass@10 | 8.03% | 8.69% |
|pass@100 | 14.96% | 17.88% |

The [pass@k metric](https://huggingface.co/metrics/code_eval) tells the probability that at least one out of k generations passes the tests. 

## Resources

- Dataset: [full](https://huggingface.co/datasets/codeparrot/codeparrot-clean), [train](https://huggingface.co/datasets/codeparrot/codeparrot-clean-train), [valid](https://huggingface.co/datasets/codeparrot/codeparrot-clean-valid)
- Code: [repository](https://github.com/huggingface/transformers/tree/master/examples/research_projects/codeparrot)
- Spaces: [generation](), [highlighting]()
{"language": "code", "tags": ["code", "gpt2", "generation"], "datasets": ["codeparrot/codeparrot-clean-train"], "widget": [{"text": "from transformer import", "example_title": "Transformers"}, {"text": "def print_hello_world():\n\t", "example_title": "Hello World!"}, {"text": "def get_file_size(filepath):", "example_title": "File size"}, {"text": "import numpy as", "example_title": "Numpy"}], "model-index": [{"name": "codeparrot", "results": [{"task": {"type": "code-generation", "name": "Code Generation"}, "dataset": {"name": "HumanEval", "type": "openai_humaneval"}, "metrics": [{"type": "code_eval", "value": 3.99, "name": "pass@1"}, {"type": "code_eval", "value": 8.69, "name": "pass@10"}, {"type": "code_eval", "value": 17.88, "name": "pass@100"}]}]}]}
transformers pytorch tensorboard gpt2 text-generation code generation dataset:codeparrot/codeparrot-clean-train model-index autotrain_compatible endpoints_compatible has_space text-generation-inference region:us
TAGS
#transformers #pytorch #tensorboard #gpt2 #text-generation #code #generation #dataset-codeparrot/codeparrot-clean-train #model-index #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 

CodeParrot
==========


CodeParrot is a GPT-2 model (1.5B parameters) trained to generate Python code. After the initial training and release of v1.0 we trained the model some more and released v1.1 (see below for details).


Usage
-----


You can load the CodeParrot model and tokenizer directly in 'transformers':


or with a 'pipeline':


Training
--------


The model was trained on the cleaned CodeParrot dataset in two steps. After the initial training (v1.0) the model was trained for another 30k steps resulting in v1.1 and you find the settings in the following table:


Config: Batch size, v1.0: 512, v1.1: 512
Config: Context size, v1.0: 1024, v1.1: 1024
Config: Training steps, v1.0: 50'000, v1.1: 30'000
Config: Gradient accumulation, v1.0: 16, v1.1: 16
Config: Gradient checkpointing, v1.0: True, v1.1: True
Config: Learning rate, v1.0: 2e-4, v1.1: 5e-5
Config: Weight decay, v1.0: 0.1, v1.1: 0.1
Config: Warmup steps, v1.0: 750, v1.1: 750
Config: Schedule, v1.0: Cosine, v1.1: Cosine


The training was executed on 16 x A100 (40GB) GPUs. This setting amounts to roughly 26 + 15 billion tokens.


Performance
-----------


We evaluated the model on OpenAI's HumanEval benchmark which consists of programming challenges:


Metric: pass@1, v1.0: 3.58%, v1.1: 3.99%
Metric: pass@10, v1.0: 8.03%, v1.1: 8.69%
Metric: pass@100, v1.0: 14.96%, v1.1: 17.88%


The pass@k metric tells the probability that at least one out of k generations passes the tests.


Resources
---------


* Dataset: full, train, valid
* Code: repository
* Spaces: generation, highlighting



TAGS
#transformers #pytorch #tensorboard #gpt2 #text-generation #code #generation #dataset-codeparrot/codeparrot-clean-train #model-index #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 

passage: TAGS
#transformers #pytorch #tensorboard #gpt2 #text-generation #code #generation #dataset-codeparrot/codeparrot-clean-train #model-index #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 
