Delete architectures

architectures/codegen.md

@@ -1,25 +0,0 @@
-The CodeGen architecture follows a standard transformer decoder with left-to-right causal masking. With rotary position embedding for the positional encoding [(Su et al., 2021)](https://arxiv.org/abs/2104.09864), and a context length of 2048. CodeGen models are trained in various sizes. 
-
-<div align="center">
-
-|Model | # parameters | 
-|   -   |   -  | 
-| [Salesforce/codegen-350m-mono](https://huggingface.co/Salesforce/codegen-350-mono) | 350M |
-| [Salesforce/codegen-2B-mono](https://huggingface.co/Salesforce/codegen-2B-mono) | 2.7B |
-| [Salesforce/codegen-6B-mono](https://huggingface.co/Salesforce/codegen-6B-mono) | 6.1B |
-| [Salesforce/codegen-16B-mono](https://huggingface.co/Salesforce/codegen-16B-mono) | 16.1B |
-
-</div>
-
-
-You can load the model and tokenizer directly from 🤗 [`transformers`](https://huggingface.co/docs/transformers/index):
-
-```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
-
-tokenizer = AutoTokenizer.from_pretrained('Salesforce/codegen-16B-mono')
-model = AutoModelForCausalLM.from_pretrained('Salesforce/codegen-16B-mono')
-
-inputs = tokenizer("def hello_world():", return_tensors="pt")
-outputs = model(**inputs)
-```

architectures/codeparrot.md

@@ -1,33 +0,0 @@
-[CodeParrot](https://huggingface.co/codeparrot/codeparrot) uses GPT-2 architecture with BPE tokenizer trained on Python code from the training split of the data, and a context length of 1024. This model was released as an educational tool for training large language models from scratch on code, with detailed tutorials and descriptions of the training process. It makes use of 🤗 [`accelerate`](https://huggingface.co/docs/accelerate/index) for distributed training and mixed precision. See this [blog](https://huggingface.co/blog/codeparrot) and [repo](https://github.com/huggingface/transformers/tree/main/examples/research_projects/codeparrot) for more details.
-
-<div align="center">
-
-|Model | # parameters | 
-|   -   |   -  | 
-| [codeparrot-small](https://huggingface.co/codeparrot/codeparrot-small) | 110M |
-| [codeparrot](https://huggingface.co/codeparrot/codeparrot) | 1.5B |
-
-</div>
-
-
-You can load the model and tokenizer directly from 🤗 [`transformers`](https://huggingface.co/docs/transformers/index):
-
-```python
-from transformers import AutoTokenizer, AutoModelWithLMHead
-  
-tokenizer = AutoTokenizer.from_pretrained("codeparrot/codeparrot")
-model = AutoModelWithLMHead.from_pretrained("codeparrot/codeparrot")
-
-inputs = tokenizer("def hello_world():", return_tensors="pt")
-outputs = model(**inputs)
-
-```
-
-You can also use `pipeline` to generate code:
-
-```python
-from transformers import pipeline
-
-pipe = pipeline("text-generation", model="codeparrot/codeparrot")
-outputs = pipe("def hello_world():")
-```

architectures/incoder.md

@@ -1,31 +0,0 @@
-[InCoder](https://huggingface.co/facebook/incoder-6B) uses a decoder-only Transformer with Causal Masking objective, to train a left-to-right language model to fill in masked token segments, with a context length of 2048.
-<div align="center">
-
-|Model | # parameters |
-|   -   |   -  | 
-| [facebook/incoder-1B](https://huggingface.co/facebook/incoder-1B) |1.3B |
-| [facebook/incoder-6B](https://huggingface.co/facebook/incoder-6B) |6.7B |
-
-</div>
-
-[Causal Masking objective](https://arxiv.org/abs/2201.07520) is a hybrid approach of Causal and Masked language models, "it combines the benefit of per-token generation with optional bi-directionality specifically tailored to prompting".
-During the training of InCoder, spans of code were randomly masked and moved to the end of each file, which allows for bidirectional context. Figure below from InCoder [paper](https://arxiv.org/pdf/2204.05999.pdf) illustrates the training process.
-
-<p align="center">
-    <img src="https://huggingface.co/datasets/loubnabnl/repo-images/raw/main/incoder.png" alt="drawing" width="750"/>
-</p>
-
-So in addition to program synthesis (via left-to-right generation), InCoder can also perform editing (via infilling). The model gives promising results in some zero-shot code infilling tasks such as type prediction, variable re-naming and comment generation.
-
-You can load the model and tokenizer directly from 🤗 [`transformers`](https://huggingface.co/docs/transformers/index):
-
-```python
-from transformers import AutoTokenizer, AutoModelWithLMHead
-  
-tokenizer = AutoTokenizer.from_pretrained("facebook/incoder-6B")
-model = AutoModelWithLMHead.from_pretrained("facebook/incoder-6B")
-
-inputs = tokenizer("def hello_world():", return_tensors="pt")
-outputs = model(**inputs)
-
-```

architectures/intro.md

@@ -1,8 +0,0 @@
-Various architectures are used in code generation models, but most of them use the auto-regressive left-to-right setting, such as GPT. However InCoder used a decoder-only Transformer with Causal Masking objective,
-that combines both next token prediction and bidirectional context through masking. AlphaCode used an encoder-decoder architecture.
-
-<p align="center">
-    <img src="https://huggingface.co/datasets/loubnabnl/repo-images/resolve/main/model_size.png" alt="drawing" width="440"/>
-</p>
-
-For model-specific information about each architecture, please select a model below:

architectures/polycoder.md

@@ -1,14 +0,0 @@
-[PolyCoder](https://github.com/VHellendoorn/Code-LMs) uses GPT2 architecture, with BPE tokenizer trained on a random 5% subset of the data (all languages), and a context length of 2048. To study the effect of scaling of model size, the odel was trained in 3 different sizes.
-
-<div align="center">
-
-|Model | # parameters | 
-|   -   |   -  | 
-| GPT2 | 160M |
-| GPT2 | 400M |
-| GPT2 | 2.7B |
-
-</div>
-
-
-PolyCoder is currently being integrated in 🤗 `transformers`. Meanwhile it can be loaded following the instructions in the original GitHub [repo](https://github.com/vhellendoorn/code-lms#models).