
# GPT-2 finetuned on French Dataset

### Tokenizer
We first trained a tokenizer on OSCAR's `unshuffled_original_fr` French data subset by following the training of GPT2 tokenizer (same vocab size of 50,257). Here's the [Python file](https://github.com/bigscience-workshop/multilingual-modeling/blob/gpt2-fr/experiments/exp-001/train_tokenizer_gpt2.py) for the training.

### Model
We finetuned the `wte` and `wpe` layers of GPT-2 (while freezing the parameters of all other layers) on OSCAR's `unshuffled_original_fr` French data subset. We used [Huggingface's code](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm.py) for fine-tuning the causal language model GPT-2, but with the following parameters changed
```
- preprocessing_num_workers: 8
- per_device_train_batch_size: 2
- gradient_accumulation_steps: 4
- per_device_eval_batch_size: 2
- eval_accumulation_steps: 4
- eval_steps: 1000 
- evaluation_strategy: "steps"
- max_eval_samples: 5000
```

**Setup**: 8 RTX-3090 GPUs, trained for seven days (total training steps: 110500, effective train batch size: 64, tokens per batch: 1024)

**Final checkpoint**: checkpoint-111500
{"language": ["fr"], "license": "mit", "tags": ["text-generation"], "datasets": ["oscar"], "widget": [{"text": "Je suis ravi de vous "}]}
transformers pytorch gpt2 feature-extraction text-generation fr dataset:oscar license:mit endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #gpt2 #feature-extraction #text-generation #fr #dataset-oscar #license-mit #endpoints_compatible #text-generation-inference #region-us 


# GPT-2 finetuned on French Dataset

### Tokenizer
We first trained a tokenizer on OSCAR's 'unshuffled_original_fr' French data subset by following the training of GPT2 tokenizer (same vocab size of 50,257). Here's the Python file for the training.

### Model
We finetuned the 'wte' and 'wpe' layers of GPT-2 (while freezing the parameters of all other layers) on OSCAR's 'unshuffled_original_fr' French data subset. We used Huggingface's code for fine-tuning the causal language model GPT-2, but with the following parameters changed


Setup: 8 RTX-3090 GPUs, trained for seven days (total training steps: 110500, effective train batch size: 64, tokens per batch: 1024)

Final checkpoint: checkpoint-111500
# GPT-2 finetuned on French Dataset ### Tokenizer
We first trained a tokenizer on OSCAR's 'unshuffled_original_fr' French data subset by following the training of GPT2 tokenizer (same vocab size of 50,257). Here's the Python file for the training. ### Model
We finetuned the 'wte' and 'wpe' layers of GPT-2 (while freezing the parameters of all other layers) on OSCAR's 'unshuffled_original_fr' French data subset. We used Huggingface's code for fine-tuning the causal language model GPT-2, but with the following parameters changed


Setup: 8 RTX-3090 GPUs, trained for seven days (total training steps: 110500, effective train batch size: 64, tokens per batch: 1024)

Final checkpoint: checkpoint-111500
TAGS
#transformers #pytorch #gpt2 #feature-extraction #text-generation #fr #dataset-oscar #license-mit #endpoints_compatible #text-generation-inference #region-us 
 # GPT-2 finetuned on French Dataset ### Tokenizer
We first trained a tokenizer on OSCAR's 'unshuffled_original_fr' French data subset by following the training of GPT2 tokenizer (same vocab size of 50,257). Here's the Python file for the training. ### Model
We finetuned the 'wte' and 'wpe' layers of GPT-2 (while freezing the parameters of all other layers) on OSCAR's 'unshuffled_original_fr' French data subset. We used Huggingface's code for fine-tuning the causal language model GPT-2, but with the following parameters changed


Setup: 8 RTX-3090 GPUs, trained for seven days (total training steps: 110500, effective train batch size: 64, tokens per batch: 1024)

Final checkpoint: checkpoint-111500
passage: TAGS
#transformers #pytorch #gpt2 #feature-extraction #text-generation #fr #dataset-oscar #license-mit #endpoints_compatible #text-generation-inference #region-us 
# GPT-2 finetuned on French Dataset### Tokenizer
We first trained a tokenizer on OSCAR's 'unshuffled_original_fr' French data subset by following the training of GPT2 tokenizer (same vocab size of 50,257). Here's the Python file for the training.### Model
We finetuned the 'wte' and 'wpe' layers of GPT-2 (while freezing the parameters of all other layers) on OSCAR's 'unshuffled_original_fr' French data subset. We used Huggingface's code for fine-tuning the causal language model GPT-2, but with the following parameters changed


Setup: 8 RTX-3090 GPUs, trained for seven days (total training steps: 110500, effective train batch size: 64, tokens per batch: 1024)

Final checkpoint: checkpoint-111500