
# Replication of [gpt2-wechsel-german](https://huggingface.co/benjamin/gpt2-wechsel-german)

- trained with [BigScience's DeepSpeed-Megatron-LM code base](https://github.com/bigscience-workshop/Megatron-DeepSpeed)
- 22hrs on 4xA100 GPUs (~ 80 TFLOPs / GPU)
- stopped after 100k steps
- less than a single epoch on `oscar_unshuffled_deduplicated_de` (excluding validation set; original model was trained for 75 epochs on less data)
- bf16
- zero stage 1
- tp/pp = 1

## Evaluation

| Model | PPL |
|---|---|
| `gpt2-wechsel-german-ds-meg` | **26.4** |
| `gpt2-wechsel-german` | 26.8 |
| `gpt2` (retrained from scratch)  | 27.63 |

## License

MIT



{"language": "de", "license": "mit", "widget": [{"text": "In einer schockierenden Entdeckung fanden Wissenschaftler eine Herde Einh\u00f6rner, die in einem abgelegenen, zuvor unerforschten Tal in den Anden lebten."}]}
transformers pytorch safetensors gpt2 text-generation de license:mit autotrain_compatible endpoints_compatible has_space text-generation-inference region:us
TAGS
#transformers #pytorch #safetensors #gpt2 #text-generation #de #license-mit #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 

Replication of gpt2-wechsel-german
==================================


* trained with BigScience's DeepSpeed-Megatron-LM code base
* 22hrs on 4xA100 GPUs (~ 80 TFLOPs / GPU)
* stopped after 100k steps
* less than a single epoch on 'oscar\_unshuffled\_deduplicated\_de' (excluding validation set; original model was trained for 75 epochs on less data)
* bf16
* zero stage 1
* tp/pp = 1


Evaluation
----------



License
-------


MIT



TAGS
#transformers #pytorch #safetensors #gpt2 #text-generation #de #license-mit #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 

passage: TAGS
#transformers #pytorch #safetensors #gpt2 #text-generation #de #license-mit #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 
