pytorch gpt2 text-generation de transformers license:mit
---
license: mit
language: de
widget:
- text: "In einer schockierenden Entdeckung fanden Wissenschaftler eine Herde Einhörner, die in einem abgelegenen, zuvor unerforschten Tal in den Anden lebten."
---

# Replication of [gpt2-wechsel-german](https://huggingface.co/benjamin/gpt2-wechsel-german)

- trained with [BigScience's DeepSpeed-Megatron-LM code base](https://github.com/bigscience-workshop/Megatron-DeepSpeed)
- 22hrs on 4xA100 GPUs (~ 80 TFLOPs / GPU)
- stopped after 100k steps
- less than a single epoch on `oscar_unshuffled_deduplicated_de` (excluding validation set; original model was trained for 75 epochs on less data)
- bf16
- zero stage 1
- tp/pp = 1

## Evaluation

| Model | PPL |
|---|---|
| `gpt2-wechsel-german-ds-meg` | **26.4** |
| `gpt2-wechsel-german` | 26.8 |
| `gpt2` (retrained from scratch)  | 27.63 |

## License

MIT


