
# German GPT2-XL (1.5B)

- trained with [BigScience's DeepSpeed-Megatron-LM code base](https://github.com/bigscience-workshop/Megatron-DeepSpeed)
- word embedding initialized with [WECHSEL](https://arxiv.org/abs/2112.06598) and all other weights taken from English [gpt2-xl](https://huggingface.co/gpt2-xl)
- ~ 3 days on 16xA100 GPUs (~ 80 TFLOPs / GPU)
- stopped after 100k steps
- 26.2B tokens
- less than a single epoch on `oscar_unshuffled_deduplicated_de` (excluding validation set; original model was trained for 75 epochs on less data)
- bf16
- zero stage 0
- tp/pp = 1


### How to use

You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
set a seed for reproducibility:

```python
>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='malteos/gpt2-xl-wechsel-german')
>>> set_seed(42)
>>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)

[{'generated_text': "Hello, I'm a language model, a language for thinking, a language for expressing thoughts."},
 {'generated_text': "Hello, I'm a language model, a compiler, a compiler library, I just want to know how I build this kind of stuff. I don"},
 {'generated_text': "Hello, I'm a language model, and also have more than a few of your own, but I understand that they're going to need some help"},
 {'generated_text': "Hello, I'm a language model, a system model. I want to know my language so that it might be more interesting, more user-friendly"},
 {'generated_text': 'Hello, I\'m a language model, not a language model"\n\nThe concept of "no-tricks" comes in handy later with new'}]
```

Here is how to use this model to get the features of a given text in PyTorch:

```python
from transformers import GPT2Tokenizer, GPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('malteos/gpt2-xl-wechsel-german')
model = GPT2Model.from_pretrained('malteos/gpt2-xl-wechsel-german')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

## Evaluation

| Model (size) | PPL |
|---|---|
| `gpt2-xl-wechsel-german` (1.5B) | **14.5** |
| `gpt2-wechsel-german-ds-meg` (117M) | 26.4 |
| `gpt2-wechsel-german`  (117M) | 26.8 |
| `gpt2` (retrained from scratch) (117M)  | 27.63 |


## Other German language models

- https://huggingface.co/malteos/bloom-1b5-clp-german
- https://huggingface.co/malteos/bloom-6b4-clp-german
- https://huggingface.co/malteos/bloom-6b4-clp-german-oasst-v0.1

## License

MIT



{"language": "de", "license": "mit", "pipeline_tag": "text-generation", "widget": [{"text": "In einer schockierenden Entdeckung fanden Wissenschaftler eine Herde Einh\u00f6rner, die in ", "example_title": "Einh\u00f6rner ..."}, {"text": "Definiere folgende W\u00f6rter\n\nWort: Einhorn\nDefinition: Das Einhorn ist ein Fabelwesen von Pferde- oder Ziegengestalt mit einem geraden Horn auf der Stirnmitte. \n\nWort: Regierungschef\nDefinition: Der Regierungschef ist der Leiter der Regierung eines Staates (z. B. National- oder Gliedstaat).\n\nWort: Waffendrill\nDefinition:", "example_title": "Definiere ..."}]}
transformers pytorch gpt2 text-generation de arxiv:2112.06598 license:mit autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #gpt2 #text-generation #de #arxiv-2112.06598 #license-mit #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

German GPT2-XL (1.5B)
=====================


* trained with BigScience's DeepSpeed-Megatron-LM code base
* word embedding initialized with WECHSEL and all other weights taken from English gpt2-xl
* ~ 3 days on 16xA100 GPUs (~ 80 TFLOPs / GPU)
* stopped after 100k steps
* 26.2B tokens
* less than a single epoch on 'oscar\_unshuffled\_deduplicated\_de' (excluding validation set; original model was trained for 75 epochs on less data)
* bf16
* zero stage 0
* tp/pp = 1


### How to use


You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
set a seed for reproducibility:


Here is how to use this model to get the features of a given text in PyTorch:


Evaluation
----------



Other German language models
----------------------------


* URL
* URL
* URL


License
-------


MIT



### How to use


You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
set a seed for reproducibility:


Here is how to use this model to get the features of a given text in PyTorch:


Evaluation
----------



Other German language models
----------------------------


* URL
* URL
* URL


License
-------


MIT
TAGS
#transformers #pytorch #gpt2 #text-generation #de #arxiv-2112.06598 #license-mit #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
 ### How to use


You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
set a seed for reproducibility:


Here is how to use this model to get the features of a given text in PyTorch:


Evaluation
----------



Other German language models
----------------------------


* URL
* URL
* URL


License
-------


MIT
passage: TAGS
#transformers #pytorch #gpt2 #text-generation #de #arxiv-2112.06598 #license-mit #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
### How to use


You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
set a seed for reproducibility:


Here is how to use this model to get the features of a given text in PyTorch:


Evaluation
----------



Other German language models
----------------------------


* URL
* URL
* URL


License
-------


MIT