
# Easy German GPT2 Model

A language model for german easy language ("leichte Sprache") based on [German GPT-2 model](https://huggingface.co/dbmdz/german-gpt2)

## Model Details

Initialized using the weights of [German GPT-2 model](https://huggingface.co/dbmdz/german-gpt2).  
Then fine-tuned for one epoch on "leichte Sprache" corpora consisting of:
 - encyclopedia like data
 - news like data
 
 Hyperparamters used for fine-tuning:
 - tokenizer:
   - max_length: 1024 (but trained with dynamic length, using the collator functions 'pad_to_multiple_of=8')
   - stride: 64
   - return_overflowing_tokens=True
   
 - training arguments
   - num_train_epochs=1
   - learning_rate=1e-3
   - weight_decay=0.01
   - per_device_train_batch_size=4 
   - gradient_accumulation_steps=4
   - warmup_steps=200
   - fp16=True
   
 → 25112 training items, trained on google colab GPU (30 min)
   
 
 ## Evaluation results
 
The perplexity value is calculated based on an unseen dataset containing manually aligned standard german and "leichte Sprache" texts.  
For calculation, the method described in [this tutorial](https://huggingface.co/docs/transformers/perplexity) was used with the following values:  
- max_length = 512
- stride = 256

For comparison: running the modified function on [this example](https://huggingface.co/docs/transformers/perplexity#example-calculating-perplexity-with-gpt2-in-transformers) gives us a perplexity score of **18.2551**

| Model | Perplexity "leichte Sprache" [(Easy MDR News)](https://www.mdr.de/nachrichten-leicht/index.html)) |Perplexity standard german ([Standard MDR News](https://www.mdr.de/nachrichten/index.html))|
|:-----:|:----------------------------------------:|:------------------------------------:|
|[German GPT-2 model](https://huggingface.co/dbmdz/german-gpt2)| 23.8257   | **24.0301** |
|our model| **17.3053**   | 48.6314 |
{"language": ["de"], "widget": [{"text": "Der Sinn des Lebens ist", "example_title": "Sinn des Lebens"}], "inference": {"parameters": {"temperature": 0.7, "repetition_penalty": 1.4}}}
transformers pytorch gpt2 text-generation de autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #gpt2 #text-generation #de #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

Easy German GPT2 Model
======================


A language model for german easy language ("leichte Sprache") based on German GPT-2 model


Model Details
-------------


Initialized using the weights of German GPT-2 model.  

Then fine-tuned for one epoch on "leichte Sprache" corpora consisting of:


* encyclopedia like data
* news like data


Hyperparamters used for fine-tuning:


* tokenizer:


	+ max\_length: 1024 (but trained with dynamic length, using the collator functions 'pad\_to\_multiple\_of=8')
	+ stride: 64
	+ return\_overflowing\_tokens=True
* training arguments


	+ num\_train\_epochs=1
	+ learning\_rate=1e-3
	+ weight\_decay=0.01
	+ per\_device\_train\_batch\_size=4
	+ gradient\_accumulation\_steps=4
	+ warmup\_steps=200
	+ fp16=True


→ 25112 training items, trained on google colab GPU (30 min)


Evaluation results
------------------


The perplexity value is calculated based on an unseen dataset containing manually aligned standard german and "leichte Sprache" texts.  

For calculation, the method described in this tutorial was used with the following values:


* max\_length = 512
* stride = 256


For comparison: running the modified function on this example gives us a perplexity score of 18.2551




TAGS
#transformers #pytorch #gpt2 #text-generation #de #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

passage: TAGS
#transformers #pytorch #gpt2 #text-generation #de #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
