metadata
license: apache-2.0
base_model: mhenrichsen/danskgpt-tiny
tags:
- generated_from_trainer
DanskGPT-chat-tiny
DanskGPT-chat-tiny er chat-varianten af mhenrichsen/danskgpt-tiny trænet på private chat datasæt.
Model beskrivelse
Modellen er beregnet til at være en lightweight version af DanskGPT, der kan køre på næsten alle enheder.
Prompt template
Modellen er trænet med ChatML format (samme som OpenAI's modeller), og skal bruges på følgende måde:
<|im_start|>system\nDu er en hjælpsom assistent.<|im_end|>\n<|im_start|>user\nHvad er skak?<|im_end|>\n<|im_start|>assistant
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 4
- total_train_batch_size: 256
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 3
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.3599 | 0.0 | 1 | 1.4118 |
| 0.7896 | 0.25 | 136 | 0.7813 |
| 0.7339 | 0.5 | 272 | 0.7490 |
| 0.7378 | 0.75 | 408 | 0.7285 |
| 0.7112 | 1.0 | 544 | 0.7146 |
| 0.6377 | 1.23 | 680 | 0.7135 |
| 0.6192 | 1.49 | 816 | 0.7133 |
| 0.5985 | 1.74 | 952 | 0.7073 |
| 0.6067 | 1.99 | 1088 | 0.7026 |
| 0.5139 | 2.22 | 1224 | 0.7167 |
| 0.5099 | 2.47 | 1360 | 0.7193 |
| 0.5217 | 2.72 | 1496 | 0.7168 |
Framework versions
- Transformers 4.37.0.dev0
- Pytorch 2.0.1+cu118
- Datasets 2.16.1
- Tokenizers 0.15.0