--- license: apache-2.0 library_name: peft tags: - trl - sft - generated_from_trainer datasets: - generator base_model: mistralai/Mistral-7B-v0.1 model-index: - name: mistral7b_instruct_generation results: [] --- # mistral7b_instruct_generation This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the generator dataset. It achieves the following results on the evaluation set: - Loss: 1.8004 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 4 - eval_batch_size: 8 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: constant - lr_scheduler_warmup_steps: 0.03 - training_steps: 500 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | 1.857 | 0.0 | 20 | 1.8252 | | 1.9363 | 0.01 | 40 | 1.7948 | | 1.7643 | 0.01 | 60 | 1.7910 | | 1.861 | 0.01 | 80 | 1.7915 | | 1.8702 | 0.01 | 100 | 1.7903 | | 1.8619 | 0.02 | 120 | 1.7905 | | 1.7669 | 0.02 | 140 | 1.7956 | | 1.8062 | 0.02 | 160 | 1.7895 | | 1.7802 | 0.03 | 180 | 1.7958 | | 1.7773 | 0.03 | 200 | 1.7855 | | 1.8692 | 0.03 | 220 | 1.7936 | | 1.7815 | 0.03 | 240 | 1.7939 | | 1.8642 | 0.04 | 260 | 1.7990 | | 1.8715 | 0.04 | 280 | 1.7953 | | 1.8999 | 0.04 | 300 | 1.7999 | | 1.7691 | 0.04 | 320 | 1.7919 | | 1.7743 | 0.05 | 340 | 1.7973 | | 1.7692 | 0.05 | 360 | 1.7919 | | 1.954 | 0.05 | 380 | 1.7934 | | 1.8872 | 0.06 | 400 | 1.7966 | | 1.8925 | 0.06 | 420 | 1.7942 | | 1.8384 | 0.06 | 440 | 1.7949 | | 1.825 | 0.06 | 460 | 1.7982 | | 1.8151 | 0.07 | 480 | 1.7959 | | 1.8599 | 0.07 | 500 | 1.8004 | ### Framework versions - PEFT 0.7.1 - Transformers 4.37.0 - Pytorch 2.1.0+cu121 - Datasets 2.16.1 - Tokenizers 0.15.0