base_model: unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit
language:
- en
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- qwen2
- trl
- sft
Description
Qwen2.5-Coder-7B-Instruct trained on a merged dataset of Unity3d q&a from these two datasets: ibranze/codellama_unity3d_v2 (Full) Hypersniper/unity_api_2022_3 (5%)
15062 rows in total with a 10% validation split
Consider this a preview as I develop a dataset of my own that I'm pleased with.
Uploaded model
- Developed by: neph1
- License: apache-2.0
- Finetuned from model : unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit
This qwen2 model was trained 2x faster with Unsloth and Huggingface's TRL library.
Training details
About 1 epoch.
Rank: 128
Alpha: 256
TrainingArguments( per_device_train_batch_size =2, gradient_accumulation_steps = 64, #max_steps=10, num_train_epochs=3, warmup_steps = 5, learning_rate = 1e-4, fp16 = not torch.cuda.is_bf16_supported(), bf16 = torch.cuda.is_bf16_supported(), logging_steps = 10, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, per_device_eval_batch_size = 2, eval_strategy="steps", eval_accumulation_steps = 64, eval_steps = 10, eval_delay = 0, save_strategy="steps", save_steps=25, report_to="none", ),
Step Training Loss Validation Loss 10 2.097300 1.165832 20 1.058100 1.013441 30 0.898500 0.969640 40 0.866600 0.943687 50 0.847300 0.926879 60 0.838200 0.903914 70 0.797600 0.888580 80 0.777700 0.873389 90 0.793900 0.859501 100 0.725500 0.846339 110 0.739400 0.843786 120 0.675200 0.833775