Update README.md

README.md

@@ -12,6 +12,18 @@ tags:
 - sft
 ---
 
+# Description
+
+Qwen2.5-Coder-7B-Instruct trained on a merged dataset of Unity3d q&a from these two datasets:
+[ibranze/codellama_unity3d_v2](https://huggingface.co/datasets/ibranze/codellama_unity3d_v2) (Full)
+[Hypersniper/unity_api_2022_3](https://huggingface.co/datasets/Hypersniper/unity_api_2022_3) (5%)
+
+15062 rows in total with a 10% validation split
+
+Consider this a preview as I develop a dataset of my own that I'm pleased with.
+
+
+
 # Uploaded  model
 
 - **Developed by:** neph1
@@ -21,3 +33,50 @@ tags:
 This qwen2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
 
 [<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
+
+# Training details
+
+About 1 epoch.
+
+Rank: 128
+
+Alpha: 256
+
+TrainingArguments(
+        per_device_train_batch_size =2,
+        gradient_accumulation_steps = 64,
+        #max_steps=10,
+        num_train_epochs=3,
+        warmup_steps = 5,
+        learning_rate = 1e-4,
+        fp16 = not torch.cuda.is_bf16_supported(),
+        bf16 = torch.cuda.is_bf16_supported(),
+        logging_steps = 10,
+        optim = "adamw_8bit",
+        weight_decay = 0.01,
+        lr_scheduler_type = "linear",
+        seed = 3407,
+        per_device_eval_batch_size = 2,
+        eval_strategy="steps",
+        eval_accumulation_steps = 64,
+        eval_steps = 10,
+        eval_delay = 0,
+        save_strategy="steps",
+        save_steps=25,
+        report_to="none",
+    ),
+
+
+Step 	Training Loss 	Validation Loss
+10 	2.097300 	1.165832
+20 	1.058100 	1.013441
+30 	0.898500 	0.969640
+40 	0.866600 	0.943687
+50 	0.847300 	0.926879
+60 	0.838200 	0.903914
+70 	0.797600 	0.888580
+80 	0.777700 	0.873389
+90 	0.793900 	0.859501
+100 	0.725500 	0.846339
+110 	0.739400 	0.843786
+120 	0.675200 	0.833775