hs-094/Llama-3.2-3B-Instruct-medical-dataset

llama-3.2-3b-medical-dataset-fine-tuned

This model is a fine-tuned version of meta-llama/Llama-3.2-3B-Instruct on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 1.8519

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 2
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
2.2124	0.0465	100	2.2831
2.0834	0.0930	200	2.1037
2.0729	0.1394	300	2.0504
1.881	0.1859	400	2.0172
1.9543	0.2324	500	1.9913
1.9713	0.2789	600	1.9725
1.9492	0.3254	700	1.9590
1.9655	0.3718	800	1.9478
2.0255	0.4183	900	1.9369
1.9839	0.4648	1000	1.9279
1.8153	0.5113	1100	1.9212
2.069	0.5578	1200	1.9156
1.8085	0.6042	1300	1.9107
1.8947	0.6507	1400	1.9061
1.8708	0.6972	1500	1.9022
1.8659	0.7437	1600	1.8984
1.951	0.7901	1700	1.8951
1.9871	0.8366	1800	1.8917
1.8627	0.8831	1900	1.8892
1.8984	0.9296	2000	1.8865
1.9381	0.9761	2100	1.8838
1.8315	1.0225	2200	1.8819
1.9927	1.0690	2300	1.8797
1.7257	1.1155	2400	1.8783
1.9064	1.1620	2500	1.8762
1.8463	1.2085	2600	1.8744
1.864	1.2549	2700	1.8728
1.8915	1.3014	2800	1.8714
1.8045	1.3479	2900	1.8698
1.7097	1.3944	3000	1.8688
1.8884	1.4409	3100	1.8672
1.9608	1.4873	3200	1.8657
1.9233	1.5338	3300	1.8645
1.908	1.5803	3400	1.8637
1.8181	1.6268	3500	1.8624
1.7803	1.6733	3600	1.8614
1.8635	1.7197	3700	1.8603
1.763	1.7662	3800	1.8596
1.7074	1.8127	3900	1.8588
1.7098	1.8592	4000	1.8579
1.7753	1.9056	4100	1.8574
1.8458	1.9521	4200	1.8567
1.8413	1.9986	4300	1.8560
1.8139	2.0451	4400	1.8557
1.813	2.0916	4500	1.8554
1.8516	2.1380	4600	1.8550
1.7899	2.1845	4700	1.8545
1.8442	2.2310	4800	1.8541
1.9263	2.2775	4900	1.8538
1.8216	2.3240	5000	1.8534
1.6517	2.3704	5100	1.8531
1.7538	2.4169	5200	1.8530
1.7886	2.4634	5300	1.8526
1.7547	2.5099	5400	1.8525
1.8083	2.5564	5500	1.8524
1.7888	2.6028	5600	1.8524
1.6415	2.6493	5700	1.8522
1.6981	2.6958	5800	1.8521
1.8211	2.7423	5900	1.8520
1.7189	2.7888	6000	1.8519
1.7251	2.8352	6100	1.8519
1.8117	2.8817	6200	1.8518
1.8117	2.9282	6300	1.8519
1.9351	2.9747	6400	1.8519

Framework versions

• PEFT 0.13.2
• Transformers 4.46.3
• Pytorch 2.4.1+cu121
• Datasets 3.1.0
• Tokenizers 0.20.3