d8a5982d-8829-4a65-bdb7-96f9ef9515f2

This model is a fine-tuned version of fxmarty/tiny-random-GemmaForCausalLM on the None dataset. It achieves the following results on the evaluation set:

• Loss: 12.3859

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.000207
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 500

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0010	1	12.4603
12.4574	0.0498	50	12.4570
12.4057	0.0996	100	12.4009
12.3862	0.1493	150	12.3896
12.3922	0.1991	200	12.3878
12.383	0.2489	250	12.3871
12.3829	0.2987	300	12.3861
12.3847	0.3484	350	12.3861
12.3866	0.3982	400	12.3858
12.3872	0.4480	450	12.3857
12.3848	0.4978	500	12.3859

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1