09562065-19e2-4296-812c-7420d36612d2

This model is a fine-tuned version of fxmarty/tiny-random-GemmaForCausalLM on the None dataset. It achieves the following results on the evaluation set:

• Loss: 12.3853

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.000218
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 500

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0010	1	12.4603
12.4566	0.0498	50	12.4551
12.4036	0.0996	100	12.3979
12.3863	0.1493	150	12.3890
12.3915	0.1991	200	12.3869
12.3782	0.2489	250	12.3859
12.383	0.2987	300	12.3853
12.3854	0.3484	350	12.3855
12.3848	0.3982	400	12.3853
12.3851	0.4480	450	12.3852
12.3824	0.4978	500	12.3853

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1