halcyon-llm/SmolLM2-360M-japanese_patch-11393

results

This model is a fine-tuned version of HuggingFaceTB/SmolLM2-360M on the kajuma/training_01-09_patch dataset. It achieves the following results on the evaluation set:

• Loss: 0.1002

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 32
• total_train_batch_size: 256
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.95) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine_with_min_lr
• lr_scheduler_warmup_steps: 1000
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
3.584	0.0878	1000	0.1139
3.4937	0.1755	2000	0.1080
3.3486	0.2633	3000	0.1059
3.4019	0.3511	4000	0.1044
3.2915	0.4388	5000	0.1033
3.2729	0.5266	6000	0.1023
3.2451	0.6144	7000	0.1015
3.2339	0.7022	8000	0.1009
3.2383	0.7899	9000	0.1005
3.2441	0.8777	10000	0.1003
3.2744	0.9655	11000	0.1002

Framework versions

• Transformers 4.48.0.dev0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0