See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: katuni4ka/tiny-random-qwen1.5-moe
bf16: true
chat_template: llama3
dataset_prepared_path: null
datasets:
- data_files:
  - 4af3f7c1fa5610ea_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/4af3f7c1fa5610ea_train_data.json
  type:
    field_input: input
    field_instruction: instruction
    field_output: output
    format: '{instruction} {input}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
debug: null
device_map:
  ? ''
  : 0,1,2,3,4,5,6,7
early_stopping_patience: 2
eval_max_new_tokens: 128
eval_steps: 100
eval_table_size: null
flash_attention: true
gradient_accumulation_steps: 8
gradient_checkpointing: true
group_by_length: false
hub_model_id: Alphatao/d233361e-a60b-4716-9cd1-c06404238b41
hub_repo: null
hub_strategy: null
hub_token: null
learning_rate: 0.0002
load_best_model_at_end: true
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 1
lora_alpha: 128
lora_dropout: 0.1
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 64
lora_target_linear: true
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: 7680
micro_batch_size: 4
mlflow_experiment_name: /tmp/4af3f7c1fa5610ea_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 2
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 100
sequence_len: 1024
strict: false
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.045018682753342636
wandb_entity: null
wandb_mode: online
wandb_name: 1798553a-3812-4302-9a96-9f20992e85cb
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: 1798553a-3812-4302-9a96-9f20992e85cb
warmup_steps: 10
weight_decay: 0.0
xformers_attention: null

d233361e-a60b-4716-9cd1-c06404238b41

This model is a fine-tuned version of katuni4ka/tiny-random-qwen1.5-moe on the None dataset. It achieves the following results on the evaluation set:

• Loss: 11.7379

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• training_steps: 6630

Training results

Training Loss	Epoch	Step	Validation Loss
11.9386	0.0003	1	11.9363
11.8365	0.0302	100	11.8407
11.8103	0.0603	200	11.8085
11.797	0.0905	300	11.7974
11.7911	0.1207	400	11.7903
11.7862	0.1508	500	11.7848
11.7865	0.1810	600	11.7808
11.7987	0.2112	700	11.7781
11.784	0.2414	800	11.7752
11.7833	0.2715	900	11.7730
11.7675	0.3017	1000	11.7709
11.7851	0.3319	1100	11.7690
11.7597	0.3620	1200	11.7674
11.7693	0.3922	1300	11.7657
11.7708	0.4224	1400	11.7640
11.7505	0.4525	1500	11.7627
11.7645	0.4827	1600	11.7610
11.7602	0.5129	1700	11.7595
11.7658	0.5430	1800	11.7580
11.7688	0.5732	1900	11.7562
11.7588	0.6034	2000	11.7547
11.7459	0.6336	2100	11.7533
11.774	0.6637	2200	11.7521
11.7575	0.6939	2300	11.7512
11.7576	0.7241	2400	11.7503
11.7534	0.7542	2500	11.7491
11.7671	0.7844	2600	11.7483
11.7611	0.8146	2700	11.7476
11.7464	0.8447	2800	11.7469
11.759	0.8749	2900	11.7463
11.771	0.9051	3000	11.7452
11.7354	0.9352	3100	11.7448
11.7581	0.9654	3200	11.7442
11.7636	0.9956	3300	11.7439
11.8293	1.0259	3400	11.7432
12.0921	1.0561	3500	11.7428
11.8367	1.0862	3600	11.7425
11.4115	1.1164	3700	11.7421
11.6317	1.1466	3800	11.7416
11.3068	1.1768	3900	11.7411
11.5171	1.2069	4000	11.7410
11.88	1.2371	4100	11.7407
11.7468	1.2673	4200	11.7403
12.3395	1.2974	4300	11.7402
12.6814	1.3276	4400	11.7399
11.6404	1.3578	4500	11.7396
11.8672	1.3879	4600	11.7394
11.3736	1.4181	4700	11.7392
11.9053	1.4483	4800	11.7390
11.2669	1.4784	4900	11.7389
11.574	1.5086	5000	11.7388
11.2534	1.5388	5100	11.7386
11.4932	1.5690	5200	11.7385
11.7849	1.5991	5300	11.7384
11.632	1.6293	5400	11.7383
11.4767	1.6595	5500	11.7382
11.985	1.6896	5600	11.7381
11.9087	1.7198	5700	11.7381
12.0167	1.7500	5800	11.7380
11.9729	1.7801	5900	11.7379
11.4693	1.8103	6000	11.7379
11.4057	1.8405	6100	11.7379
12.4109	1.8706	6200	11.7379
12.0206	1.9008	6300	11.7379
11.5193	1.9310	6400	11.7379
11.6339	1.9612	6500	11.7379
11.7818	1.9913	6600	11.7379

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1