session_logs/lora_finetuning.log

2025-02-20 03:59:01,605 - Logging initialized for session: 339ce426-59b0-464b-adc2-a2c9508d5a37
2025-02-20 03:59:03,499 - We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
2025-02-21 06:55:58,104 - Using default tokenizer.
2025-02-21 06:55:59,683 - Hyperparameters: {'output_dir': './lora_finetuned', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': True, 'do_predict': False, 'eval_strategy': <IntervalStrategy.STEPS: 'steps'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 2, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 1, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 0.0002, 'weight_decay': 0.0, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 0.3, 'num_train_epochs': 1, 'max_steps': 17000, 'lr_scheduler_type': <SchedulerType.LINEAR: 'linear'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.0, 'warmup_steps': 1000, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': './logs', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 50, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 100, 'save_total_limit': None, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': True, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': 10, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': './lora_finetuned', 'disable_tqdm': False, 'remove_unused_columns': False, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': AcceleratorConfig(split_batches=False, dispatch_batches=None, even_batches=True, use_seedable_sampler=True, non_blocking=False, gradient_accumulation_kwargs=None, use_configured_state=False), 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.PAGED_ADAMW_8BIT: 'paged_adamw_8bit'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['tensorboard'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': True, 'resume_from_checkpoint': None, 'hub_model_id': 'Udith-Sandaruwan/format-instruction-test', 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'evaluation_strategy': 'steps', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'dispatch_batches': None, 'split_batches': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': False, 'eval_use_gather_object': False, 'average_tokens_across_devices': False, 'distributed_state': Distributed environment: NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda
, '_n_gpu': 1, '__cached__setup_devices': device(type='cuda', index=0), 'deepspeed_plugin': None}
2025-02-21 06:55:59,683 - Training details: {'Epochs': 1, 'Training Steps': 17000, 'Final Loss': 1.6536, 'Final Learning Rate': 0.0, 'Total Training Time (s)': '96182.97'}
2025-02-21 06:55:59,684 - Training metrics: {'epochs': [50, 100, 150, 200, 250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1000, 1050, 1100, 1150, 1200, 1250, 1300, 1350, 1400, 1450, 1500, 1550, 1600, 1650, 1700, 1750, 1800, 1850, 1900, 1950, 2000, 2050, 2100, 2150, 2200, 2250, 2300, 2350, 2400, 2450, 2500, 2550, 2600, 2650, 2700, 2750, 2800, 2850, 2900, 2950, 3000, 3050, 3100, 3150, 3200, 3250, 3300, 3350, 3400, 3450, 3500, 3550, 3600, 3650, 3700, 3750, 3800, 3850, 3900, 3950, 4000, 4050, 4100, 4150, 4200, 4250, 4300, 4350, 4400, 4450, 4500, 4550, 4600, 4650, 4700, 4750, 4800, 4850, 4900, 4950, 5000, 5050, 5100, 5150, 5200, 5250, 5300, 5350, 5400, 5450, 5500, 5550, 5600, 5650, 5700, 5750, 5800, 5850, 5900, 5950, 6000, 6050, 6100, 6150, 6200, 6250, 6300, 6350, 6400, 6450, 6500, 6550, 6600, 6650, 6700, 6750, 6800, 6850, 6900, 6950, 7000, 7050, 7100, 7150, 7200, 7250, 7300, 7350, 7400, 7450, 7500, 7550, 7600, 7650, 7700, 7750, 7800, 7850, 7900, 7950, 8000, 8050, 8100, 8150, 8200, 8250, 8300, 8350, 8400, 8450, 8500, 8550, 8600, 8650, 8700, 8750, 8800, 8850, 8900, 8950, 9000, 9050, 9100, 9150, 9200, 9250, 9300, 9350, 9400, 9450, 9500, 9550, 9600, 9650, 9700, 9750, 9800, 9850, 9900, 9950, 10000, 10050, 10100, 10150, 10200, 10250, 10300, 10350, 10400, 10450, 10500, 10550, 10600, 10650, 10700, 10750, 10800, 10850, 10900, 10950, 11000, 11050, 11100, 11150, 11200, 11250, 11300, 11350, 11400, 11450, 11500, 11550, 11600, 11650, 11700, 11750, 11800, 11850, 11900, 11950, 12000, 12050, 12100, 12150, 12200, 12250, 12300, 12350, 12400, 12450, 12500, 12550, 12600, 12650, 12700, 12750, 12800, 12850, 12900, 12950, 13000, 13050, 13100, 13150, 13200, 13250, 13300, 13350, 13400, 13450, 13500, 13550, 13600, 13650, 13700, 13750, 13800, 13850, 13900, 13950, 14000, 14050, 14100, 14150, 14200, 14250, 14300, 14350, 14400, 14450, 14500, 14550, 14600, 14650, 14700, 14750, 14800, 14850, 14900, 14950, 15000, 15050, 15100, 15150, 15200, 15250, 15300, 15350, 15400, 15450, 15500, 15550, 15600, 15650, 15700, 15750, 15800, 15850, 15900, 15950, 16000, 16050, 16100, 16150, 16200, 16250, 16300, 16350, 16400, 16450, 16500, 16550, 16600, 16650, 16700, 16750, 16800, 16850, 16900, 16950, 17000], 'loss': [3.5027, 2.9223, 2.6941, 2.501, 2.4575, 2.3535, 2.2747, 2.3175, 2.1638, 2.2116, 2.1548, 2.2079, 2.1333, 2.153, 2.1851, 2.0635, 2.2095, 2.1928, 2.0895, 2.022, 2.026, 2.1152, 2.0978, 2.0804, 2.1602, 2.142, 2.1053, 2.0437, 2.0566, 2.0834, 2.044, 2.0385, 2.065, 1.9956, 1.8957, 1.9039, 1.9536, 2.0164, 2.0423, 1.9837, 1.8703, 1.8998, 2.0845, 1.9579, 1.9671, 1.957, 1.943, 1.9238, 1.9465, 1.9917, 1.9835, 1.8988, 1.9071, 1.9399, 1.8589, 1.8431, 1.9175, 1.9773, 1.8704, 1.9167, 1.8417, 1.8395, 1.8771, 1.9028, 1.8359, 1.8086, 1.8983, 1.8832, 1.8622, 1.8627, 1.9234, 1.8987, 1.8446, 1.8577, 1.8855, 1.8136, 1.8079, 1.8287, 1.8664, 1.8276, 1.909, 1.8036, 1.7673, 1.7578, 1.8987, 1.7899, 1.8453, 1.841, 1.8332, 1.8509, 1.8669, 1.7947, 1.8745, 1.7744, 1.8155, 1.8678, 1.7985, 1.7651, 1.8392, 1.7977, 1.7574, 1.8493, 1.8076, 1.7608, 1.8567, 1.9245, 1.8291, 1.7926, 1.7467, 1.8155, 1.8188, 1.8379, 1.7732, 1.6402, 1.7885, 1.7941, 1.7155, 1.7969, 1.7421, 1.8238, 1.7222, 1.7063, 1.7919, 1.7815, 1.8001, 1.7636, 1.7194, 1.718, 1.729, 1.6959, 1.7704, 1.8851, 1.7513, 1.8183, 1.7612, 1.6372, 1.6722, 1.7216, 1.753, 1.7265, 1.8189, 1.8496, 1.7138, 1.7056, 1.6834, 1.695, 1.82, 1.6692, 1.725, 1.7623, 1.6733, 1.6575, 1.8754, 1.711, 1.758, 1.7224, 1.7278, 1.7084, 1.718, 1.7623, 1.8046, 1.7466, 1.8245, 1.7275, 1.8115, 1.6826, 1.6141, 1.7309, 1.6694, 1.7154, 1.8163, 1.7156, 1.7386, 1.7391, 1.8131, 1.6538, 1.7069, 1.6552, 1.6659, 1.7449, 1.7295, 1.7446, 1.6683, 1.7045, 1.675, 1.7246, 1.7177, 1.8505, 1.6699, 1.6654, 1.6845, 1.6447, 1.6267, 1.6994, 1.7051, 1.6266, 1.7863, 1.6978, 1.6633, 1.6664, 1.5766, 1.663, 1.7196, 1.5681, 1.7442, 1.6349, 1.6257, 1.6546, 1.6717, 1.6967, 1.6965, 1.6777, 1.6917, 1.6019, 1.6552, 1.6806, 1.6052, 1.6932, 1.6626, 1.619, 1.6067, 1.6603, 1.6406, 1.6686, 1.5976, 1.7215, 1.6112, 1.6271, 1.6747, 1.6639, 1.6235, 1.5832, 1.6538, 1.6869, 1.6201, 1.5443, 1.6072, 1.6968, 1.5753, 1.7444, 1.7185, 1.6375, 1.6466, 1.6798, 1.6456, 1.6617, 1.6785, 1.6718, 1.6937, 1.6537, 1.7704, 1.5563, 1.6567, 1.6569, 1.6848, 1.6793, 1.5993, 1.622, 1.6092, 1.5952, 1.6136, 1.6751, 1.5653, 1.6877, 1.7, 1.608, 1.6246, 1.6263, 1.5799, 1.6379, 1.6848, 1.6419, 1.6229, 1.6338, 1.7418, 1.6401, 1.5894, 1.5949, 1.634, 1.658, 1.6169, 1.6038, 1.5563, 1.6405, 1.6129, 1.5372, 1.5494, 1.7385, 1.6757, 1.5992, 1.6599, 1.6322, 1.6999, 1.6406, 1.6684, 1.5549, 1.6345, 1.616, 1.5564, 1.6007, 1.5802, 1.6755, 1.67, 1.5693, 1.6714, 1.6232, 1.5959, 1.5276, 1.5953, 1.6329, 1.6137, 1.5949, 1.5779, 1.5616, 1.6064, 1.6264, 1.766, 1.7036, 1.6683, 1.5984, 1.584, 1.5704, 1.6446, 1.6091, 1.6722, 1.5689, 1.5787, 1.6236, 1.5789, 1.6468, 1.5907, 1.6147, 1.5872, 1.657, 1.6609, 1.6016, 1.6178, 1.6246, 1.6039, 1.6536], 'learning_rate': [1e-05, 2e-05, 3e-05, 4e-05, 5e-05, 6e-05, 7e-05, 8e-05, 9e-05, 0.0001, 0.00011000000000000002, 0.00012, 0.00013000000000000002, 0.00014, 0.00015000000000000001, 0.00016, 0.00017, 0.00018, 0.00019, 0.0002, 0.000199375, 0.00019875, 0.000198125, 0.00019750000000000003, 0.000196875, 0.00019625, 0.00019562500000000003, 0.000195, 0.00019437500000000002, 0.00019375000000000002, 0.000193125, 0.00019250000000000002, 0.00019187500000000002, 0.00019125000000000001, 0.000190625, 0.00019, 0.000189375, 0.00018875, 0.000188125, 0.0001875, 0.000186875, 0.00018625, 0.000185625, 0.00018500000000000002, 0.000184375, 0.00018375, 0.00018312500000000002, 0.0001825, 0.00018187500000000002, 0.00018125000000000001, 0.000180625, 0.00018, 0.000179375, 0.00017875, 0.000178125, 0.0001775, 0.000176875, 0.00017625, 0.00017562500000000003, 0.000175, 0.000174375, 0.00017375000000000002, 0.000173125, 0.00017250000000000002, 0.00017187500000000002, 0.00017125, 0.00017062500000000001, 0.00017, 0.000169375, 0.00016875, 0.000168125, 0.0001675, 0.000166875, 0.00016625000000000003, 0.000165625, 0.000165, 0.00016437500000000002, 0.00016375, 0.00016312500000000002, 0.00016250000000000002, 0.000161875, 0.00016125000000000002, 0.00016062500000000001, 0.00016, 0.000159375, 0.00015875, 0.000158125, 0.0001575, 0.000156875, 0.00015625, 0.000155625, 0.000155, 0.000154375, 0.00015375000000000002, 0.000153125, 0.0001525, 0.00015187500000000002, 0.00015125, 0.00015062500000000002, 0.00015000000000000001, 0.00014937499999999999, 0.00014875, 0.000148125, 0.0001475, 0.000146875, 0.00014625, 0.000145625, 0.000145, 0.00014437500000000003, 0.00014375, 0.000143125, 0.00014250000000000002, 0.000141875, 0.00014125000000000002, 0.00014062500000000002, 0.00014, 0.000139375, 0.00013875, 0.000138125, 0.0001375, 0.000136875, 0.00013625, 0.000135625, 0.00013500000000000003, 0.000134375, 0.00013375, 0.00013312500000000002, 0.0001325, 0.00013187500000000002, 0.00013125000000000002, 0.000130625, 0.00013000000000000002, 0.00012937500000000001, 0.00012875, 0.000128125, 0.0001275, 0.000126875, 0.00012625, 0.000125625, 0.000125, 0.000124375, 0.00012375, 0.000123125, 0.00012250000000000002, 0.00012187500000000001, 0.00012124999999999999, 0.000120625, 0.00012, 0.00011937500000000001, 0.00011875, 0.000118125, 0.00011750000000000001, 0.000116875, 0.00011625000000000002, 0.000115625, 0.00011499999999999999, 0.00011437500000000002, 0.00011375, 0.00011312500000000001, 0.00011250000000000001, 0.000111875, 0.00011125000000000001, 0.000110625, 0.00011000000000000002, 0.000109375, 0.00010875, 0.00010812500000000001, 0.0001075, 0.00010687500000000001, 0.00010625000000000001, 0.00010562499999999999, 0.000105, 0.000104375, 0.00010375000000000001, 0.000103125, 0.0001025, 0.00010187500000000001, 0.00010125, 0.00010062500000000002, 0.0001, 9.9375e-05, 9.875000000000002e-05, 9.8125e-05, 9.75e-05, 9.687500000000001e-05, 9.625000000000001e-05, 9.562500000000001e-05, 9.5e-05, 9.4375e-05, 9.375e-05, 9.3125e-05, 9.250000000000001e-05, 9.1875e-05, 9.125e-05, 9.062500000000001e-05, 9e-05, 8.9375e-05, 8.875e-05, 8.8125e-05, 8.75e-05, 8.687500000000001e-05, 8.625000000000001e-05, 8.5625e-05, 8.5e-05, 8.4375e-05, 8.375e-05, 8.312500000000001e-05, 8.25e-05, 8.1875e-05, 8.125000000000001e-05, 8.062500000000001e-05, 8e-05, 7.9375e-05, 7.875e-05, 7.8125e-05, 7.75e-05, 7.687500000000001e-05, 7.625e-05, 7.5625e-05, 7.500000000000001e-05, 7.4375e-05, 7.375e-05, 7.3125e-05, 7.25e-05, 7.1875e-05, 7.125000000000001e-05, 7.062500000000001e-05, 7e-05, 6.9375e-05, 6.875e-05, 6.8125e-05, 6.750000000000001e-05, 6.6875e-05, 6.625e-05, 6.562500000000001e-05, 6.500000000000001e-05, 6.4375e-05, 6.375e-05, 6.3125e-05, 6.25e-05, 6.1875e-05, 6.125000000000001e-05, 6.0624999999999996e-05, 6e-05, 5.9375e-05, 5.8750000000000005e-05, 5.812500000000001e-05, 5.7499999999999995e-05, 5.6875e-05, 5.6250000000000005e-05, 5.5625000000000004e-05, 5.500000000000001e-05, 5.4375e-05, 5.375e-05, 5.3125000000000004e-05, 5.25e-05, 5.187500000000001e-05, 5.125e-05, 5.0625e-05, 5e-05, 4.937500000000001e-05, 4.875e-05, 4.8125000000000004e-05, 4.75e-05, 4.6875e-05, 4.6250000000000006e-05, 4.5625e-05, 4.5e-05, 4.4375e-05, 4.375e-05, 4.3125000000000005e-05, 4.25e-05, 4.1875e-05, 4.125e-05, 4.0625000000000005e-05, 4e-05, 3.9375e-05, 3.875e-05, 3.8125e-05, 3.7500000000000003e-05, 3.6875e-05, 3.625e-05, 3.5625000000000005e-05, 3.5e-05, 3.4375e-05, 3.375000000000001e-05, 3.3125e-05, 3.2500000000000004e-05, 3.1875e-05, 3.125e-05, 3.0625000000000006e-05, 3e-05, 2.9375000000000003e-05, 2.8749999999999997e-05, 2.8125000000000003e-05, 2.7500000000000004e-05, 2.6875e-05, 2.625e-05, 2.5625e-05, 2.5e-05, 2.4375e-05, 2.375e-05, 2.3125000000000003e-05, 2.25e-05, 2.1875e-05, 2.125e-05, 2.0625e-05, 2e-05, 1.9375e-05, 1.8750000000000002e-05, 1.8125e-05, 1.75e-05, 1.6875000000000004e-05, 1.6250000000000002e-05, 1.5625e-05, 1.5e-05, 1.4374999999999999e-05, 1.3750000000000002e-05, 1.3125e-05, 1.25e-05, 1.1875e-05, 1.125e-05, 1.0625e-05, 1e-05, 9.375000000000001e-06, 8.75e-06, 8.125000000000001e-06, 7.5e-06, 6.875000000000001e-06, 6.25e-06, 5.625e-06, 5e-06, 4.375e-06, 3.75e-06, 3.125e-06, 2.5e-06, 1.875e-06, 1.25e-06, 6.25e-07, 0.0], 'training_time': 96182.9735352993}
2025-02-21 06:55:59,684 - Evaluation results: {'meteor_scores': {'meteor': 0.07062146892655369}, 'rouge_scores': {'rouge1': 0.0, 'rouge2': 0.0, 'rougeL': 0.0, 'rougeLsum': 0.0}, 'bleu_scores': {'bleu': 0.0, 'precisions': [0.06382978723404255, 0.0, 0.0, 0.0], 'brevity_penalty': 1.0, 'length_ratio': 3.2413793103448274, 'translation_length': 94, 'reference_length': 29}, 'perplexity': 565417100.0}