2025-01-08 17:40:36,980 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmp76e8z5da/test.c -o /tmp/tmp76e8z5da/test.o
2025-01-08 17:40:37,011 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmp76e8z5da/test.o -laio -o /tmp/tmp76e8z5da/a.out
2025-01-08 17:40:37,024 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmp0z9hrob6/test.c -o /tmp/tmp0z9hrob6/test.o
2025-01-08 17:40:37,042 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmp0z9hrob6/test.o -laio -o /tmp/tmp0z9hrob6/a.out
2025-01-08 17:40:37,093 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpivmpgbne/test.c -o /tmp/tmpivmpgbne/test.o
2025-01-08 17:40:37,096 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmp20zdp2x1/test.c -o /tmp/tmp20zdp2x1/test.o
2025-01-08 17:40:37,111 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpivmpgbne/test.o -laio -o /tmp/tmpivmpgbne/a.out
2025-01-08 17:40:37,121 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmp20zdp2x1/test.o -laio -o /tmp/tmp20zdp2x1/a.out
2025-01-08 17:40:37,424 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpb15bvnyi/test.c -o /tmp/tmpb15bvnyi/test.o
2025-01-08 17:40:37,452 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpb15bvnyi/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpb15bvnyi/a.out
2025-01-08 17:40:37,477 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpaqf_tb6f/test.c -o /tmp/tmpaqf_tb6f/test.o
2025-01-08 17:40:37,505 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpaqf_tb6f/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpaqf_tb6f/a.out
2025-01-08 17:40:37,520 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpkcvvfuyn/test.c -o /tmp/tmpkcvvfuyn/test.o
2025-01-08 17:40:37,542 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpcde9wj__/test.c -o /tmp/tmpcde9wj__/test.o
2025-01-08 17:40:37,544 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpkcvvfuyn/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpkcvvfuyn/a.out
2025-01-08 17:40:37,564 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpcde9wj__/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpcde9wj__/a.out
2025-01-08 17:45:22,791 - INFO - Training started
2025-01-08 17:45:22,791 - INFO - Total steps: 2
2025-01-08 17:48:31,433 - INFO - Loss improved from inf to 1.97711
2025-01-08 17:48:31,433 - INFO - Loss improved from inf to 1.97711
2025-01-08 17:48:31,433 - INFO - Loss improved from inf to 1.97711
2025-01-08 17:48:31,435 - INFO - Step 1/2 (50.0%), epoch: 1.0000, step_time: 572.94s, elapsed_time: 572.94s
2025-01-08 17:48:31,436 - INFO - Evaluation Results:
  eval_loss: 1.9771
  eval_runtime: 24.9754
  eval_samples_per_second: 0.3200
  eval_steps_per_second: 0.0800
  epoch: 1.0000
  elapsed_time: 572.94s
  step_time: 572.94s
2025-01-08 17:48:31,436 - INFO - Loss improved from inf to 1.97711
2025-01-08 17:51:58,678 - INFO - Saving model to mistral-sft-lora-fsdp2/checkpoint-1/pytorch_model_fsdp_0
2025-01-08 17:52:02,292 - INFO - Model saved to mistral-sft-lora-fsdp2/checkpoint-1/pytorch_model_fsdp_0
2025-01-08 17:52:08,611 - INFO - Saving Optimizer state to mistral-sft-lora-fsdp2/checkpoint-1/optimizer_0
2025-01-08 17:52:14,794 - INFO - Optimizer state saved in mistral-sft-lora-fsdp2/checkpoint-1/optimizer_0
2025-01-08 17:56:48,201 - INFO - Saving model to mistral-sft-lora-fsdp2/checkpoint-2/pytorch_model_fsdp_0
2025-01-08 17:56:51,286 - INFO - Model saved to mistral-sft-lora-fsdp2/checkpoint-2/pytorch_model_fsdp_0
2025-01-08 17:56:57,344 - INFO - Saving Optimizer state to mistral-sft-lora-fsdp2/checkpoint-2/optimizer_0
2025-01-08 17:57:03,316 - INFO - Optimizer state saved in mistral-sft-lora-fsdp2/checkpoint-2/optimizer_0