Hugging Face's logo

Adil1567
/
mistral-sft-lora-fsdp2

PEFT
TensorBoard
Safetensors
trl
sft
Generated from Trainer
Model card Files Metrics Community
Adil1567 commited on
Commit
5a54817
·
verified ·
1 Parent(s): 0bc4c6d

Training in progress, epoch 1

Browse files
Files changed (3) hide show
  1. logs/training_log.txt +34 -40
  2. runs/Jan08_13-47-23_gpu-server/events.out.tfevents.1736344266.gpu-server.882619.0 +3 -0
  3. training_args.bin +1 -1
logs/training_log.txt CHANGED
@@ -1,41 +1,35 @@
1
- 2025-01-08 13:17:15,790 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpy2w4xzh7/test.c -o /tmp/tmpy2w4xzh7/test.o
2
- 2025-01-08 13:17:15,818 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpy2w4xzh7/test.o -laio -o /tmp/tmpy2w4xzh7/a.out
3
- 2025-01-08 13:17:15,979 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmp_ss7f3o8/test.c -o /tmp/tmp_ss7f3o8/test.o
4
- 2025-01-08 13:17:15,994 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmp01z9ld03/test.c -o /tmp/tmp01z9ld03/test.o
5
- 2025-01-08 13:17:15,998 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpvxnp3r53/test.c -o /tmp/tmpvxnp3r53/test.o
6
- 2025-01-08 13:17:16,006 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmp_ss7f3o8/test.o -laio -o /tmp/tmp_ss7f3o8/a.out
7
- 2025-01-08 13:17:16,019 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmp01z9ld03/test.o -laio -o /tmp/tmp01z9ld03/a.out
8
- 2025-01-08 13:17:16,027 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpvxnp3r53/test.o -laio -o /tmp/tmpvxnp3r53/a.out
9
- 2025-01-08 13:17:16,260 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmp3xk4vmw9/test.c -o /tmp/tmp3xk4vmw9/test.o
10
- 2025-01-08 13:17:16,288 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmp3xk4vmw9/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmp3xk4vmw9/a.out
11
- 2025-01-08 13:17:16,423 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpvwuc1you/test.c -o /tmp/tmpvwuc1you/test.o
12
- 2025-01-08 13:17:16,443 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpvwuc1you/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpvwuc1you/a.out
13
- 2025-01-08 13:17:16,452 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpi2xei08o/test.c -o /tmp/tmpi2xei08o/test.o
14
- 2025-01-08 13:17:16,454 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmp7ewsdzvg/test.c -o /tmp/tmp7ewsdzvg/test.o
15
- 2025-01-08 13:17:16,474 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpi2xei08o/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpi2xei08o/a.out
16
- 2025-01-08 13:17:16,475 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmp7ewsdzvg/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmp7ewsdzvg/a.out
17
- 2025-01-08 13:19:32,863 - INFO - Training started
18
- 2025-01-08 13:19:32,863 - INFO - Total steps: 1
19
- 2025-01-08 13:23:00,937 - INFO - Saving model to mistral-sft-lora-fsdp2/checkpoint-1/pytorch_model_fsdp_0
20
- 2025-01-08 13:23:04,224 - INFO - Model saved to mistral-sft-lora-fsdp2/checkpoint-1/pytorch_model_fsdp_0
21
- 2025-01-08 13:23:09,119 - INFO - Saving Optimizer state to mistral-sft-lora-fsdp2/checkpoint-1/optimizer_0
22
- 2025-01-08 13:23:15,399 - INFO - Optimizer state saved in mistral-sft-lora-fsdp2/checkpoint-1/optimizer_0
23
- 2025-01-08 13:23:39,370 - INFO - Loss improved from inf to 1.97963
24
- 2025-01-08 13:23:39,370 - INFO - Loss improved from inf to 1.97963
25
- 2025-01-08 13:23:39,370 - INFO - Loss improved from inf to 1.97963
26
- 2025-01-08 13:23:39,372 - INFO - Step 1/1 (100.0%), epoch: 1.0000, step_time: 493.83s, elapsed_time: 493.83s
27
- 2025-01-08 13:23:39,373 - INFO - Evaluation Results:
28
- eval_loss: 1.9796
29
- eval_runtime: 23.7191
30
- eval_samples_per_second: 0.3370
31
- eval_steps_per_second: 0.0840
32
  epoch: 1.0000
33
- elapsed_time: 493.83s
34
- step_time: 493.83s
35
- 2025-01-08 13:23:39,373 - INFO - Loss improved from inf to 1.97963
36
- 2025-01-08 13:24:40,570 - INFO - Saving model to mistral-sft-lora-fsdp2/checkpoint-1/pytorch_model_fsdp_0
37
- 2025-01-08 13:24:43,636 - INFO - Model saved to mistral-sft-lora-fsdp2/checkpoint-1/pytorch_model_fsdp_0
38
- 2025-01-08 13:24:48,787 - INFO - Saving Optimizer state to mistral-sft-lora-fsdp2/checkpoint-1/optimizer_0
39
- 2025-01-08 13:24:54,891 - INFO - Optimizer state saved in mistral-sft-lora-fsdp2/checkpoint-1/optimizer_0
40
- 2025-01-08 13:24:55,351 - INFO - Step 1/1 (100.0%), epoch: 1.0000, step_time: 75.98s, elapsed_time: 569.81s
41
- 2025-01-08 13:24:55,353 - INFO - Training completed in 569.81 seconds
 
1
+ 2025-01-08 13:49:01,305 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpejuv75wg/test.c -o /tmp/tmpejuv75wg/test.o
2
+ 2025-01-08 13:49:01,329 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpejuv75wg/test.o -laio -o /tmp/tmpejuv75wg/a.out
3
+ 2025-01-08 13:49:01,445 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmph1wqufet/test.c -o /tmp/tmph1wqufet/test.o
4
+ 2025-01-08 13:49:01,461 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpn7ir_9h3/test.c -o /tmp/tmpn7ir_9h3/test.o
5
+ 2025-01-08 13:49:01,462 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmplbt1g8ij/test.c -o /tmp/tmplbt1g8ij/test.o
6
+ 2025-01-08 13:49:01,470 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmph1wqufet/test.o -laio -o /tmp/tmph1wqufet/a.out
7
+ 2025-01-08 13:49:01,490 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmplbt1g8ij/test.o -laio -o /tmp/tmplbt1g8ij/a.out
8
+ 2025-01-08 13:49:01,490 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpn7ir_9h3/test.o -laio -o /tmp/tmpn7ir_9h3/a.out
9
+ 2025-01-08 13:49:01,791 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmp387_6tzl/test.c -o /tmp/tmp387_6tzl/test.o
10
+ 2025-01-08 13:49:01,819 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmp387_6tzl/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmp387_6tzl/a.out
11
+ 2025-01-08 13:49:01,929 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpbb4cu51n/test.c -o /tmp/tmpbb4cu51n/test.o
12
+ 2025-01-08 13:49:01,940 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmp6me18vjb/test.c -o /tmp/tmp6me18vjb/test.o
13
+ 2025-01-08 13:49:01,951 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpbb4cu51n/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpbb4cu51n/a.out
14
+ 2025-01-08 13:49:01,961 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -O2 -isystem /root/anaconda3/envs/faiss_1.8.0/include -fPIC -c /tmp/tmpu5nbozol/test.c -o /tmp/tmpu5nbozol/test.o
15
+ 2025-01-08 13:49:01,966 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmp6me18vjb/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmp6me18vjb/a.out
16
+ 2025-01-08 13:49:01,985 - INFO - gcc -pthread -B /root/anaconda3/envs/faiss_1.8.0/compiler_compat /tmp/tmpu5nbozol/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpu5nbozol/a.out
17
+ 2025-01-08 13:51:07,759 - INFO - Training started
18
+ 2025-01-08 13:51:07,759 - INFO - Total steps: 2
19
+ 2025-01-08 13:53:58,210 - INFO - Loss improved from inf to 1.97976
20
+ 2025-01-08 13:53:58,210 - INFO - Loss improved from inf to 1.97976
21
+ 2025-01-08 13:53:58,210 - INFO - Loss improved from inf to 1.97976
22
+ 2025-01-08 13:53:58,212 - INFO - Step 1/2 (50.0%), epoch: 1.0000, step_time: 394.79s, elapsed_time: 394.79s
23
+ 2025-01-08 13:53:58,213 - INFO - Evaluation Results:
24
+ eval_loss: 1.9798
25
+ eval_runtime: 23.5550
26
+ eval_samples_per_second: 0.3400
27
+ eval_steps_per_second: 0.0850
 
 
 
 
28
  epoch: 1.0000
29
+ elapsed_time: 394.79s
30
+ step_time: 394.79s
31
+ 2025-01-08 13:53:58,213 - INFO - Loss improved from inf to 1.97976
32
+ 2025-01-08 13:54:59,859 - INFO - Saving model to mistral-sft-lora-fsdp2/checkpoint-1/pytorch_model_fsdp_0
33
+ 2025-01-08 13:55:03,136 - INFO - Model saved to mistral-sft-lora-fsdp2/checkpoint-1/pytorch_model_fsdp_0
34
+ 2025-01-08 13:55:08,046 - INFO - Saving Optimizer state to mistral-sft-lora-fsdp2/checkpoint-1/optimizer_0
35
+ 2025-01-08 13:55:14,259 - INFO - Optimizer state saved in mistral-sft-lora-fsdp2/checkpoint-1/optimizer_0
 
 
runs/Jan08_13-47-23_gpu-server/events.out.tfevents.1736344266.gpu-server.882619.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8dc1e98fd384467f592194e00f17315e978da77c8188ec70ec9d163ed6d145e6
3
+ size 5873
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:fb0a9b546c498f7273f6bb40df7393fc178510b49b42b419889178dbe21b9258
3
  size 5560
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c371cb2d8df70dd7c76903b3ee3adcead61312f419be560aae3002d5638c614
3
  size 5560