kertob commited on
Commit
8ea32f0
·
verified ·
1 Parent(s): fd5952b

Training in progress, step 20

Browse files
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2a28934476a18443bfe50c0997a6cba44bcf12405bcedda3f5cecd5e89f37b5b
3
  size 261131840
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:62652c0d02f97dbd4b25e5ab09ff70e675566b6bbb8e566635c600415c02166a
3
  size 261131840
runs/Mar13_11-36-50_7acbe5a74b63/events.out.tfevents.1710329812.7acbe5a74b63.337.1 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ef2b607d9c2c2ec4203367ba38a7d3126b24cbf8d06d77f595ab56562f697dbf
3
- size 5757
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4a13dd6993a0b35369e6dbfae04abf6013f2700eeba4fb5136ea1ff7214ba3ed
3
+ size 5964
wandb/debug-internal.log CHANGED
@@ -602,3 +602,46 @@
602
  2024-03-13 11:38:16,518 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
603
  2024-03-13 11:38:21,519 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
604
  2024-03-13 11:38:22,321 DEBUG SenderThread:5135 [sender.py:send():378] send: stats
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
602
  2024-03-13 11:38:16,518 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
603
  2024-03-13 11:38:21,519 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
604
  2024-03-13 11:38:22,321 DEBUG SenderThread:5135 [sender.py:send():378] send: stats
605
+ 2024-03-13 11:38:26,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
606
+ 2024-03-13 11:38:26,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
607
+ 2024-03-13 11:38:26,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
608
+ 2024-03-13 11:38:27,136 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
609
+ 2024-03-13 11:38:32,138 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
610
+ 2024-03-13 11:38:37,138 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
611
+ 2024-03-13 11:38:41,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
612
+ 2024-03-13 11:38:41,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
613
+ 2024-03-13 11:38:41,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
614
+ 2024-03-13 11:38:42,144 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
615
+ 2024-03-13 11:38:47,145 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
616
+ 2024-03-13 11:38:52,146 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
617
+ 2024-03-13 11:38:52,322 DEBUG SenderThread:5135 [sender.py:send():378] send: stats
618
+ 2024-03-13 11:38:56,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
619
+ 2024-03-13 11:38:56,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
620
+ 2024-03-13 11:38:56,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
621
+ 2024-03-13 11:38:57,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
622
+ 2024-03-13 11:39:02,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
623
+ 2024-03-13 11:39:07,149 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
624
+ 2024-03-13 11:39:11,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
625
+ 2024-03-13 11:39:11,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
626
+ 2024-03-13 11:39:11,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
627
+ 2024-03-13 11:39:12,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
628
+ 2024-03-13 11:39:17,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
629
+ 2024-03-13 11:39:22,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
630
+ 2024-03-13 11:39:22,322 DEBUG SenderThread:5135 [sender.py:send():378] send: stats
631
+ 2024-03-13 11:39:26,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
632
+ 2024-03-13 11:39:26,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
633
+ 2024-03-13 11:39:26,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
634
+ 2024-03-13 11:39:28,146 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
635
+ 2024-03-13 11:39:33,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
636
+ 2024-03-13 11:39:38,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
637
+ 2024-03-13 11:39:41,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
638
+ 2024-03-13 11:39:41,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
639
+ 2024-03-13 11:39:41,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
640
+ 2024-03-13 11:39:41,592 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history
641
+ 2024-03-13 11:39:41,594 DEBUG SenderThread:5135 [sender.py:send():378] send: history
642
+ 2024-03-13 11:39:41,594 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record
643
+ 2024-03-13 11:39:41,595 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
644
+ 2024-03-13 11:39:41,911 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json
645
+ 2024-03-13 11:39:43,633 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
646
+ 2024-03-13 11:39:43,911 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/output.log
647
+ 2024-03-13 11:39:48,634 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240313_112023-t8o614bo/files/output.log CHANGED
@@ -1 +1,2 @@
1
 
 
 
1
 
2
+ Checkpoint destination directory ./checkpoint-10 already exists and is non-empty. Saving will proceed but saved results may be invalid.
wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 0.7841, "train/grad_norm": 1.0399196147918701, "train/learning_rate": 9.989514131188559e-05, "train/epoch": 0.8, "train/global_step": 10, "_timestamp": 1710329893.4763706, "_runtime": 1070.0943615436554, "_step": 4, "train/train_runtime": 361.3918, "train/train_samples_per_second": 1.328, "train/train_steps_per_second": 0.083, "train/total_flos": 1585771096988928.0, "train/train_loss": 1.3836020787556966}
 
1
+ {"train/loss": 0.6043, "train/grad_norm": 1.03736412525177, "train/learning_rate": 9.872052623234632e-05, "train/epoch": 1.6, "train/global_step": 20, "_timestamp": 1710329981.5923584, "_runtime": 1158.2103493213654, "_step": 5, "train/train_runtime": 361.3918, "train/train_samples_per_second": 1.328, "train/train_steps_per_second": 0.083, "train/total_flos": 1585771096988928.0, "train/train_loss": 1.3836020787556966}
wandb/run-20240313_112023-t8o614bo/logs/debug-internal.log CHANGED
@@ -602,3 +602,46 @@
602
  2024-03-13 11:38:16,518 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
603
  2024-03-13 11:38:21,519 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
604
  2024-03-13 11:38:22,321 DEBUG SenderThread:5135 [sender.py:send():378] send: stats
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
602
  2024-03-13 11:38:16,518 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
603
  2024-03-13 11:38:21,519 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
604
  2024-03-13 11:38:22,321 DEBUG SenderThread:5135 [sender.py:send():378] send: stats
605
+ 2024-03-13 11:38:26,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
606
+ 2024-03-13 11:38:26,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
607
+ 2024-03-13 11:38:26,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
608
+ 2024-03-13 11:38:27,136 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
609
+ 2024-03-13 11:38:32,138 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
610
+ 2024-03-13 11:38:37,138 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
611
+ 2024-03-13 11:38:41,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
612
+ 2024-03-13 11:38:41,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
613
+ 2024-03-13 11:38:41,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
614
+ 2024-03-13 11:38:42,144 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
615
+ 2024-03-13 11:38:47,145 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
616
+ 2024-03-13 11:38:52,146 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
617
+ 2024-03-13 11:38:52,322 DEBUG SenderThread:5135 [sender.py:send():378] send: stats
618
+ 2024-03-13 11:38:56,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
619
+ 2024-03-13 11:38:56,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
620
+ 2024-03-13 11:38:56,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
621
+ 2024-03-13 11:38:57,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
622
+ 2024-03-13 11:39:02,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
623
+ 2024-03-13 11:39:07,149 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
624
+ 2024-03-13 11:39:11,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
625
+ 2024-03-13 11:39:11,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
626
+ 2024-03-13 11:39:11,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
627
+ 2024-03-13 11:39:12,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
628
+ 2024-03-13 11:39:17,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
629
+ 2024-03-13 11:39:22,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
630
+ 2024-03-13 11:39:22,322 DEBUG SenderThread:5135 [sender.py:send():378] send: stats
631
+ 2024-03-13 11:39:26,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
632
+ 2024-03-13 11:39:26,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
633
+ 2024-03-13 11:39:26,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
634
+ 2024-03-13 11:39:28,146 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
635
+ 2024-03-13 11:39:33,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
636
+ 2024-03-13 11:39:38,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
637
+ 2024-03-13 11:39:41,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status
638
+ 2024-03-13 11:39:41,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status
639
+ 2024-03-13 11:39:41,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages
640
+ 2024-03-13 11:39:41,592 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history
641
+ 2024-03-13 11:39:41,594 DEBUG SenderThread:5135 [sender.py:send():378] send: history
642
+ 2024-03-13 11:39:41,594 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record
643
+ 2024-03-13 11:39:41,595 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
644
+ 2024-03-13 11:39:41,911 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json
645
+ 2024-03-13 11:39:43,633 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
646
+ 2024-03-13 11:39:43,911 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/output.log
647
+ 2024-03-13 11:39:48,634 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240313_112023-t8o614bo/run-t8o614bo.wandb CHANGED
Binary files a/wandb/run-20240313_112023-t8o614bo/run-t8o614bo.wandb and b/wandb/run-20240313_112023-t8o614bo/run-t8o614bo.wandb differ