Mohamed Boghdady
commited on
End of training
Browse files
wandb/debug-internal.log
CHANGED
@@ -5133,3 +5133,33 @@ subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after
|
|
5133 |
2024-07-19 10:06:43,661 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240719_090532-oy10h8oj/files/output.log
|
5134 |
2024-07-19 10:06:44,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5135 |
2024-07-19 10:06:45,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
5133 |
2024-07-19 10:06:43,661 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240719_090532-oy10h8oj/files/output.log
|
5134 |
2024-07-19 10:06:44,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5135 |
2024-07-19 10:06:45,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5136 |
+
2024-07-19 10:06:45,884 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: status_report
|
5137 |
+
2024-07-19 10:06:46,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5138 |
+
2024-07-19 10:06:47,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5139 |
+
2024-07-19 10:06:48,287 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5140 |
+
2024-07-19 10:06:48,324 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: pause
|
5141 |
+
2024-07-19 10:06:48,325 INFO HandlerThread:143 [handler.py:handle_request_pause():724] stopping system metrics thread
|
5142 |
+
2024-07-19 10:06:48,325 INFO HandlerThread:143 [system_monitor.py:finish():203] Stopping system monitor
|
5143 |
+
2024-07-19 10:06:48,325 INFO HandlerThread:143 [interfaces.py:finish():200] Joined cpu monitor
|
5144 |
+
2024-07-19 10:06:48,325 DEBUG SystemMonitor:143 [system_monitor.py:_start():172] Starting system metrics aggregation loop
|
5145 |
+
2024-07-19 10:06:48,326 DEBUG SystemMonitor:143 [system_monitor.py:_start():179] Finished system metrics aggregation loop
|
5146 |
+
2024-07-19 10:06:48,326 DEBUG SystemMonitor:143 [system_monitor.py:_start():183] Publishing last batch of metrics
|
5147 |
+
2024-07-19 10:06:48,326 INFO HandlerThread:143 [interfaces.py:finish():200] Joined disk monitor
|
5148 |
+
2024-07-19 10:06:48,334 INFO HandlerThread:143 [interfaces.py:finish():200] Joined gpu monitor
|
5149 |
+
2024-07-19 10:06:48,334 INFO HandlerThread:143 [interfaces.py:finish():200] Joined memory monitor
|
5150 |
+
2024-07-19 10:06:48,334 INFO HandlerThread:143 [interfaces.py:finish():200] Joined network monitor
|
5151 |
+
2024-07-19 10:06:48,335 DEBUG SenderThread:143 [sender.py:send():379] send: stats
|
5152 |
+
2024-07-19 10:06:48,365 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: resume
|
5153 |
+
2024-07-19 10:06:48,365 INFO HandlerThread:143 [handler.py:handle_request_resume():715] starting system metrics thread
|
5154 |
+
2024-07-19 10:06:48,365 INFO HandlerThread:143 [system_monitor.py:start():194] Starting system monitor
|
5155 |
+
2024-07-19 10:06:48,366 INFO SystemMonitor:143 [system_monitor.py:_start():158] Starting system asset monitoring threads
|
5156 |
+
2024-07-19 10:06:48,366 INFO SystemMonitor:143 [interfaces.py:start():188] Started cpu monitoring
|
5157 |
+
2024-07-19 10:06:48,367 INFO SystemMonitor:143 [interfaces.py:start():188] Started disk monitoring
|
5158 |
+
2024-07-19 10:06:48,368 INFO SystemMonitor:143 [interfaces.py:start():188] Started gpu monitoring
|
5159 |
+
2024-07-19 10:06:48,369 INFO SystemMonitor:143 [interfaces.py:start():188] Started memory monitoring
|
5160 |
+
2024-07-19 10:06:48,370 INFO SystemMonitor:143 [interfaces.py:start():188] Started network monitoring
|
5161 |
+
2024-07-19 10:06:49,287 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5162 |
+
2024-07-19 10:06:49,525 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: stop_status
|
5163 |
+
2024-07-19 10:06:49,526 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
|
5164 |
+
2024-07-19 10:06:49,663 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240719_090532-oy10h8oj/files/output.log
|
5165 |
+
2024-07-19 10:06:50,287 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
wandb/debug.log
CHANGED
@@ -74,3 +74,6 @@ config: {}
|
|
74 |
2024-07-19 10:06:42,046 INFO MainThread:35 [jupyter.py:save_ipynb():372] not saving jupyter notebook
|
75 |
2024-07-19 10:06:42,047 INFO MainThread:35 [wandb_init.py:_pause_backend():440] pausing backend
|
76 |
2024-07-19 10:06:42,064 INFO MainThread:35 [wandb_init.py:_resume_backend():445] resuming backend
|
|
|
|
|
|
|
|
74 |
2024-07-19 10:06:42,046 INFO MainThread:35 [jupyter.py:save_ipynb():372] not saving jupyter notebook
|
75 |
2024-07-19 10:06:42,047 INFO MainThread:35 [wandb_init.py:_pause_backend():440] pausing backend
|
76 |
2024-07-19 10:06:42,064 INFO MainThread:35 [wandb_init.py:_resume_backend():445] resuming backend
|
77 |
+
2024-07-19 10:06:48,324 INFO MainThread:35 [jupyter.py:save_ipynb():372] not saving jupyter notebook
|
78 |
+
2024-07-19 10:06:48,324 INFO MainThread:35 [wandb_init.py:_pause_backend():440] pausing backend
|
79 |
+
2024-07-19 10:06:48,330 INFO MainThread:35 [wandb_init.py:_resume_backend():445] resuming backend
|
wandb/run-20240719_090532-oy10h8oj/files/output.log
CHANGED
@@ -13,3 +13,5 @@ Non-default generation parameters: {'max_length': 512, 'num_beams': 4, 'bad_word
|
|
13 |
Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model) instead. This warning will be raised to an exception in v4.41.
|
14 |
Non-default generation parameters: {'max_length': 512, 'num_beams': 4, 'bad_words_ids': [[62833]], 'forced_eos_token_id': 0}
|
15 |
Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model) instead. This warning will be raised to an exception in v4.41.
|
|
|
|
|
|
13 |
Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model) instead. This warning will be raised to an exception in v4.41.
|
14 |
Non-default generation parameters: {'max_length': 512, 'num_beams': 4, 'bad_words_ids': [[62833]], 'forced_eos_token_id': 0}
|
15 |
Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model) instead. This warning will be raised to an exception in v4.41.
|
16 |
+
Non-default generation parameters: {'max_length': 512, 'num_beams': 4, 'bad_words_ids': [[62833]], 'forced_eos_token_id': 0}
|
17 |
+
Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model) instead. This warning will be raised to an exception in v4.41.
|
wandb/run-20240719_090532-oy10h8oj/logs/debug-internal.log
CHANGED
@@ -5133,3 +5133,33 @@ subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after
|
|
5133 |
2024-07-19 10:06:43,661 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240719_090532-oy10h8oj/files/output.log
|
5134 |
2024-07-19 10:06:44,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5135 |
2024-07-19 10:06:45,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
5133 |
2024-07-19 10:06:43,661 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240719_090532-oy10h8oj/files/output.log
|
5134 |
2024-07-19 10:06:44,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5135 |
2024-07-19 10:06:45,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5136 |
+
2024-07-19 10:06:45,884 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: status_report
|
5137 |
+
2024-07-19 10:06:46,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5138 |
+
2024-07-19 10:06:47,286 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5139 |
+
2024-07-19 10:06:48,287 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5140 |
+
2024-07-19 10:06:48,324 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: pause
|
5141 |
+
2024-07-19 10:06:48,325 INFO HandlerThread:143 [handler.py:handle_request_pause():724] stopping system metrics thread
|
5142 |
+
2024-07-19 10:06:48,325 INFO HandlerThread:143 [system_monitor.py:finish():203] Stopping system monitor
|
5143 |
+
2024-07-19 10:06:48,325 INFO HandlerThread:143 [interfaces.py:finish():200] Joined cpu monitor
|
5144 |
+
2024-07-19 10:06:48,325 DEBUG SystemMonitor:143 [system_monitor.py:_start():172] Starting system metrics aggregation loop
|
5145 |
+
2024-07-19 10:06:48,326 DEBUG SystemMonitor:143 [system_monitor.py:_start():179] Finished system metrics aggregation loop
|
5146 |
+
2024-07-19 10:06:48,326 DEBUG SystemMonitor:143 [system_monitor.py:_start():183] Publishing last batch of metrics
|
5147 |
+
2024-07-19 10:06:48,326 INFO HandlerThread:143 [interfaces.py:finish():200] Joined disk monitor
|
5148 |
+
2024-07-19 10:06:48,334 INFO HandlerThread:143 [interfaces.py:finish():200] Joined gpu monitor
|
5149 |
+
2024-07-19 10:06:48,334 INFO HandlerThread:143 [interfaces.py:finish():200] Joined memory monitor
|
5150 |
+
2024-07-19 10:06:48,334 INFO HandlerThread:143 [interfaces.py:finish():200] Joined network monitor
|
5151 |
+
2024-07-19 10:06:48,335 DEBUG SenderThread:143 [sender.py:send():379] send: stats
|
5152 |
+
2024-07-19 10:06:48,365 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: resume
|
5153 |
+
2024-07-19 10:06:48,365 INFO HandlerThread:143 [handler.py:handle_request_resume():715] starting system metrics thread
|
5154 |
+
2024-07-19 10:06:48,365 INFO HandlerThread:143 [system_monitor.py:start():194] Starting system monitor
|
5155 |
+
2024-07-19 10:06:48,366 INFO SystemMonitor:143 [system_monitor.py:_start():158] Starting system asset monitoring threads
|
5156 |
+
2024-07-19 10:06:48,366 INFO SystemMonitor:143 [interfaces.py:start():188] Started cpu monitoring
|
5157 |
+
2024-07-19 10:06:48,367 INFO SystemMonitor:143 [interfaces.py:start():188] Started disk monitoring
|
5158 |
+
2024-07-19 10:06:48,368 INFO SystemMonitor:143 [interfaces.py:start():188] Started gpu monitoring
|
5159 |
+
2024-07-19 10:06:48,369 INFO SystemMonitor:143 [interfaces.py:start():188] Started memory monitoring
|
5160 |
+
2024-07-19 10:06:48,370 INFO SystemMonitor:143 [interfaces.py:start():188] Started network monitoring
|
5161 |
+
2024-07-19 10:06:49,287 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
5162 |
+
2024-07-19 10:06:49,525 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: stop_status
|
5163 |
+
2024-07-19 10:06:49,526 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
|
5164 |
+
2024-07-19 10:06:49,663 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240719_090532-oy10h8oj/files/output.log
|
5165 |
+
2024-07-19 10:06:50,287 DEBUG HandlerThread:143 [handler.py:handle_request():158] handle_request: internal_messages
|
wandb/run-20240719_090532-oy10h8oj/logs/debug.log
CHANGED
@@ -74,3 +74,6 @@ config: {}
|
|
74 |
2024-07-19 10:06:42,046 INFO MainThread:35 [jupyter.py:save_ipynb():372] not saving jupyter notebook
|
75 |
2024-07-19 10:06:42,047 INFO MainThread:35 [wandb_init.py:_pause_backend():440] pausing backend
|
76 |
2024-07-19 10:06:42,064 INFO MainThread:35 [wandb_init.py:_resume_backend():445] resuming backend
|
|
|
|
|
|
|
|
74 |
2024-07-19 10:06:42,046 INFO MainThread:35 [jupyter.py:save_ipynb():372] not saving jupyter notebook
|
75 |
2024-07-19 10:06:42,047 INFO MainThread:35 [wandb_init.py:_pause_backend():440] pausing backend
|
76 |
2024-07-19 10:06:42,064 INFO MainThread:35 [wandb_init.py:_resume_backend():445] resuming backend
|
77 |
+
2024-07-19 10:06:48,324 INFO MainThread:35 [jupyter.py:save_ipynb():372] not saving jupyter notebook
|
78 |
+
2024-07-19 10:06:48,324 INFO MainThread:35 [wandb_init.py:_pause_backend():440] pausing backend
|
79 |
+
2024-07-19 10:06:48,330 INFO MainThread:35 [wandb_init.py:_resume_backend():445] resuming backend
|
wandb/run-20240719_090532-oy10h8oj/run-oy10h8oj.wandb
CHANGED
Binary files a/wandb/run-20240719_090532-oy10h8oj/run-oy10h8oj.wandb and b/wandb/run-20240719_090532-oy10h8oj/run-oy10h8oj.wandb differ
|
|