| 2024-03-13 11:20:23,382 INFO StreamThr :5135 [internal.py:wandb_internal():86] W&B internal server running at pid: 5135, started at: 2024-03-13 11:20:23.382058 | |
| 2024-03-13 11:20:23,384 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status | |
| 2024-03-13 11:20:25,307 INFO WriterThread:5135 [datastore.py:open_for_write():87] open: /content/wandb/run-20240313_112023-t8o614bo/run-t8o614bo.wandb | |
| 2024-03-13 11:20:25,307 DEBUG SenderThread:5135 [sender.py:send():378] send: header | |
| 2024-03-13 11:20:25,309 DEBUG SenderThread:5135 [sender.py:send():378] send: run | |
| 2024-03-13 11:20:25,535 INFO SenderThread:5135 [dir_watcher.py:__init__():211] watching files in: /content/wandb/run-20240313_112023-t8o614bo/files | |
| 2024-03-13 11:20:25,535 INFO SenderThread:5135 [sender.py:_start_run_threads():1099] run started: t8o614bo with start time 1710328823.382009 | |
| 2024-03-13 11:20:25,540 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-03-13 11:20:25,540 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: check_version | |
| 2024-03-13 11:20:25,679 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-03-13 11:20:25,688 DEBUG HandlerThread:5135 [system_info.py:__init__():26] System info init | |
| 2024-03-13 11:20:25,688 DEBUG HandlerThread:5135 [system_info.py:__init__():41] System info init done | |
| 2024-03-13 11:20:25,688 INFO HandlerThread:5135 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-13 11:20:25,689 INFO SystemMonitor:5135 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-13 11:20:25,689 INFO HandlerThread:5135 [system_monitor.py:probe():214] Collecting system info | |
| 2024-03-13 11:20:25,689 INFO SystemMonitor:5135 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-13 11:20:25,690 INFO SystemMonitor:5135 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-13 11:20:25,691 INFO SystemMonitor:5135 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-13 11:20:25,692 INFO SystemMonitor:5135 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-13 11:20:25,693 INFO SystemMonitor:5135 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-13 11:20:25,703 DEBUG HandlerThread:5135 [system_info.py:probe():150] Probing system | |
| 2024-03-13 11:20:25,704 DEBUG HandlerThread:5135 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-03-13 11:20:25,704 DEBUG HandlerThread:5135 [system_info.py:probe():198] Probing system done | |
| 2024-03-13 11:20:25,704 DEBUG HandlerThread:5135 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-13T11:20:25.703508', 'startedAt': '2024-03-13T11:20:23.376192', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Falcon-FineTuning.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1kv2VO7q0c8xX_DQt5NERLeJJZtHW6zHO', 'host': '7acbe5a74b63', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 2299.998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 201.22513961791992, 'used': 40.209312438964844}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 50.993690490722656}} | |
| 2024-03-13 11:20:25,704 INFO HandlerThread:5135 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-03-13 11:20:25,704 INFO HandlerThread:5135 [system_monitor.py:probe():227] Publishing system info | |
| 2024-03-13 11:20:25,705 INFO HandlerThread:5135 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-03-13 11:20:25,709 DEBUG SenderThread:5135 [sender.py:send():378] send: files | |
| 2024-03-13 11:20:25,709 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now | |
| 2024-03-13 11:20:26,064 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-03-13 11:20:26,064 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: python_packages | |
| 2024-03-13 11:20:26,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:20:26,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:20:26,107 DEBUG SenderThread:5135 [sender.py:send():378] send: telemetry | |
| 2024-03-13 11:20:26,108 DEBUG SenderThread:5135 [sender.py:send():378] send: config | |
| 2024-03-13 11:20:26,109 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:20:26,109 DEBUG SenderThread:5135 [sender.py:send():378] send: telemetry | |
| 2024-03-13 11:20:26,109 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:20:26,109 WARNING SenderThread:5135 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-03-13 11:20:26,109 DEBUG SenderThread:5135 [sender.py:send():378] send: telemetry | |
| 2024-03-13 11:20:26,110 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:20:26,133 INFO wandb-upload_0:5135 [upload_job.py:push():131] Uploaded file /tmp/tmpuwvyq0ztwandb/v6gi54r6-wandb-metadata.json | |
| 2024-03-13 11:20:26,537 INFO Thread-12 :5135 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240313_112023-t8o614bo/files/requirements.txt | |
| 2024-03-13 11:20:26,537 INFO Thread-12 :5135 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-metadata.json | |
| 2024-03-13 11:20:29,253 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:34,254 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:39,254 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:41,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:20:41,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:20:41,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:20:45,172 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:50,173 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:55,178 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:55,544 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/config.yaml | |
| 2024-03-13 11:20:56,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:20:56,065 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:20:56,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:21:01,127 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:06,128 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:11,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:21:11,065 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:21:11,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:21:11,189 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:16,190 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:21,191 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:25,693 DEBUG SystemMonitor:5135 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-13 11:21:25,694 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:21:26,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:21:26,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:21:26,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:21:26,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:31,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:36,197 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:41,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:21:41,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:21:41,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:21:41,199 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:46,200 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:47,552 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:21:47,554 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:21:47,554 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:21:47,554 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:21:47,554 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:21:47,555 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:21:47,555 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:21:47,556 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:21:47,560 INFO Thread-12 :5135 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:21:51,558 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:55,695 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:21:56,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:21:56,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:21:56,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:21:57,208 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:57,563 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/config.yaml | |
| 2024-03-13 11:22:02,349 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:07,350 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:11,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:22:11,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:22:11,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:22:13,224 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:18,225 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:23,226 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:25,695 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:22:26,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:22:26,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:22:26,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:22:29,191 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:34,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:39,193 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:41,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:22:41,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:22:41,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:22:45,190 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:50,191 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:55,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:55,696 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:22:56,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:22:56,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:22:56,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:23:00,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:05,231 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:10,232 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:11,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:23:11,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:23:11,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:23:16,001 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:23:16,002 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:23:16,003 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:23:16,004 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:16,005 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:23:16,588 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:23:21,006 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:25,697 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:23:26,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:23:26,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:23:26,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:23:26,173 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:31,174 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:36,175 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:41,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:23:41,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:23:41,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:23:41,207 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:46,208 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:51,209 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:55,698 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:23:56,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:23:56,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:23:56,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:23:57,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:02,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:07,163 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:11,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:24:11,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:24:11,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:24:12,169 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:17,170 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:22,171 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:25,699 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:24:26,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:24:26,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:24:26,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:24:27,228 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:32,229 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:37,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:41,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:24:41,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:24:41,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:24:43,204 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:44,484 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:24:44,485 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:24:44,485 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:24:44,487 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:24:44,617 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:24:48,489 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:48,768 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:24:48,770 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:24:48,770 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:24:48,770 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:24:48,771 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:24:48,771 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:24:48,771 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:24:48,771 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:24:48,771 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:24:49,618 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:24:53,773 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:55,700 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:24:56,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:24:56,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:24:56,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:24:58,385 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: pause | |
| 2024-03-13 11:24:58,385 INFO HandlerThread:5135 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-03-13 11:24:58,385 INFO HandlerThread:5135 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-03-13 11:24:58,386 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-03-13 11:24:58,386 DEBUG SystemMonitor:5135 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-13 11:24:58,386 DEBUG SystemMonitor:5135 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-13 11:24:58,387 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-03-13 11:24:58,393 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-03-13 11:24:58,393 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-03-13 11:24:58,393 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined network monitor | |
| 2024-03-13 11:24:58,394 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:24:59,395 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:04,400 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:08,625 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/config.yaml | |
| 2024-03-13 11:25:11,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:25:11,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:25:11,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:25:13,150 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:18,151 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:23,153 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:26,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:25:26,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:25:26,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:25:28,172 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:33,173 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:38,174 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:41,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:25:41,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:25:41,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:25:43,194 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:48,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:53,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:56,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:25:56,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:25:56,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:25:58,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:03,213 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:08,214 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:11,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:26:11,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:26:11,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:26:14,163 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:19,164 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:24,165 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:26,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:26:26,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:26:26,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:26:30,139 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:35,140 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:40,141 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:41,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:26:41,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:26:41,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:26:45,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:50,168 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:55,169 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:56,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:26:56,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:26:56,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:27:01,153 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:06,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:11,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:27:11,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:27:11,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:27:11,157 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:16,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:21,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:26,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:27:26,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:27:26,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:27:27,154 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:32,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:37,156 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:41,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:27:41,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:27:41,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:27:42,179 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:47,180 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:52,181 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:56,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:27:56,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:27:56,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:27:57,186 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:02,187 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:07,188 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:11,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:28:11,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:28:11,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:28:12,199 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:17,200 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:22,201 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:26,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:28:26,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:28:26,111 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:28:27,247 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:32,248 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:37,249 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:41,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:28:41,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:28:41,112 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:28:43,229 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:48,231 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:53,232 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:56,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:28:56,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:28:56,111 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:28:59,146 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:04,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:09,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:11,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:29:11,071 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:29:11,111 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:29:15,127 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:20,128 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:25,129 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:26,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:29:26,071 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:29:26,111 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:29:30,141 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:35,142 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:40,143 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:41,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:29:41,071 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:29:41,111 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:29:45,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:50,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:55,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:56,071 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:29:56,071 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:29:56,112 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:30:00,231 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:05,232 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:10,233 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:11,071 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:30:11,071 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:30:11,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:30:16,222 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:21,223 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:26,071 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:30:26,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:30:26,112 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:30:27,194 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:32,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:37,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:41,071 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:30:41,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:30:41,112 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:30:43,166 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:48,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:53,168 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:56,071 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:30:56,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:30:56,112 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:30:58,211 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:03,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:08,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:11,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:31:11,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:31:11,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:31:14,166 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:19,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:24,168 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:26,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:31:26,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:31:26,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:31:29,181 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:34,182 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:39,183 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:41,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:31:41,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:31:41,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:31:44,220 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:49,221 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:54,222 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:56,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:31:56,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:31:56,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:31:59,233 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:04,234 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:09,235 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:11,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:32:11,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:32:11,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:32:15,235 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:20,236 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:25,237 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:26,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:32:26,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:32:26,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:32:31,182 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:36,182 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:41,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:32:41,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:32:41,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:32:41,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:46,213 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:51,214 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:56,073 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:32:56,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:32:56,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:32:57,152 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:02,154 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:07,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:11,073 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:33:11,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:33:11,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:33:12,216 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:17,218 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:22,218 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:26,073 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:33:26,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:33:26,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:33:28,210 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:33,211 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:38,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:41,073 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:33:41,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:33:41,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:33:43,222 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:48,223 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:53,224 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:56,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:33:56,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:33:56,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:33:59,169 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:04,170 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:09,171 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:11,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:34:11,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:34:11,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:34:14,190 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:19,191 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:24,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:26,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:34:26,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:34:26,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:34:30,150 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:35,151 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:40,152 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:41,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:34:41,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:34:41,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:34:45,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:50,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:55,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:56,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:34:56,075 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:34:56,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:35:00,225 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:05,226 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:10,227 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:11,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:35:11,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:35:11,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:35:16,219 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:21,220 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:26,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:35:26,075 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:35:26,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:35:27,207 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:32,209 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:37,210 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:41,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:35:41,075 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:35:41,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:35:42,234 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:47,235 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:52,236 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:56,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:35:56,075 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:35:56,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:35:58,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:03,156 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:08,157 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:11,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:36:11,075 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:36:11,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:36:13,183 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:18,184 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:23,185 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:26,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:36:26,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:36:26,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:36:28,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:33,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:38,197 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:41,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:36:41,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:36:41,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:36:43,357 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:48,358 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:50,278 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: resume | |
| 2024-03-13 11:36:50,278 INFO HandlerThread:5135 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-03-13 11:36:50,278 INFO HandlerThread:5135 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-13 11:36:50,279 INFO SystemMonitor:5135 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-13 11:36:50,279 INFO SystemMonitor:5135 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-13 11:36:50,280 INFO SystemMonitor:5135 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-13 11:36:50,280 INFO SystemMonitor:5135 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-13 11:36:50,281 INFO SystemMonitor:5135 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-13 11:36:50,281 INFO SystemMonitor:5135 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-13 11:36:50,318 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: pause | |
| 2024-03-13 11:36:50,319 INFO HandlerThread:5135 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-03-13 11:36:50,319 INFO HandlerThread:5135 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-03-13 11:36:50,319 DEBUG SystemMonitor:5135 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-13 11:36:50,319 DEBUG SystemMonitor:5135 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-13 11:36:50,319 DEBUG SystemMonitor:5135 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-13 11:36:50,319 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-03-13 11:36:50,320 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-03-13 11:36:50,325 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-03-13 11:36:50,325 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-03-13 11:36:50,325 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined network monitor | |
| 2024-03-13 11:36:50,326 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:36:50,728 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: resume | |
| 2024-03-13 11:36:50,728 INFO HandlerThread:5135 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-03-13 11:36:50,728 INFO HandlerThread:5135 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-13 11:36:50,728 INFO SystemMonitor:5135 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-13 11:36:50,729 INFO SystemMonitor:5135 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-13 11:36:50,729 INFO SystemMonitor:5135 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-13 11:36:50,730 INFO SystemMonitor:5135 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-13 11:36:50,731 INFO SystemMonitor:5135 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-13 11:36:50,731 INFO SystemMonitor:5135 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-13 11:36:51,012 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: pause | |
| 2024-03-13 11:36:51,012 INFO HandlerThread:5135 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-03-13 11:36:51,012 INFO HandlerThread:5135 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-03-13 11:36:51,012 DEBUG SystemMonitor:5135 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-13 11:36:51,012 DEBUG SystemMonitor:5135 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-13 11:36:51,012 DEBUG SystemMonitor:5135 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-13 11:36:51,013 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-03-13 11:36:51,013 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-03-13 11:36:51,019 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-03-13 11:36:51,019 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-03-13 11:36:51,019 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined network monitor | |
| 2024-03-13 11:36:51,020 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:36:51,210 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: resume | |
| 2024-03-13 11:36:51,210 INFO HandlerThread:5135 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-03-13 11:36:51,210 INFO HandlerThread:5135 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-13 11:36:51,210 INFO SystemMonitor:5135 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-13 11:36:51,211 INFO SystemMonitor:5135 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-13 11:36:51,211 INFO SystemMonitor:5135 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-13 11:36:51,212 INFO SystemMonitor:5135 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-13 11:36:51,213 INFO SystemMonitor:5135 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-13 11:36:51,213 INFO SystemMonitor:5135 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-13 11:36:51,216 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: pause | |
| 2024-03-13 11:36:51,216 INFO HandlerThread:5135 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-03-13 11:36:51,216 INFO HandlerThread:5135 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-03-13 11:36:51,216 DEBUG SystemMonitor:5135 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-13 11:36:51,216 DEBUG SystemMonitor:5135 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-13 11:36:51,216 DEBUG SystemMonitor:5135 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-13 11:36:51,216 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-03-13 11:36:51,217 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-03-13 11:36:51,225 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-03-13 11:36:51,225 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-03-13 11:36:51,225 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined network monitor | |
| 2024-03-13 11:36:51,226 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:36:52,316 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: resume | |
| 2024-03-13 11:36:52,316 INFO HandlerThread:5135 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-03-13 11:36:52,316 INFO HandlerThread:5135 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-13 11:36:52,317 INFO SystemMonitor:5135 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-13 11:36:52,317 INFO SystemMonitor:5135 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-13 11:36:52,318 INFO SystemMonitor:5135 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-13 11:36:52,318 INFO SystemMonitor:5135 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-13 11:36:52,318 INFO SystemMonitor:5135 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-13 11:36:52,319 INFO SystemMonitor:5135 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-13 11:36:52,779 DEBUG SenderThread:5135 [sender.py:send():378] send: config | |
| 2024-03-13 11:36:52,781 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:36:52,781 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:36:52,781 WARNING SenderThread:5135 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-03-13 11:36:53,782 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:56,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:36:56,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:36:56,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:36:59,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:59,858 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/config.yaml | |
| 2024-03-13 11:37:04,288 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:09,289 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:11,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:37:11,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:37:11,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:37:15,201 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:20,202 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:25,203 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:26,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:37:26,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:37:26,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:37:30,214 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:35,215 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:40,216 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:41,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:37:41,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:37:41,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:37:46,194 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:51,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:52,319 DEBUG SystemMonitor:5135 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-13 11:37:52,320 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:37:56,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:37:56,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:37:56,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:37:56,199 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:01,199 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:06,200 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:11,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:38:11,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:38:11,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:38:11,201 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:13,476 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:38:13,478 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:38:13,478 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:38:13,479 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:38:13,881 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:38:13,882 INFO Thread-12 :5135 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240313_112023-t8o614bo/files/output.log | |
| 2024-03-13 11:38:15,882 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/output.log | |
| 2024-03-13 11:38:16,518 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:21,519 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:22,321 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:38:26,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:38:26,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:38:26,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:38:27,136 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:32,138 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:37,138 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:41,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:38:41,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:38:41,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:38:42,144 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:47,145 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:52,146 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:52,322 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:38:56,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:38:56,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:38:56,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:38:57,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:02,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:07,149 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:11,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:39:11,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:39:11,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:39:12,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:17,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:22,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:22,322 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:39:26,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:39:26,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:39:26,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:39:28,146 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:33,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:38,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:41,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:39:41,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:39:41,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:39:41,592 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:39:41,594 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:39:41,594 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:39:41,595 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:39:41,911 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:39:43,633 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:43,911 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/output.log | |
| 2024-03-13 11:39:48,634 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:52,323 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:39:54,324 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:56,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:39:56,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:39:56,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:40:00,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:05,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:10,197 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:11,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:40:11,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:40:11,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:40:16,154 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:21,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:22,323 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:40:26,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:40:26,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:40:26,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:40:26,175 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:31,175 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:36,176 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:41,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:40:41,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:40:41,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:40:41,208 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:46,209 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:51,209 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:52,324 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:40:56,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:40:56,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:40:56,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:40:57,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:02,149 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:07,150 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:08,398 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:41:08,399 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:41:08,399 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:41:08,399 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:41:08,939 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:41:09,939 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/output.log | |
| 2024-03-13 11:41:11,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:41:11,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:41:11,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:41:12,150 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:17,151 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:22,152 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:22,325 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:41:26,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:41:26,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:41:26,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:41:27,209 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:32,210 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:37,211 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:41,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:41:41,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:41:41,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:41:43,164 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:48,165 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:52,326 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:41:53,326 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:56,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:41:56,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:41:56,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:41:59,228 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:04,229 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:09,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:11,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:42:11,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:42:11,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:42:15,199 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:20,201 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:22,331 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:42:25,332 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:26,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:42:26,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:42:26,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:42:31,253 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:36,254 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:37,062 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:42:37,063 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:42:37,063 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:42:37,064 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:42:37,969 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:42:41,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:42:41,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:42:41,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:42:42,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:47,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:52,193 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:52,327 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:42:56,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:42:56,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:42:56,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:42:58,177 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:03,178 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:08,179 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:11,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:43:11,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:43:11,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:43:14,151 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:19,152 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:22,329 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:43:24,330 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:26,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:43:26,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:43:26,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:43:30,138 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:35,139 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:40,140 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:41,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:43:41,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:43:41,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:43:45,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:50,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:52,329 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:43:55,330 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:56,080 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:43:56,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:43:56,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:44:01,228 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:06,229 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:06,928 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:44:06,930 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:44:06,930 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:44:06,930 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:44:06,996 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:44:11,080 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:44:11,080 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:44:11,081 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:44:12,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:17,168 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:22,169 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:22,329 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:44:26,080 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:44:26,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:44:26,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:44:27,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:32,214 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:37,215 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:41,080 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:44:41,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:44:41,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:44:43,190 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:48,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:52,330 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:44:53,331 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:56,080 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:44:56,081 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:44:56,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:44:59,250 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:04,251 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:09,252 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:11,080 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:45:11,081 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:45:11,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:45:15,206 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:20,207 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:22,331 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:45:25,332 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:26,081 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:45:26,081 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:45:26,123 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:45:31,214 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:36,215 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:38,586 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:45:38,587 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:45:38,587 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:45:38,588 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:45:39,026 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:45:41,081 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:45:41,081 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:45:41,082 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:45:42,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:47,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:52,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:45:52,331 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:45:56,081 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:45:56,081 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:45:56,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:45:58,141 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:03,142 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:08,143 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:11,081 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:46:11,082 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:46:11,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:46:13,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:18,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:22,332 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:46:23,333 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:26,081 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:46:26,082 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:46:26,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:46:29,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:34,149 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:39,150 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:41,081 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:46:41,082 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:46:41,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:46:44,182 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:49,183 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:52,333 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:46:54,334 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:46:56,082 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:46:56,082 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:46:56,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:47:00,183 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:05,184 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:10,185 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:11,082 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:47:11,082 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:47:11,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:47:11,307 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:47:11,308 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:47:11,308 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:47:11,309 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:47:12,056 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:47:15,311 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:20,312 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:22,333 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:47:25,334 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:26,082 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:47:26,083 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:47:26,083 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:47:31,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:36,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:41,082 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:47:41,082 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:47:41,123 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:47:42,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:47,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:52,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:47:52,334 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:47:56,082 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:47:56,082 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:47:56,123 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:47:57,347 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:02,349 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:07,350 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:11,082 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:48:11,083 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:48:11,123 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:48:13,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:18,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:22,335 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:48:23,335 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:26,082 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:48:26,083 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:48:26,123 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:48:29,165 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:34,166 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:39,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:39,870 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:48:39,871 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:48:39,871 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:48:39,871 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:48:40,085 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:48:41,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:48:41,114 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:48:41,191 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:48:44,236 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:49,237 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:52,335 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:48:54,336 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:48:56,095 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:48:56,095 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:48:56,136 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:49:00,243 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:05,244 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:10,245 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:11,095 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:49:11,095 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:49:11,135 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:49:16,168 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:21,169 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:22,336 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:49:26,095 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:49:26,095 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:49:26,136 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:49:26,213 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:31,214 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:36,214 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:41,095 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:49:41,095 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:49:41,135 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:49:42,203 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:47,204 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:52,205 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:49:52,337 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:49:56,095 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:49:56,095 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:49:56,135 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:49:57,226 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:02,226 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:07,227 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:10,262 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:50:10,263 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:50:10,264 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:50:10,264 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:50:11,114 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:50:11,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:50:11,178 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:50:11,179 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:50:12,350 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:17,351 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:22,338 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:50:23,338 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:26,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:50:26,108 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:50:26,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:50:29,260 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:34,261 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:39,262 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:41,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:50:41,108 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:50:41,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:50:44,272 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:49,273 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:52,338 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:50:54,339 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:50:56,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:50:56,108 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:50:56,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:51:00,226 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:05,227 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:10,228 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:11,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:51:11,108 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:51:11,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:51:15,252 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:20,253 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:22,339 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:51:25,340 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:26,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:51:26,108 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:51:26,149 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:51:31,255 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:36,256 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:40,289 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:51:40,290 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:51:40,291 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:51:40,291 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:51:41,128 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:51:41,129 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:51:41,143 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:51:41,169 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:51:41,292 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:46,293 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:51,294 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:51:52,340 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:51:56,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:51:56,115 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:51:56,156 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:51:57,233 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:02,234 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:07,235 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:11,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:52:11,115 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:52:11,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:52:13,229 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:18,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:22,340 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:52:23,341 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:26,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:52:26,116 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:52:26,156 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:52:29,247 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:34,248 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:39,249 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:41,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:52:41,116 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:52:41,156 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:52:45,211 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:50,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:52,341 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:52:55,342 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:52:56,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:52:56,116 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:52:56,156 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:53:01,283 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:06,284 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:11,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:53:11,116 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:53:11,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:53:12,186 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:14,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:53:14,066 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:53:14,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:53:14,067 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:53:14,173 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:53:18,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:22,342 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:53:23,342 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:26,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:53:26,116 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:53:26,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:53:29,173 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:34,173 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:39,175 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:41,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:53:41,116 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:53:41,157 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:53:44,185 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:49,187 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:52,342 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:53:54,343 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:53:56,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:53:56,116 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:53:56,157 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:54:00,239 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:05,240 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:10,240 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:11,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:54:11,116 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:54:11,156 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:54:16,240 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:21,240 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:22,343 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:54:26,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:54:26,116 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:54:26,157 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:54:27,226 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:32,226 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:37,227 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:41,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:54:41,116 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:54:41,156 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:54:42,237 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:43,652 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:54:43,653 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:54:43,654 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:54:43,654 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:54:44,201 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:54:47,655 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:52,344 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:54:53,344 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:54:56,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:54:56,117 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:54:56,157 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:54:59,190 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:04,190 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:09,191 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:11,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:55:11,117 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:55:11,157 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:55:14,276 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:19,277 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:22,344 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:55:24,345 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:26,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:55:26,117 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:55:26,157 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:55:30,239 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:35,240 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:40,241 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:41,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:55:41,117 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:55:41,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:55:46,234 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:51,234 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:55:52,345 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:55:56,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:55:56,117 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:55:56,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:55:56,262 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:01,263 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:06,264 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:11,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:56:11,117 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:56:11,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:56:12,194 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:13,134 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:56:13,135 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:56:13,136 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:56:13,136 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:56:13,230 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:56:18,137 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:22,346 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:56:23,346 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:26,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:56:26,117 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:56:26,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:56:29,185 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:34,186 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:39,187 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:41,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:56:41,118 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:56:41,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:56:44,197 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:49,198 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:52,346 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:56:54,347 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:56:56,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:56:56,118 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:56:56,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:57:00,221 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:05,222 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:10,223 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:11,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:57:11,118 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:57:11,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:57:16,205 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:21,205 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:22,347 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:57:26,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:57:26,118 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:57:26,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:57:26,263 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:31,264 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:36,265 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:41,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:57:41,118 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:57:41,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:57:42,231 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:44,993 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:57:44,994 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:57:44,994 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:57:44,995 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:57:45,259 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:57:47,996 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:52,348 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:57:53,348 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:57:56,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:57:56,118 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:57:56,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:57:59,193 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:04,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:09,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:11,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:58:11,118 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:58:11,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:58:14,245 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:19,246 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:22,348 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:58:24,349 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:26,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:58:26,118 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:58:26,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:58:30,227 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:35,228 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:40,229 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:41,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:58:41,118 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:58:41,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:58:45,268 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:50,268 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:52,349 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:58:55,350 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:58:56,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:58:56,119 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:58:56,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:59:00,881 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:05,881 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:10,882 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:11,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:59:11,119 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:59:11,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:59:14,130 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:59:14,132 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:59:14,132 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:59:14,132 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:59:14,287 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:59:16,133 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:21,134 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:22,349 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:59:26,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:59:26,119 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:59:26,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:59:26,294 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:31,295 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:36,296 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:41,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:59:41,119 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:59:41,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:59:42,243 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:47,244 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:52,244 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:59:52,350 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:59:56,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:59:56,119 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:59:56,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:59:57,245 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:02,246 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:07,247 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:11,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:00:11,119 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:00:11,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:00:12,282 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:17,283 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:22,283 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:22,351 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:00:26,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:00:26,119 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:00:26,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:00:28,231 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:33,231 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:38,232 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:41,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:00:41,119 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:00:41,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:00:43,341 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:46,486 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 12:00:46,487 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 12:00:46,488 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 12:00:46,488 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 12:00:47,316 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 12:00:48,489 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:52,351 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:00:54,352 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:00:56,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:00:56,120 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:00:56,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:01:00,340 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:05,341 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:10,341 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:11,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:01:11,120 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:01:11,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:01:16,311 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:21,312 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:22,352 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:01:26,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:01:26,120 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:01:26,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:01:27,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:32,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:37,232 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:41,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:01:41,120 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:01:41,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:01:42,297 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:47,298 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:52,298 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:01:52,357 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:01:56,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:01:56,120 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:01:56,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:01:58,210 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:03,210 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:08,211 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:11,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:02:11,120 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:02:11,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:02:13,242 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:16,992 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 12:02:16,993 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 12:02:16,993 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 12:02:16,994 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 12:02:17,343 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 12:02:18,995 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:22,354 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:02:24,354 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:26,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:02:26,120 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:02:26,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:02:30,241 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:35,242 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:40,242 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:41,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:02:41,120 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:02:41,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:02:45,303 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:50,304 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:52,354 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:02:55,355 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:02:56,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:02:56,121 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:02:56,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:03:01,282 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:06,283 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:11,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:03:11,121 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:03:11,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:03:11,597 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:16,598 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:21,598 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:22,355 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:03:26,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:03:26,121 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:03:26,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:03:27,251 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:32,252 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:37,252 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:41,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:03:41,121 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:03:41,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:03:42,350 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:46,970 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 12:03:46,971 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 12:03:46,971 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 12:03:46,972 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 12:03:47,374 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 12:03:47,972 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:52,355 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:03:53,356 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:03:56,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:03:56,121 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:03:56,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:03:59,248 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:04,248 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:09,249 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:11,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:04:11,121 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:04:11,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:04:14,873 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:19,874 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:22,356 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:04:25,357 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:26,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:04:26,122 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:04:26,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:04:31,311 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:36,312 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:41,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:04:41,122 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:04:41,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:04:42,224 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:47,225 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:52,226 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:04:52,357 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:04:56,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:04:56,122 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:04:56,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:04:57,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:02,231 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:07,232 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:11,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:05:11,122 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:05:11,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:05:12,271 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:17,271 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:17,487 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 12:05:17,488 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 12:05:17,488 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 12:05:17,489 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 12:05:18,403 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 12:05:22,357 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:05:22,358 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:26,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:05:26,122 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:05:26,163 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:05:28,258 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:33,258 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:38,259 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:41,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:05:41,122 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:05:41,163 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:05:44,215 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:49,216 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:52,358 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:05:54,359 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:05:56,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:05:56,122 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:05:56,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:06:00,275 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:05,276 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:10,277 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:11,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:06:11,122 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:06:11,163 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:06:16,275 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:21,276 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:22,359 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:06:26,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:06:26,122 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:06:26,163 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:06:27,275 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:32,276 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:37,277 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:41,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:06:41,123 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:06:41,163 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 12:06:42,295 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:46,399 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 12:06:46,400 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 12:06:46,400 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 12:06:46,400 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 12:06:46,432 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 12:06:47,401 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:50,500 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 12:06:50,501 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 12:06:50,501 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 12:06:50,502 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 12:06:51,434 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 12:06:52,359 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 12:06:53,360 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 12:06:56,122 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 12:06:56,123 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 12:06:56,163 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |