| 2024-03-13 11:20:23,382 INFO StreamThr :5135 [internal.py:wandb_internal():86] W&B internal server running at pid: 5135, started at: 2024-03-13 11:20:23.382058 | |
| 2024-03-13 11:20:23,384 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status | |
| 2024-03-13 11:20:25,307 INFO WriterThread:5135 [datastore.py:open_for_write():87] open: /content/wandb/run-20240313_112023-t8o614bo/run-t8o614bo.wandb | |
| 2024-03-13 11:20:25,307 DEBUG SenderThread:5135 [sender.py:send():378] send: header | |
| 2024-03-13 11:20:25,309 DEBUG SenderThread:5135 [sender.py:send():378] send: run | |
| 2024-03-13 11:20:25,535 INFO SenderThread:5135 [dir_watcher.py:__init__():211] watching files in: /content/wandb/run-20240313_112023-t8o614bo/files | |
| 2024-03-13 11:20:25,535 INFO SenderThread:5135 [sender.py:_start_run_threads():1099] run started: t8o614bo with start time 1710328823.382009 | |
| 2024-03-13 11:20:25,540 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-03-13 11:20:25,540 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: check_version | |
| 2024-03-13 11:20:25,679 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-03-13 11:20:25,688 DEBUG HandlerThread:5135 [system_info.py:__init__():26] System info init | |
| 2024-03-13 11:20:25,688 DEBUG HandlerThread:5135 [system_info.py:__init__():41] System info init done | |
| 2024-03-13 11:20:25,688 INFO HandlerThread:5135 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-13 11:20:25,689 INFO SystemMonitor:5135 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-13 11:20:25,689 INFO HandlerThread:5135 [system_monitor.py:probe():214] Collecting system info | |
| 2024-03-13 11:20:25,689 INFO SystemMonitor:5135 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-13 11:20:25,690 INFO SystemMonitor:5135 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-13 11:20:25,691 INFO SystemMonitor:5135 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-13 11:20:25,692 INFO SystemMonitor:5135 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-13 11:20:25,693 INFO SystemMonitor:5135 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-13 11:20:25,703 DEBUG HandlerThread:5135 [system_info.py:probe():150] Probing system | |
| 2024-03-13 11:20:25,704 DEBUG HandlerThread:5135 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-03-13 11:20:25,704 DEBUG HandlerThread:5135 [system_info.py:probe():198] Probing system done | |
| 2024-03-13 11:20:25,704 DEBUG HandlerThread:5135 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-13T11:20:25.703508', 'startedAt': '2024-03-13T11:20:23.376192', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Falcon-FineTuning.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1kv2VO7q0c8xX_DQt5NERLeJJZtHW6zHO', 'host': '7acbe5a74b63', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 2299.998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 201.22513961791992, 'used': 40.209312438964844}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 50.993690490722656}} | |
| 2024-03-13 11:20:25,704 INFO HandlerThread:5135 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-03-13 11:20:25,704 INFO HandlerThread:5135 [system_monitor.py:probe():227] Publishing system info | |
| 2024-03-13 11:20:25,705 INFO HandlerThread:5135 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-03-13 11:20:25,709 DEBUG SenderThread:5135 [sender.py:send():378] send: files | |
| 2024-03-13 11:20:25,709 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now | |
| 2024-03-13 11:20:26,064 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-03-13 11:20:26,064 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: python_packages | |
| 2024-03-13 11:20:26,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:20:26,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:20:26,107 DEBUG SenderThread:5135 [sender.py:send():378] send: telemetry | |
| 2024-03-13 11:20:26,108 DEBUG SenderThread:5135 [sender.py:send():378] send: config | |
| 2024-03-13 11:20:26,109 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:20:26,109 DEBUG SenderThread:5135 [sender.py:send():378] send: telemetry | |
| 2024-03-13 11:20:26,109 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:20:26,109 WARNING SenderThread:5135 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-03-13 11:20:26,109 DEBUG SenderThread:5135 [sender.py:send():378] send: telemetry | |
| 2024-03-13 11:20:26,110 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:20:26,133 INFO wandb-upload_0:5135 [upload_job.py:push():131] Uploaded file /tmp/tmpuwvyq0ztwandb/v6gi54r6-wandb-metadata.json | |
| 2024-03-13 11:20:26,537 INFO Thread-12 :5135 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240313_112023-t8o614bo/files/requirements.txt | |
| 2024-03-13 11:20:26,537 INFO Thread-12 :5135 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-metadata.json | |
| 2024-03-13 11:20:29,253 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:34,254 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:39,254 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:41,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:20:41,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:20:41,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:20:45,172 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:50,173 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:55,178 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:20:55,544 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/config.yaml | |
| 2024-03-13 11:20:56,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:20:56,065 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:20:56,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:21:01,127 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:06,128 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:11,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:21:11,065 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:21:11,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:21:11,189 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:16,190 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:21,191 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:25,693 DEBUG SystemMonitor:5135 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-13 11:21:25,694 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:21:26,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:21:26,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:21:26,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:21:26,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:31,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:36,197 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:41,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:21:41,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:21:41,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:21:41,199 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:46,200 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:47,552 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:21:47,554 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:21:47,554 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:21:47,554 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:21:47,554 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:21:47,555 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:21:47,555 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:21:47,556 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:21:47,560 INFO Thread-12 :5135 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:21:51,558 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:55,695 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:21:56,065 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:21:56,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:21:56,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:21:57,208 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:21:57,563 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/config.yaml | |
| 2024-03-13 11:22:02,349 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:07,350 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:11,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:22:11,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:22:11,106 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:22:13,224 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:18,225 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:23,226 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:25,695 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:22:26,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:22:26,066 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:22:26,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:22:29,191 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:34,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:39,193 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:41,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:22:41,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:22:41,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:22:45,190 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:50,191 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:55,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:22:55,696 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:22:56,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:22:56,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:22:56,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:23:00,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:05,231 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:10,232 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:11,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:23:11,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:23:11,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:23:16,001 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:23:16,002 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:23:16,003 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:23:16,004 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:16,005 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:23:16,588 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:23:21,006 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:25,697 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:23:26,066 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:23:26,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:23:26,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:23:26,173 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:31,174 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:36,175 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:41,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:23:41,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:23:41,107 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:23:41,207 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:46,208 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:51,209 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:23:55,698 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:23:56,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:23:56,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:23:56,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:23:57,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:02,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:07,163 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:11,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:24:11,067 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:24:11,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:24:12,169 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:17,170 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:22,171 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:25,699 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:24:26,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:24:26,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:24:26,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:24:27,228 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:32,229 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:37,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:41,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:24:41,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:24:41,108 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:24:43,204 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:44,484 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:24:44,485 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:24:44,485 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:24:44,487 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:24:44,617 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:24:48,489 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:48,768 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:24:48,770 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:24:48,770 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:24:48,770 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:24:48,771 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:24:48,771 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:24:48,771 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:24:48,771 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:24:48,771 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:24:49,618 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:24:53,773 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:24:55,700 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:24:56,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:24:56,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:24:56,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:24:58,385 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: pause | |
| 2024-03-13 11:24:58,385 INFO HandlerThread:5135 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-03-13 11:24:58,385 INFO HandlerThread:5135 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-03-13 11:24:58,386 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-03-13 11:24:58,386 DEBUG SystemMonitor:5135 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-13 11:24:58,386 DEBUG SystemMonitor:5135 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-13 11:24:58,387 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-03-13 11:24:58,393 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-03-13 11:24:58,393 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-03-13 11:24:58,393 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined network monitor | |
| 2024-03-13 11:24:58,394 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:24:59,395 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:04,400 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:08,625 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/config.yaml | |
| 2024-03-13 11:25:11,067 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:25:11,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:25:11,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:25:13,150 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:18,151 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:23,153 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:26,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:25:26,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:25:26,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:25:28,172 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:33,173 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:38,174 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:41,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:25:41,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:25:41,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:25:43,194 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:48,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:53,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:25:56,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:25:56,068 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:25:56,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:25:58,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:03,213 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:08,214 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:11,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:26:11,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:26:11,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:26:14,163 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:19,164 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:24,165 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:26,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:26:26,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:26:26,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:26:30,139 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:35,140 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:40,141 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:41,068 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:26:41,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:26:41,109 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:26:45,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:50,168 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:55,169 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:26:56,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:26:56,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:26:56,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:27:01,153 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:06,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:11,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:27:11,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:27:11,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:27:11,157 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:16,158 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:21,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:26,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:27:26,069 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:27:26,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:27:27,154 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:32,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:37,156 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:41,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:27:41,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:27:41,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:27:42,179 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:47,180 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:52,181 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:27:56,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:27:56,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:27:56,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:27:57,186 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:02,187 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:07,188 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:11,069 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:28:11,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:28:11,110 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:28:12,199 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:17,200 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:22,201 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:26,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:28:26,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:28:26,111 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:28:27,247 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:32,248 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:37,249 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:41,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:28:41,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:28:41,112 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:28:43,229 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:48,231 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:53,232 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:28:56,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:28:56,070 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:28:56,111 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:28:59,146 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:04,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:09,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:11,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:29:11,071 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:29:11,111 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:29:15,127 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:20,128 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:25,129 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:26,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:29:26,071 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:29:26,111 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:29:30,141 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:35,142 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:40,143 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:41,070 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:29:41,071 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:29:41,111 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:29:45,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:50,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:55,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:29:56,071 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:29:56,071 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:29:56,112 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:30:00,231 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:05,232 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:10,233 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:11,071 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:30:11,071 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:30:11,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:30:16,222 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:21,223 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:26,071 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:30:26,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:30:26,112 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:30:27,194 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:32,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:37,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:41,071 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:30:41,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:30:41,112 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:30:43,166 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:48,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:53,168 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:30:56,071 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:30:56,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:30:56,112 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:30:58,211 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:03,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:08,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:11,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:31:11,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:31:11,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:31:14,166 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:19,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:24,168 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:26,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:31:26,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:31:26,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:31:29,181 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:34,182 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:39,183 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:41,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:31:41,072 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:31:41,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:31:44,220 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:49,221 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:54,222 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:31:56,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:31:56,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:31:56,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:31:59,233 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:04,234 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:09,235 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:11,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:32:11,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:32:11,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:32:15,235 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:20,236 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:25,237 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:26,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:32:26,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:32:26,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:32:31,182 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:36,182 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:41,072 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:32:41,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:32:41,113 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:32:41,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:46,213 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:51,214 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:32:56,073 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:32:56,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:32:56,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:32:57,152 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:02,154 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:07,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:11,073 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:33:11,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:33:11,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:33:12,216 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:17,218 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:22,218 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:26,073 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:33:26,073 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:33:26,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:33:28,210 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:33,211 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:38,212 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:41,073 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:33:41,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:33:41,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:33:43,222 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:48,223 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:53,224 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:33:56,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:33:56,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:33:56,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:33:59,169 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:04,170 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:09,171 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:11,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:34:11,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:34:11,114 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:34:14,190 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:19,191 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:24,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:26,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:34:26,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:34:26,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:34:30,150 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:35,151 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:40,152 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:41,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:34:41,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:34:41,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:34:45,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:50,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:55,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:34:56,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:34:56,075 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:34:56,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:35:00,225 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:05,226 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:10,227 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:11,074 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:35:11,074 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:35:11,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:35:16,219 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:21,220 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:26,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:35:26,075 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:35:26,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:35:27,207 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:32,209 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:37,210 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:41,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:35:41,075 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:35:41,115 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:35:42,234 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:47,235 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:52,236 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:35:56,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:35:56,075 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:35:56,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:35:58,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:03,156 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:08,157 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:11,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:36:11,075 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:36:11,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:36:13,183 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:18,184 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:23,185 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:26,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:36:26,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:36:26,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:36:28,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:33,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:38,197 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:41,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:36:41,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:36:41,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:36:43,357 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:48,358 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:50,278 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: resume | |
| 2024-03-13 11:36:50,278 INFO HandlerThread:5135 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-03-13 11:36:50,278 INFO HandlerThread:5135 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-13 11:36:50,279 INFO SystemMonitor:5135 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-13 11:36:50,279 INFO SystemMonitor:5135 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-13 11:36:50,280 INFO SystemMonitor:5135 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-13 11:36:50,280 INFO SystemMonitor:5135 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-13 11:36:50,281 INFO SystemMonitor:5135 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-13 11:36:50,281 INFO SystemMonitor:5135 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-13 11:36:50,318 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: pause | |
| 2024-03-13 11:36:50,319 INFO HandlerThread:5135 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-03-13 11:36:50,319 INFO HandlerThread:5135 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-03-13 11:36:50,319 DEBUG SystemMonitor:5135 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-13 11:36:50,319 DEBUG SystemMonitor:5135 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-13 11:36:50,319 DEBUG SystemMonitor:5135 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-13 11:36:50,319 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-03-13 11:36:50,320 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-03-13 11:36:50,325 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-03-13 11:36:50,325 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-03-13 11:36:50,325 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined network monitor | |
| 2024-03-13 11:36:50,326 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:36:50,728 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: resume | |
| 2024-03-13 11:36:50,728 INFO HandlerThread:5135 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-03-13 11:36:50,728 INFO HandlerThread:5135 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-13 11:36:50,728 INFO SystemMonitor:5135 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-13 11:36:50,729 INFO SystemMonitor:5135 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-13 11:36:50,729 INFO SystemMonitor:5135 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-13 11:36:50,730 INFO SystemMonitor:5135 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-13 11:36:50,731 INFO SystemMonitor:5135 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-13 11:36:50,731 INFO SystemMonitor:5135 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-13 11:36:51,012 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: pause | |
| 2024-03-13 11:36:51,012 INFO HandlerThread:5135 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-03-13 11:36:51,012 INFO HandlerThread:5135 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-03-13 11:36:51,012 DEBUG SystemMonitor:5135 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-13 11:36:51,012 DEBUG SystemMonitor:5135 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-13 11:36:51,012 DEBUG SystemMonitor:5135 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-13 11:36:51,013 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-03-13 11:36:51,013 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-03-13 11:36:51,019 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-03-13 11:36:51,019 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-03-13 11:36:51,019 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined network monitor | |
| 2024-03-13 11:36:51,020 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:36:51,210 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: resume | |
| 2024-03-13 11:36:51,210 INFO HandlerThread:5135 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-03-13 11:36:51,210 INFO HandlerThread:5135 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-13 11:36:51,210 INFO SystemMonitor:5135 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-13 11:36:51,211 INFO SystemMonitor:5135 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-13 11:36:51,211 INFO SystemMonitor:5135 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-13 11:36:51,212 INFO SystemMonitor:5135 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-13 11:36:51,213 INFO SystemMonitor:5135 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-13 11:36:51,213 INFO SystemMonitor:5135 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-13 11:36:51,216 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: pause | |
| 2024-03-13 11:36:51,216 INFO HandlerThread:5135 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-03-13 11:36:51,216 INFO HandlerThread:5135 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-03-13 11:36:51,216 DEBUG SystemMonitor:5135 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-13 11:36:51,216 DEBUG SystemMonitor:5135 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-13 11:36:51,216 DEBUG SystemMonitor:5135 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-13 11:36:51,216 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-03-13 11:36:51,217 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-03-13 11:36:51,225 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-03-13 11:36:51,225 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-03-13 11:36:51,225 INFO HandlerThread:5135 [interfaces.py:finish():202] Joined network monitor | |
| 2024-03-13 11:36:51,226 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:36:52,316 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: resume | |
| 2024-03-13 11:36:52,316 INFO HandlerThread:5135 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-03-13 11:36:52,316 INFO HandlerThread:5135 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-13 11:36:52,317 INFO SystemMonitor:5135 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-13 11:36:52,317 INFO SystemMonitor:5135 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-13 11:36:52,318 INFO SystemMonitor:5135 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-13 11:36:52,318 INFO SystemMonitor:5135 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-13 11:36:52,318 INFO SystemMonitor:5135 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-13 11:36:52,319 INFO SystemMonitor:5135 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-13 11:36:52,779 DEBUG SenderThread:5135 [sender.py:send():378] send: config | |
| 2024-03-13 11:36:52,781 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:36:52,781 DEBUG SenderThread:5135 [sender.py:send():378] send: metric | |
| 2024-03-13 11:36:52,781 WARNING SenderThread:5135 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-03-13 11:36:53,782 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:56,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:36:56,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:36:56,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:36:59,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:36:59,858 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/config.yaml | |
| 2024-03-13 11:37:04,288 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:09,289 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:11,075 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:37:11,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:37:11,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:37:15,201 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:20,202 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:25,203 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:26,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:37:26,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:37:26,116 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:37:30,214 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:35,215 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:40,216 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:41,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:37:41,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:37:41,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:37:46,194 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:51,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:37:52,319 DEBUG SystemMonitor:5135 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-13 11:37:52,320 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:37:56,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:37:56,076 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:37:56,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:37:56,199 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:01,199 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:06,200 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:11,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:38:11,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:38:11,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:38:11,201 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:13,476 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:38:13,478 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:38:13,478 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:38:13,479 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:38:13,881 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:38:13,882 INFO Thread-12 :5135 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240313_112023-t8o614bo/files/output.log | |
| 2024-03-13 11:38:15,882 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/output.log | |
| 2024-03-13 11:38:16,518 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:21,519 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:22,321 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:38:26,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:38:26,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:38:26,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:38:27,136 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:32,138 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:37,138 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:41,076 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:38:41,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:38:41,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:38:42,144 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:47,145 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:52,146 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:38:52,322 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:38:56,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:38:56,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:38:56,117 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:38:57,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:02,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:07,149 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:11,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:39:11,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:39:11,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:39:12,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:17,161 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:22,162 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:22,322 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:39:26,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:39:26,077 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:39:26,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:39:28,146 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:33,147 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:38,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:41,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:39:41,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:39:41,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:39:41,592 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:39:41,594 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:39:41,594 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:39:41,595 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:39:41,911 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:39:43,633 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:43,911 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/output.log | |
| 2024-03-13 11:39:48,634 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:52,323 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:39:54,324 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:39:56,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:39:56,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:39:56,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:40:00,195 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:05,196 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:10,197 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:11,077 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:40:11,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:40:11,118 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:40:16,154 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:21,155 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:22,323 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:40:26,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:40:26,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:40:26,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:40:26,175 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:31,175 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:36,176 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:41,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:40:41,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:40:41,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:40:41,208 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:46,209 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:51,209 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:40:52,324 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:40:56,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:40:56,078 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:40:56,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:40:57,148 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:02,149 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:07,150 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:08,398 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:41:08,399 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:41:08,399 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:41:08,399 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:41:08,939 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:41:09,939 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/output.log | |
| 2024-03-13 11:41:11,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:41:11,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:41:11,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:41:12,150 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:17,151 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:22,152 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:22,325 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:41:26,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:41:26,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:41:26,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:41:27,209 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:32,210 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:37,211 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:41,078 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:41:41,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:41:41,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:41:43,164 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:48,165 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:52,326 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:41:53,326 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:41:56,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:41:56,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:41:56,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:41:59,228 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:04,229 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:09,230 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:11,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:42:11,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:42:11,119 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:42:15,199 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:20,201 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:22,331 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:42:25,332 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:26,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:42:26,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:42:26,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:42:31,253 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:36,254 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:37,062 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:42:37,063 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:42:37,063 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:42:37,064 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:42:37,969 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:42:41,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:42:41,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:42:41,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:42:42,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:47,192 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:52,193 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:42:52,327 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:42:56,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:42:56,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:42:56,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:42:58,177 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:03,178 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:08,179 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:11,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:43:11,079 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:43:11,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:43:14,151 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:19,152 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:22,329 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:43:24,330 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:26,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:43:26,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:43:26,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:43:30,138 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:35,139 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:40,140 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:41,079 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:43:41,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:43:41,120 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:43:45,159 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:50,160 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:52,329 DEBUG SenderThread:5135 [sender.py:send():378] send: stats | |
| 2024-03-13 11:43:55,330 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:43:56,080 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:43:56,080 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:43:56,121 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:44:01,228 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:06,229 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-13 11:44:06,928 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-13 11:44:06,930 DEBUG SenderThread:5135 [sender.py:send():378] send: history | |
| 2024-03-13 11:44:06,930 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-13 11:44:06,930 INFO SenderThread:5135 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-13 11:44:06,996 INFO Thread-12 :5135 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240313_112023-t8o614bo/files/wandb-summary.json | |
| 2024-03-13 11:44:11,080 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-13 11:44:11,080 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-13 11:44:11,081 DEBUG SenderThread:5135 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-13 11:44:12,167 DEBUG HandlerThread:5135 [handler.py:handle_request():146] handle_request: status_report | |