ogpotereplicantv3/20230623072223/wandb/debug-internal.log

2023-06-23 07:23:33,432 INFO    StreamThr :12261 [internal.py:wandb_internal():86] W&B internal server running at pid: 12261, started at: 2023-06-23 07:23:33.425995
2023-06-23 07:23:33,437 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status
2023-06-23 07:23:33,442 INFO    WriterThread:12261 [datastore.py:open_for_write():85] open: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/run-hni6fhql.wandb
2023-06-23 07:23:33,447 DEBUG   SenderThread:12261 [sender.py:send():375] send: header
2023-06-23 07:23:33,529 DEBUG   SenderThread:12261 [sender.py:send():375] send: run
2023-06-23 07:23:34,147 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: check_version
2023-06-23 07:23:34,156 INFO    SenderThread:12261 [dir_watcher.py:__init__():219] watching files in: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files
2023-06-23 07:23:34,157 INFO    SenderThread:12261 [sender.py:_start_run_threads():1124] run started: hni6fhql with start time 1687505013.434268
2023-06-23 07:23:34,157 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:23:34,163 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:23:34,165 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: check_version
2023-06-23 07:23:34,232 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: run_start
2023-06-23 07:23:34,242 DEBUG   HandlerThread:12261 [system_info.py:__init__():31] System info init
2023-06-23 07:23:34,242 DEBUG   HandlerThread:12261 [system_info.py:__init__():46] System info init done
2023-06-23 07:23:34,243 INFO    HandlerThread:12261 [system_monitor.py:start():181] Starting system monitor
2023-06-23 07:23:34,243 INFO    SystemMonitor:12261 [system_monitor.py:_start():145] Starting system asset monitoring threads
2023-06-23 07:23:34,243 INFO    HandlerThread:12261 [system_monitor.py:probe():201] Collecting system info
2023-06-23 07:23:34,245 INFO    SystemMonitor:12261 [interfaces.py:start():190] Started cpu monitoring
2023-06-23 07:23:34,246 INFO    SystemMonitor:12261 [interfaces.py:start():190] Started disk monitoring
2023-06-23 07:23:34,247 INFO    SystemMonitor:12261 [interfaces.py:start():190] Started gpu monitoring
2023-06-23 07:23:34,248 INFO    SystemMonitor:12261 [interfaces.py:start():190] Started memory monitoring
2023-06-23 07:23:34,249 INFO    SystemMonitor:12261 [interfaces.py:start():190] Started network monitoring
2023-06-23 07:23:34,250 DEBUG   HandlerThread:12261 [system_info.py:probe():195] Probing system
2023-06-23 07:23:34,269 DEBUG   HandlerThread:12261 [system_info.py:_probe_git():180] Probing git
2023-06-23 07:23:34,319 DEBUG   HandlerThread:12261 [system_info.py:_probe_git():188] Probing git done
2023-06-23 07:23:34,320 DEBUG   HandlerThread:12261 [system_info.py:probe():240] Probing system done
2023-06-23 07:23:34,320 DEBUG   HandlerThread:12261 [system_monitor.py:probe():210] {'os': 'Linux-5.15.107+-x86_64-with-glibc2.31', 'python': '3.10.12', 'heartbeatAt': '2023-06-23T07:23:34.251530', 'startedAt': '2023-06-23T07:23:33.337714', 'docker': None, 'cuda': None, 'args': ('--v2', '--v_parameterization', '--enable_bucket', '--pretrained_model_name_or_path=/content/gdrive/MyDrive/Replicant-V3.0_fp16.safetensors', '--train_data_dir=/content/gdrive/MyDrive/ogipotereplicantv3', '--resolution=768,768', '--output_dir=/content/gdrive/MyDrive/ogipotereplicantv3', '--logging_dir=/content/gdrive/MyDrive/ogipotereplicantv3', '--network_alpha=8', '--save_model_as=safetensors', '--network_module=networks.lora', '--text_encoder_lr=3.5e-06', '--unet_lr=3.5e-05', '--network_dim=16', '--output_name=ogipotereplicant_16_8', '--lr_scheduler_num_cycles=10', '--learning_rate=0.0001', '--lr_scheduler=cosine', '--train_batch_size=2', '--max_train_steps=3500', '--save_every_n_epochs=1', '--mixed_precision=fp16', '--save_precision=fp16', '--seed=210432', '--caption_extension=.txt', '--optimizer_type=AdamW8bit', '--max_data_loader_n_workers=0', '--max_token_length=225', '--clip_skip=2', '--keep_tokens=1', '--bucket_reso_steps=64', '--min_snr_gamma=1', '--shuffle_caption', '--xformers', '--bucket_no_upscale', '--multires_noise_iterations=6', '--multires_noise_discount=0.03', '--log_with', 'wandb', '--wandb_api_key=c2098a7cde008ba5d6158d1707171c769bf62ccf', '--sample_sampler=euler_a', '--sample_prompts=/content/gdrive/MyDrive/ogipotereplicantv3/sample/prompt.txt', '--sample_every_n_steps=250'), 'state': 'running', 'program': '/content/gdrive/MyDrive/sd/kohya_ss_colab/train_network.py', 'codePath': 'train_network.py', 'git': {'remote': 'https://github.com/YKefasu/kohya_ss_gui_google_colab', 'commit': '80bf7b22765082256ecf61634e7b9ccf548dd7fe'}, 'email': None, 'root': '/content/gdrive/MyDrive/sd/kohya_ss_colab', 'host': 'a618d0d264d4', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2199.998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2199.998, 'min': 0.0, 'max': 0.0}, {'current': 2199.998, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 78.1898422241211, 'used': 38.080936431884766}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.678390502929688}}
2023-06-23 07:23:34,321 INFO    HandlerThread:12261 [system_monitor.py:probe():211] Finished collecting system info
2023-06-23 07:23:34,321 INFO    HandlerThread:12261 [system_monitor.py:probe():214] Publishing system info
2023-06-23 07:23:34,321 DEBUG   HandlerThread:12261 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2023-06-23 07:23:34,325 DEBUG   HandlerThread:12261 [system_info.py:_save_pip():67] Saving pip packages done
2023-06-23 07:23:34,343 INFO    HandlerThread:12261 [system_monitor.py:probe():216] Finished publishing system info
2023-06-23 07:23:34,361 DEBUG   SenderThread:12261 [sender.py:send():375] send: files
2023-06-23 07:23:34,362 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now
2023-06-23 07:23:34,372 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:23:34,373 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:23:34,579 DEBUG   SenderThread:12261 [sender.py:send():375] send: telemetry
2023-06-23 07:23:35,012 INFO    wandb-upload_0:12261 [upload_job.py:push():137] Uploaded file /tmp/tmp5qpv4g7qwandb/e5fylei5-wandb-metadata.json
2023-06-23 07:23:35,161 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_created():278] file/dir created: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-metadata.json
2023-06-23 07:23:35,162 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_created():278] file/dir created: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:23:35,162 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_created():278] file/dir created: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/requirements.txt
2023-06-23 07:23:35,169 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_created():278] file/dir created: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/output.log
2023-06-23 07:23:38,588 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:23:43,589 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:23:48,420 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:23:48,589 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:23:49,372 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:23:49,373 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:23:50,473 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:23:50,475 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:23:50,475 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:23:50,485 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:23:51,198 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:23:52,888 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:23:52,890 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:23:52,893 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:23:52,901 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:23:53,202 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:23:53,903 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:23:55,109 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:23:55,111 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:23:55,111 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:23:55,124 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:23:55,207 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:23:57,546 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:23:57,548 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:23:57,549 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:23:57,559 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:23:58,215 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:23:59,561 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:23:59,873 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:23:59,875 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:23:59,875 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:23:59,880 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:00,219 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:01,725 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:01,727 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:01,728 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:01,736 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:02,223 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:03,438 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:03,439 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:03,440 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:03,450 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:04,229 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:04,373 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:24:04,373 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:24:04,648 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:05,231 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/config.yaml
2023-06-23 07:24:05,396 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:05,397 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:05,398 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:05,404 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:06,233 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:07,113 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:07,114 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:07,115 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:07,119 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:07,235 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:09,077 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:09,078 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:09,079 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:09,083 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:09,239 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:10,085 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:11,518 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:11,519 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:11,520 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:11,525 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:12,248 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:13,466 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:13,467 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:13,467 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:13,472 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:14,252 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:15,474 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:16,187 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:16,189 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:16,191 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:16,203 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:16,263 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:18,615 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:18,625 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:18,626 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:18,631 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:19,272 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:19,372 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:24:19,373 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:24:20,597 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:21,295 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:21,297 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:21,297 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:21,302 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:22,292 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:23,547 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:23,548 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:23,549 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:23,556 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:24,302 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:25,987 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:25,989 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:25,991 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:25,991 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:26,006 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:26,320 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:28,799 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:28,800 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:28,801 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:28,806 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:29,330 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:31,231 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:31,232 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:31,234 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:31,234 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:31,243 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:31,338 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:33,664 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:33,665 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:33,666 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:33,696 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:34,250 DEBUG   SystemMonitor:12261 [system_monitor.py:_start():159] Starting system metrics aggregation loop
2023-06-23 07:24:34,252 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:24:34,351 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:34,372 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:24:34,373 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:24:36,401 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:36,403 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:36,404 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:36,404 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:36,415 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:37,359 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:39,225 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:39,226 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:39,227 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:39,230 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:39,364 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:41,616 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:41,618 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:41,618 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:41,619 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:41,627 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:42,371 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:44,226 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:44,228 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:44,228 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:44,254 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:44,391 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:46,747 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:46,749 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:46,751 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:46,751 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:46,759 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:47,426 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:49,042 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:49,045 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:49,045 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:49,054 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:49,373 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:24:49,374 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:24:49,440 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:51,022 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:51,024 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:51,024 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:51,030 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:51,445 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:52,032 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:53,111 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:53,112 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:53,113 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:53,117 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:53,449 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:55,222 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:55,223 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:55,224 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:55,230 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:55,454 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:24:57,232 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:24:57,543 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:24:57,544 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:24:57,544 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:24:57,551 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:24:58,470 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:00,096 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:00,097 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:00,097 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:00,100 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:00,478 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:02,857 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:02,859 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:02,859 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:02,861 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:02,863 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:03,484 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:04,252 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:25:04,373 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:25:04,374 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:25:05,305 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:05,306 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:05,307 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:05,314 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:05,491 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:07,935 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:07,936 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:07,937 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:07,937 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:07,944 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:08,498 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:10,156 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:10,157 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:10,158 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:10,163 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:10,503 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:12,234 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:12,236 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:12,237 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:12,260 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:12,507 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:13,262 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:14,329 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:14,330 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:14,331 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:14,336 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:14,510 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:16,423 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:16,425 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:16,425 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:16,432 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:16,514 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:18,434 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:18,520 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:18,522 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:18,523 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:18,527 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:19,373 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:25:19,374 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:25:19,521 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:20,514 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:20,516 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:20,517 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:20,520 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:20,523 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:23,084 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:23,085 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:23,085 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:23,089 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:23,531 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:24,090 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:25,370 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:25,372 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:25,372 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:25,383 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:25,538 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:27,675 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:27,677 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:27,678 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:27,691 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:28,546 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:29,693 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:29,846 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:29,847 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:29,849 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:29,854 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:30,551 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:31,958 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:31,960 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:31,960 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:31,966 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:32,555 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:34,069 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:34,071 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:34,072 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:34,078 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:34,253 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:25:34,373 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:25:34,374 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:25:34,559 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:35,589 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:35,910 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:35,911 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:35,912 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:35,916 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:36,564 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:37,799 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:37,801 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:37,802 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:37,808 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:38,568 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:39,814 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:39,815 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:39,818 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:39,822 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:40,573 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:40,824 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:41,789 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:41,790 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:41,791 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:41,796 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:42,577 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:44,099 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:44,101 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:44,102 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:44,105 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:44,593 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:46,107 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:46,383 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:46,385 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:46,385 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:46,388 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:46,599 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:48,493 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:48,495 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:48,495 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:48,502 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:48,604 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:49,374 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:25:49,374 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:25:50,767 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:50,769 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:50,772 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:50,790 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:51,610 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:51,792 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:53,129 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:53,130 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:53,131 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:53,134 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:53,613 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:55,191 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:55,193 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:55,193 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:55,198 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:55,618 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:57,200 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:25:57,314 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:57,316 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:57,317 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:57,322 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:57,623 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:25:59,093 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:25:59,095 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:25:59,095 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:25:59,102 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:25:59,627 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:01,266 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:01,267 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:01,268 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:01,272 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:01,632 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:02,274 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:03,221 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:03,223 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:03,223 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:03,230 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:03,636 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:04,253 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:26:04,374 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:26:04,374 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:26:05,695 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:05,698 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:05,699 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:05,702 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:06,643 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:07,704 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:08,150 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:08,153 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:08,153 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:08,165 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:08,648 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:10,398 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:10,400 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:10,400 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:10,408 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:10,656 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:12,721 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:12,722 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:12,723 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:12,723 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:12,731 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:13,673 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:15,118 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:15,119 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:15,120 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:15,124 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:15,678 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:17,080 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:17,082 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:17,083 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:17,088 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:17,684 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:18,090 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:19,122 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:19,124 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:19,125 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:19,129 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:19,374 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:26:19,374 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:26:19,688 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:21,005 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:21,006 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:21,007 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:21,013 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:21,693 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:23,047 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:23,048 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:23,049 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:23,054 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:23,698 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:24,055 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:25,006 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:25,007 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:25,007 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:25,010 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:25,703 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:27,632 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:27,633 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:27,633 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:27,637 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:27,709 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:29,639 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:29,906 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:29,908 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:29,908 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:29,911 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:30,720 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:32,256 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:32,257 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:32,257 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:32,273 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:32,726 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:34,254 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:26:34,374 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:26:34,375 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:26:34,834 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:34,835 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:34,836 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:34,837 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:34,839 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:35,734 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:37,219 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:37,220 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:37,221 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:37,225 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:37,738 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:39,011 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:39,013 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:39,014 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:39,018 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:39,742 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:40,019 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:40,902 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:40,904 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:40,904 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:40,908 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:41,746 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:42,982 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:42,984 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:42,984 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:42,991 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:43,750 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:45,045 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:45,047 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:45,047 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:45,048 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:45,052 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:45,755 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:47,231 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:47,232 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:47,232 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:47,240 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:47,761 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:49,374 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:26:49,375 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:26:49,546 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:49,586 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:49,586 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:49,590 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:49,765 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:50,592 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:52,133 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:52,134 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:52,135 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:52,141 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:52,773 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:54,627 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:54,629 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:54,630 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:54,636 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:54,779 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:55,638 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:26:56,939 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:56,940 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:56,941 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:56,946 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:57,789 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:26:59,212 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:26:59,214 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:26:59,214 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:26:59,219 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:26:59,794 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:01,214 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:01,216 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:01,217 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:01,217 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:01,221 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:01,798 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:03,245 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:03,248 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:03,249 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:03,253 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:03,802 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:04,255 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:27:04,429 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:27:04,430 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:27:05,031 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:05,033 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:05,034 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:05,041 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:05,806 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:06,988 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:06,990 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:06,991 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:06,992 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:07,000 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:07,810 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:09,255 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:09,257 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:09,257 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:09,260 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:09,817 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:11,792 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:11,793 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:11,794 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:11,816 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:11,831 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:12,820 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:14,180 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:14,181 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:14,182 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:14,188 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:14,839 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:16,665 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:16,666 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:16,667 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:16,675 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:16,843 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:18,679 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:19,100 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:19,102 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:19,103 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:19,110 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:19,429 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:27:19,431 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:27:19,854 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:21,625 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:21,627 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:21,628 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:21,634 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:21,858 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:23,713 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:23,715 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:23,715 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:23,717 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:23,720 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:23,864 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:26,021 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:26,022 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:26,028 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:26,045 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:26,871 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:28,053 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:28,055 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:28,055 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:28,062 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:28,876 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:29,064 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:30,379 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:30,380 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:30,381 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:30,389 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:30,880 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:32,448 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:32,449 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:32,450 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:32,453 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:32,884 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:34,256 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:27:34,257 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:34,429 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:27:34,430 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:27:35,034 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:35,037 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:35,039 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:35,050 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:35,893 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:37,323 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:37,324 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:37,325 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:37,330 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:37,898 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:39,332 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:39,479 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:39,481 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:39,483 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:39,498 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:39,904 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:41,668 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:41,670 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:41,670 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:41,676 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:41,908 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:44,015 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:44,016 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:44,017 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:44,025 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:44,917 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:45,028 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:46,125 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:46,127 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:46,128 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:46,132 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:46,922 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:48,027 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:48,029 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:48,030 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:48,034 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:48,928 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:49,469 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:27:49,470 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:27:50,209 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:50,210 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:50,211 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:50,213 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:50,221 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:50,934 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:52,381 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:52,383 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:52,383 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:52,387 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:52,940 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:54,364 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:54,365 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:54,366 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:54,379 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:54,949 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:55,386 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:27:57,086 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:57,087 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:57,087 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:57,103 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:57,960 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:27:59,371 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:27:59,373 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:27:59,373 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:27:59,380 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:27:59,969 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:01,381 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:01,537 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:01,538 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:01,540 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:01,546 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:01,974 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:03,781 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:03,783 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:03,789 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:03,793 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:03,978 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:04,256 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:28:04,469 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:28:04,470 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:28:06,281 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:06,284 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:06,285 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:06,290 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:06,985 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:07,291 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:08,266 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:08,268 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:08,268 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:08,273 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:08,989 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:10,336 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:10,338 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:10,338 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:10,343 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:10,996 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:12,344 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:12,395 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:12,397 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:12,397 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:12,403 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:13,000 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:14,436 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:14,438 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:14,439 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:14,443 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:15,004 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:16,359 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:16,361 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:16,361 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:16,367 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:17,009 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:17,368 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:18,662 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:18,663 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:18,664 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:18,672 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:19,013 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:19,469 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:28:19,470 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:28:21,099 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:21,100 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:21,101 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:21,104 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:22,019 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:23,106 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:23,448 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:23,449 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:23,450 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:23,458 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:24,044 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:25,562 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:25,563 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:25,564 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:25,570 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:26,052 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:27,947 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:27,949 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:27,949 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:27,961 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:28,056 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:28,963 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:30,360 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:30,361 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:30,362 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:30,368 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:31,062 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:32,747 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:32,749 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:32,749 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:32,754 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:33,065 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:34,257 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:28:34,258 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:34,470 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:28:34,472 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:28:35,099 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:35,101 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:35,101 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:35,104 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:36,072 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:37,415 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:37,416 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:37,417 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:37,423 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:38,078 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:39,429 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:39,597 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:39,598 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:39,599 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:39,604 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:40,086 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:42,017 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:42,019 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:42,020 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:42,023 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:42,090 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:44,357 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:44,359 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:44,359 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:44,372 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:45,099 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:45,382 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:46,726 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:46,727 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:46,727 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:46,732 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:47,103 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:48,799 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:48,801 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:48,801 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:48,806 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:49,113 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:49,472 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:28:49,474 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:28:50,682 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:51,294 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:51,295 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:51,296 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:51,300 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:52,122 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:53,239 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:53,240 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:53,241 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:53,245 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:54,126 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:55,431 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:55,432 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:55,433 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:55,439 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:56,130 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:56,440 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:28:57,344 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:57,346 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:57,346 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:57,350 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:28:58,135 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:28:59,377 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:28:59,379 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:28:59,380 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:28:59,385 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:00,138 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:01,436 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:29:01,437 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:29:01,437 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:29:01,441 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:01,442 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:29:02,143 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:03,530 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:29:03,531 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:29:03,531 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:29:03,534 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:04,152 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:04,258 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:29:04,472 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: stop_status
2023-06-23 07:29:04,473 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: stop_status
2023-06-23 07:29:05,814 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:29:05,815 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:29:05,816 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:29:05,822 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:06,157 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:06,824 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:29:08,112 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:29:08,113 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:29:08,115 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:29:08,120 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:08,162 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:10,591 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:29:10,592 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:29:10,593 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:29:10,599 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:11,167 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:12,600 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:29:12,783 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:29:12,784 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:29:12,785 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:29:12,802 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:13,173 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:14,698 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:29:14,700 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:29:14,700 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:29:14,704 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:15,177 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:16,708 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: partial_history
2023-06-23 07:29:16,710 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:29:16,710 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:29:16,714 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:17,181 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:18,558 DEBUG   SenderThread:12261 [sender.py:send():375] send: exit
2023-06-23 07:29:18,558 INFO    SenderThread:12261 [sender.py:send_exit():598] handling exit code: 255
2023-06-23 07:29:18,559 INFO    SenderThread:12261 [sender.py:send_exit():600] handling runtime: 344
2023-06-23 07:29:18,558 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:29:18,562 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:18,563 INFO    SenderThread:12261 [sender.py:send_exit():606] send defer
2023-06-23 07:29:18,564 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:18,564 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 0
2023-06-23 07:29:18,564 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:18,564 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 0
2023-06-23 07:29:18,564 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 1
2023-06-23 07:29:18,565 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:18,565 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 1
2023-06-23 07:29:18,565 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:18,565 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 1
2023-06-23 07:29:18,565 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 2
2023-06-23 07:29:18,565 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:18,566 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 2
2023-06-23 07:29:18,566 INFO    HandlerThread:12261 [system_monitor.py:finish():190] Stopping system monitor
2023-06-23 07:29:18,566 DEBUG   SystemMonitor:12261 [system_monitor.py:_start():166] Finished system metrics aggregation loop
2023-06-23 07:29:18,571 DEBUG   SystemMonitor:12261 [system_monitor.py:_start():170] Publishing last batch of metrics
2023-06-23 07:29:18,569 INFO    HandlerThread:12261 [interfaces.py:finish():202] Joined cpu monitor
2023-06-23 07:29:18,572 INFO    HandlerThread:12261 [interfaces.py:finish():202] Joined disk monitor
2023-06-23 07:29:18,577 INFO    HandlerThread:12261 [interfaces.py:finish():202] Joined gpu monitor
2023-06-23 07:29:18,578 INFO    HandlerThread:12261 [interfaces.py:finish():202] Joined memory monitor
2023-06-23 07:29:18,578 INFO    HandlerThread:12261 [interfaces.py:finish():202] Joined network monitor
2023-06-23 07:29:18,579 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:18,579 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 2
2023-06-23 07:29:18,579 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 3
2023-06-23 07:29:18,579 DEBUG   SenderThread:12261 [sender.py:send():375] send: stats
2023-06-23 07:29:18,579 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:18,580 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 3
2023-06-23 07:29:18,580 DEBUG   SenderThread:12261 [sender.py:send():375] send: history
2023-06-23 07:29:18,581 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: summary_record
2023-06-23 07:29:18,583 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:18,585 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:18,585 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 3
2023-06-23 07:29:18,585 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 4
2023-06-23 07:29:18,585 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:18,585 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 4
2023-06-23 07:29:18,586 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:18,586 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 4
2023-06-23 07:29:18,586 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 5
2023-06-23 07:29:18,586 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:18,586 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 5
2023-06-23 07:29:18,587 DEBUG   SenderThread:12261 [sender.py:send():375] send: summary
2023-06-23 07:29:18,589 INFO    SenderThread:12261 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-06-23 07:29:18,591 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:18,591 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 5
2023-06-23 07:29:18,591 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 6
2023-06-23 07:29:18,591 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:18,591 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 6
2023-06-23 07:29:18,592 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:18,592 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 6
2023-06-23 07:29:18,592 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 7
2023-06-23 07:29:18,592 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:29:18,592 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:18,593 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 7
2023-06-23 07:29:18,593 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:18,593 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 7
2023-06-23 07:29:19,185 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:19,558 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: poll_exit
2023-06-23 07:29:19,633 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 8
2023-06-23 07:29:19,634 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: poll_exit
2023-06-23 07:29:19,634 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:19,634 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 8
2023-06-23 07:29:19,635 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:19,635 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 8
2023-06-23 07:29:19,652 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 9
2023-06-23 07:29:19,652 DEBUG   SenderThread:12261 [sender.py:send():375] send: artifact
2023-06-23 07:29:19,652 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:19,655 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 9
2023-06-23 07:29:20,188 INFO    Thread-12 :12261 [dir_watcher.py:_on_file_modified():295] file/dir modified: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/output.log
2023-06-23 07:29:20,559 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: poll_exit
2023-06-23 07:29:20,971 INFO    wandb-upload_0:12261 [upload_job.py:push():92] Skipped uploading /root/.local/share/wandb/artifacts/staging/tmpleyjcske
2023-06-23 07:29:22,082 INFO    wandb-upload_1:12261 [upload_job.py:push():95] Uploaded file /root/.local/share/wandb/artifacts/staging/tmpjb9tmmnq
2023-06-23 07:29:23,682 INFO    SenderThread:12261 [sender.py:send_artifact():1474] sent artifact job-https___github.com_YKefasu_kohya_ss_gui_google_colab_train_network.py - {'id': 'QXJ0aWZhY3Q6NDk2MzUzMTkw', 'digest': 'df6b349ca5d5f19f0762545ba62b7ffc', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjc4NTk3NjY3', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NDk2MjM4ODM3', 'versionIndex': 0}}, 'version': 'latest'}
2023-06-23 07:29:23,683 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:23,683 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: status_report
2023-06-23 07:29:23,683 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 9
2023-06-23 07:29:23,684 INFO    SenderThread:12261 [dir_watcher.py:finish():365] shutting down directory watcher
2023-06-23 07:29:24,196 INFO    SenderThread:12261 [dir_watcher.py:finish():395] scan: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files
2023-06-23 07:29:24,196 INFO    SenderThread:12261 [dir_watcher.py:finish():409] scan save: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/config.yaml config.yaml
2023-06-23 07:29:24,197 INFO    SenderThread:12261 [dir_watcher.py:finish():409] scan save: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json wandb-summary.json
2023-06-23 07:29:24,205 INFO    SenderThread:12261 [dir_watcher.py:finish():409] scan save: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/requirements.txt requirements.txt
2023-06-23 07:29:24,208 INFO    SenderThread:12261 [dir_watcher.py:finish():409] scan save: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-metadata.json wandb-metadata.json
2023-06-23 07:29:24,215 INFO    SenderThread:12261 [dir_watcher.py:finish():409] scan save: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/output.log output.log
2023-06-23 07:29:24,220 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 10
2023-06-23 07:29:24,227 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: poll_exit
2023-06-23 07:29:24,230 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:24,234 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 10
2023-06-23 07:29:24,235 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:24,235 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 10
2023-06-23 07:29:24,235 INFO    SenderThread:12261 [file_pusher.py:finish():168] shutting down file pusher
2023-06-23 07:29:24,846 INFO    wandb-upload_0:12261 [upload_job.py:push():137] Uploaded file /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/config.yaml
2023-06-23 07:29:25,061 INFO    wandb-upload_3:12261 [upload_job.py:push():137] Uploaded file /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/output.log
2023-06-23 07:29:25,561 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: keepalive
2023-06-23 07:29:25,601 INFO    wandb-upload_2:12261 [upload_job.py:push():137] Uploaded file /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/requirements.txt
2023-06-23 07:29:25,734 INFO    wandb-upload_1:12261 [upload_job.py:push():137] Uploaded file /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/files/wandb-summary.json
2023-06-23 07:29:25,934 INFO    Thread-11 (_thread_body):12261 [sender.py:transition_state():626] send defer: 11
2023-06-23 07:29:25,936 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:25,936 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 11
2023-06-23 07:29:25,936 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:25,936 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 11
2023-06-23 07:29:25,937 INFO    SenderThread:12261 [file_pusher.py:join():173] waiting for file pusher
2023-06-23 07:29:25,937 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 12
2023-06-23 07:29:25,937 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:25,937 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 12
2023-06-23 07:29:25,937 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:25,938 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 12
2023-06-23 07:29:26,364 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 13
2023-06-23 07:29:26,364 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:26,364 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 13
2023-06-23 07:29:26,365 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:26,365 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 13
2023-06-23 07:29:26,365 INFO    SenderThread:12261 [sender.py:transition_state():626] send defer: 14
2023-06-23 07:29:26,366 DEBUG   SenderThread:12261 [sender.py:send():375] send: final
2023-06-23 07:29:26,366 DEBUG   SenderThread:12261 [sender.py:send():375] send: footer
2023-06-23 07:29:26,366 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: defer
2023-06-23 07:29:26,366 INFO    HandlerThread:12261 [handler.py:handle_request_defer():170] handle defer: 14
2023-06-23 07:29:26,367 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: defer
2023-06-23 07:29:26,367 INFO    SenderThread:12261 [sender.py:send_request_defer():622] handle sender defer: 14
2023-06-23 07:29:26,368 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: poll_exit
2023-06-23 07:29:26,368 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: server_info
2023-06-23 07:29:26,369 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: get_summary
2023-06-23 07:29:26,369 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: sampled_history
2023-06-23 07:29:26,368 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: poll_exit
2023-06-23 07:29:26,371 DEBUG   SenderThread:12261 [sender.py:send_request():402] send_request: server_info
2023-06-23 07:29:26,577 INFO    MainThread:12261 [wandb_run.py:_footer_history_summary_info():3476] rendering history
2023-06-23 07:29:26,578 INFO    MainThread:12261 [wandb_run.py:_footer_history_summary_info():3508] rendering summary
2023-06-23 07:29:26,578 INFO    MainThread:12261 [wandb_run.py:_footer_sync_info():3434] logging synced files
2023-06-23 07:29:26,579 DEBUG   HandlerThread:12261 [handler.py:handle_request():144] handle_request: shutdown
2023-06-23 07:29:26,579 INFO    HandlerThread:12261 [handler.py:finish():845] shutting down handler
2023-06-23 07:29:27,369 INFO    WriterThread:12261 [datastore.py:close():298] close: /content/gdrive/MyDrive/ogipotereplicantv3/20230623072223/wandb/run-20230623_072333-hni6fhql/run-hni6fhql.wandb
2023-06-23 07:29:27,577 INFO    SenderThread:12261 [sender.py:finish():1550] shutting down sender
2023-06-23 07:29:27,577 INFO    SenderThread:12261 [file_pusher.py:finish():168] shutting down file pusher
2023-06-23 07:29:27,578 INFO    SenderThread:12261 [file_pusher.py:join():173] waiting for file pusher