2024-07-03 02:45:36 | INFO | model_worker | args: Namespace(awq_ckpt=None, awq_groupsize=-1, awq_wbits=16, controller_address='http://127.0.0.1:21002', conv_template=None, cpu_offloading=False, debug=False, device='cuda', dtype=None, embed_in_truncate=False, enable_exllama=False, enable_xft=False, exllama_cache_8bit=False, exllama_gpu_split=None, exllama_max_seq_len=4096, gptq_act_order=False, gptq_ckpt=None, gptq_groupsize=-1, gptq_wbits=16, gpus=None, host='127.0.0.1', limit_worker_concurrency=5, load_8bit=False, max_gpu_memory=None, model_names=None, model_path='lmsys/vicuna-7b-v1.5', no_register=False, num_gpus=1, port=21003, revision='main', seed=None, ssl=False, stream_interval=2, worker_address='http://127.0.0.1:21003', xft_dtype=None, xft_max_seq_len=4096)
2024-07-03 02:45:36 | INFO | model_worker | Loading the model ['vicuna-7b-v1.5'] on worker c3665333 ...
2024-07-03 02:45:36 | ERROR | stderr | /usr/local/lib/python3.8/dist-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2024-07-03 02:45:36 | ERROR | stderr |   warnings.warn(message, UserWarning)
2024-07-03 02:45:36 | ERROR | stderr | 
Loading checkpoint shards:   0%|                                                                          | 0/2 [00:00<?, ?it/s]
2024-07-03 02:45:41 | ERROR | stderr | 
Loading checkpoint shards:  50%|█████████████████████████████████                                 | 1/2 [00:04<00:04,  4.93s/it]
2024-07-03 02:45:43 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████| 2/2 [00:07<00:00,  3.27s/it]
2024-07-03 02:45:43 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████| 2/2 [00:07<00:00,  3.52s/it]
2024-07-03 02:45:43 | ERROR | stderr | 
2024-07-03 02:45:43 | ERROR | stderr | /usr/local/lib/python3.8/dist-packages/transformers/generation/configuration_utils.py:540: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
2024-07-03 02:45:43 | ERROR | stderr |   warnings.warn(
2024-07-03 02:45:43 | ERROR | stderr | /usr/local/lib/python3.8/dist-packages/transformers/generation/configuration_utils.py:545: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
2024-07-03 02:45:43 | ERROR | stderr |   warnings.warn(
2024-07-03 02:45:43 | ERROR | stderr | /usr/local/lib/python3.8/dist-packages/transformers/generation/configuration_utils.py:540: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
2024-07-03 02:45:43 | ERROR | stderr |   warnings.warn(
2024-07-03 02:45:43 | ERROR | stderr | /usr/local/lib/python3.8/dist-packages/transformers/generation/configuration_utils.py:545: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
2024-07-03 02:45:43 | ERROR | stderr |   warnings.warn(
2024-07-03 02:45:48 | INFO | model_worker | Register to controller
2024-07-03 02:45:48 | ERROR | stderr | INFO:     Started server process [97349]
2024-07-03 02:45:48 | ERROR | stderr | INFO:     Waiting for application startup.
2024-07-03 02:45:48 | ERROR | stderr | INFO:     Application startup complete.
2024-07-03 02:45:48 | ERROR | stderr | ERROR:    [Errno 98] error while attempting to bind on address ('127.0.0.1', 21003): address already in use
2024-07-03 02:45:48 | ERROR | stderr | INFO:     Waiting for application shutdown.
2024-07-03 02:45:48 | ERROR | stderr | INFO:     Application shutdown complete.