Traceback (most recent call last):


 "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 977, in _get_connection

connection = self.deque.pop()
IndexError: pop from an empty deque

Traceback (most recent call last):


 "/root/OpenNMT-py/onmt/utils/distributed.py", line 200, in consumer

gpu_rank = multi_init(opt, device_id)


 "/root/OpenNMT-py/onmt/utils/distributed.py", line 25, in multi_init

torch.distributed.init_process_group(


 "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 442, in init_process_group

barrier()


 "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 1947, in barrier

work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
