NCCL Connection Failed Using PyTorch Distributed
I am trying to send a PyTorch tensor from one machine to another with torch.distributed. The dist.init_process_group function works properly. However, there is a connection failure in the dist.broadcast function. Here is my code on node 0:

import torch
from torch import distributed as dist
import numpy as np
import os

master_addr = '47.xxx.xxx.xx'
master_port = 10000
world_size = 2
rank = 0
backend = 'nccl'

os.environ['MASTER_ADDR'] = master_addr
os.environ['MASTER_PORT'] = str(master_port)
os.environ['WORLD_SIZE'] = str(world_size)
os.environ['RANK'] = str(rank)
dist.init_process_group(backend, init_method='tcp://47.xxx.xxx.xx:10000', timeout=datetime.timedelta(0, 10), rank=rank, world_size=world_size)
print("Finished initializing process group; backend: %s, rank: %d, "
"world_size: %d" % (backend, rank, world_size))

a = torch.from_numpy(np.random.rand(3, 3)).cuda()
dist.broadcast(tensor=a, src=0)


Here is my code on node 1:


import torch
from torch import distributed as dist
import numpy as np
import os

master_addr = '47.xxx.xxx.xx'
master_port = 10000
world_size = 2
rank = 1
backend = 'nccl'

os.environ['MASTER_ADDR'] = master_addr
os.environ['MASTER_PORT'] = str(master_port)
os.environ['WORLD_SIZE'] = str(world_size)
os.environ['RANK'] = str(rank)
dist.init_process_group(backend, init_method='tcp://47.xxx.xxx.xx:10000', timeout=datetime.timedelta(0, 10), rank=rank, world_size=world_size)
print("Finished initializing process group; backend: %s, rank: %d, "
"world_size: %d" % (backend, rank, world_size))

a = torch.zeros((3,3)).cuda()
dist.broadcast(tensor=a, src=0)


I set NCCL_DEBUG=INFO before running the code. Here is the information I got on Node 1:


iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.181&lt;0&gt;
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/IB : No device found.
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.0.181&lt;0&gt;
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Setting affinity for GPU 0 to ffff
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Call to connect returned Connection timed out, retrying
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Call to connect returned Connection timed out, retrying

iZbp11ufz31riqnssil53cZ:13530:13553 [0] include/socket.h:395 NCCL WARN Connect to 192.168.0.143&lt;59811&gt; failed : Connection timed out
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO bootstrap.cc:100 -&gt; 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO bootstrap.cc:326 -&gt; 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO init.cc:695 -&gt; 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO init.cc:951 -&gt; 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO misc/group.cc:69 -&gt; 2 [Async thread]
Traceback (most recent call last):
  File "test_dist_1.py", line 25, in &lt;module&gt;
    dist.broadcast(tensor=a, src=0)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 806, in broadcast
    work = _default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /tmp/pip-req-build-58y_cjjl/torch/lib/c10d/ProcessGroupNCCL.cpp:290, unhandled system error


And Node 0 seems to stuck in function dist.broadcast:


iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.143&lt;0&gt;
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/IB : No device found.
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.0.143&lt;0&gt;
iZuf6cu11ru7evq9ybagdjZ:13530:13553 [0] NCCL INFO Setting affinity for GPU 0 to ffff


Can anyone help me with this? How can I send the tensor from node 0 to node 1? I would really appreciate any help!

I found that the two servers that I used were not under the same VPC. Therefore, they can never communicate. So I used other servers under the same VPC and it worked.
