title: How to solve dist.init_process_group from hanging (or deadlocks)?
body: <p>I was to set up DDP (distributed data parallel) on a DGX A100 but it doesn't work. Whenever I try to run it simply hangs. My code is super simple just spawning 4 processes for 4 gpus (for the sake of debugging I simply destroy the group immediately but it doesn't even reach there):</p>
  <pre><code>def find_free_port():
      &quot;&quot;&quot; https://stackoverflow.com/questions/1365265/on-localhost-how-do-i-pick-a-free-port-number &quot;&quot;&quot;
      import socket
      from contextlib import closing
  
      with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
          s.bind(('', 0))
          s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
          return str(s.getsockname()[1])
  
  def setup_process(rank, world_size, backend='gloo'):
      &quot;&quot;&quot;
      Initialize the distributed environment (for each process).
  
      gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
      it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.
  
      export NCCL_SOCKET_IFNAME=eth0
      export NCCL_IB_DISABLE=1
  
      https://stackoverflow.com/questions/61075390/about-pytorch-nccl-error-unhandled-system-error-nccl-version-2-4-8
  
      https://pytorch.org/docs/stable/distributed.html#common-environment-variables
      &quot;&quot;&quot;
      if rank != -1:  # -1 rank indicates serial code
          print(f'setting up rank={rank} (with world_size={world_size})')
          # MASTER_ADDR = 'localhost'
          MASTER_ADDR = '127.0.0.1'
          MASTER_PORT = find_free_port()
          # set up the master's ip address so this child process can coordinate
          os.environ['MASTER_ADDR'] = MASTER_ADDR
          print(f&quot;{MASTER_ADDR=}&quot;)
          os.environ['MASTER_PORT'] = MASTER_PORT
          print(f&quot;{MASTER_PORT}&quot;)
  
          # - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
          if torch.cuda.is_available():
              # unsure if this is really needed
              # os.environ['NCCL_SOCKET_IFNAME'] = 'eth0'
              # os.environ['NCCL_IB_DISABLE'] = '1'
              backend = 'nccl'
          print(f'{backend=}')
          # Initializes the default distributed process group, and this will also initialize the distributed package.
          dist.init_process_group(backend, rank=rank, world_size=world_size)
          # dist.init_process_group(backend, rank=rank, world_size=world_size)
          # dist.init_process_group(backend='nccl', init_method='env://', world_size=world_size, rank=rank)
          print(f'--&gt; done setting up rank={rank}')
          dist.destroy_process_group()
  
  mp.spawn(setup_process, args=(4,), world_size=4)
  </code></pre>
  <p>why is this hanging?</p>
  <p>nvidia-smi output:</p>
  <pre><code>$ nvidia-smi
  Fri Mar  5 12:47:17 2021       
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 450.102.04   Driver Version: 450.102.04   CUDA Version: 11.0     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |                               |                      |               MIG M. |
  |===============================+======================+======================|
  |   0  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
  | N/A   26C    P0    51W / 400W |      0MiB / 40537MiB |      0%      Default |
  |                               |                      |             Disabled |
  +-------------------------------+----------------------+----------------------+
  |   1  A100-SXM4-40GB      On   | 00000000:0F:00.0 Off |                    0 |
  | N/A   25C    P0    52W / 400W |      3MiB / 40537MiB |      0%      Default |
  |                               |                      |             Disabled |
  +-------------------------------+----------------------+----------------------+
  |   2  A100-SXM4-40GB      On   | 00000000:47:00.0 Off |                    0 |
  | N/A   25C    P0    51W / 400W |      3MiB / 40537MiB |      0%      Default |
  |                               |                      |             Disabled |
  +-------------------------------+----------------------+----------------------+
  |   3  A100-SXM4-40GB      On   | 00000000:4E:00.0 Off |                    0 |
  | N/A   25C    P0    51W / 400W |      3MiB / 40537MiB |      0%      Default |
  |                               |                      |             Disabled |
  +-------------------------------+----------------------+----------------------+
  |   4  A100-SXM4-40GB      On   | 00000000:87:00.0 Off |                    0 |
  | N/A   30C    P0    52W / 400W |      3MiB / 40537MiB |      0%      Default |
  |                               |                      |             Disabled |
  +-------------------------------+----------------------+----------------------+
  |   5  A100-SXM4-40GB      On   | 00000000:90:00.0 Off |                    0 |
  | N/A   29C    P0    53W / 400W |      0MiB / 40537MiB |      0%      Default |
  |                               |                      |             Disabled |
  +-------------------------------+----------------------+----------------------+
  |   6  A100-SXM4-40GB      On   | 00000000:B7:00.0 Off |                    0 |
  | N/A   29C    P0    52W / 400W |      0MiB / 40537MiB |      0%      Default |
  |                               |                      |             Disabled |
  +-------------------------------+----------------------+----------------------+
  |   7  A100-SXM4-40GB      On   | 00000000:BD:00.0 Off |                    0 |
  | N/A   48C    P0   231W / 400W |   7500MiB / 40537MiB |     99%      Default |
  |                               |                      |             Disabled |
  +-------------------------------+----------------------+----------------------+
                                                                                 
  +-----------------------------------------------------------------------------+
  | Processes:                                                                  |
  |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
  |        ID   ID                                                   Usage      |
  |=============================================================================|
  |    7   N/A  N/A    147243      C   python                           7497MiB |
  +-----------------------------------------------------------------------------+
  </code></pre>
  <p>How do I set up ddp in this new machine?</p>
  <hr />
  <h1>Update</h1>
  <p>btw I've successfully installed APEX because some other links say to do that but it still fails. For I did:</p>
  <p>went to: <a href="https://github.com/NVIDIA/apex" rel="noreferrer">https://github.com/NVIDIA/apex</a> follwed their instructions</p>
  <pre><code>git clone https://github.com/NVIDIA/apex
  cd apex
  pip install -v --disable-pip-version-check --no-cache-dir --global-option=&quot;--cpp_ext&quot; --global-option=&quot;--cuda_ext&quot; ./
  </code></pre>
  <p>but before the above <a href="https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6" rel="noreferrer">I had to update gcc</a>:</p>
  <pre><code>conda install -c psi4 gcc-5
  </code></pre>
  <p>it did install it as I successfully imported it but it didn't help.</p>
  <hr />
  <p>Now it actually prints an error msg:</p>
  <pre><code>Traceback (most recent call last):
    File &quot;/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/multiprocessing/process.py&quot;, line 315, in _bootstrap
      self.run()
    File &quot;/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/multiprocessing/process.py&quot;, line 108, in run
      self._target(*self._args, **self._kwargs)
    File &quot;/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py&quot;, line 19, in _wrap
      fn(i, *args)
  KeyboardInterrupt
  Process SpawnProcess-3:
  Traceback (most recent call last):
    File &quot;/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py&quot;, line 19, in _wrap
      fn(i, *args)
    File &quot;/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py&quot;, line 252, in train
      setup_process(rank, world_size=opts.world_size)
    File &quot;/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/distributed.py&quot;, line 85, in setup_process
      dist.init_process_group(backend, rank=rank, world_size=world_size)
    File &quot;/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py&quot;, line 436, in init_process_group
      store, rank, world_size = next(rendezvous_iterator)
    File &quot;/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/distributed/rendezvous.py&quot;, line 179, in _env_rendezvous_handler
      store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
  RuntimeError: connect() timed out.
  
  During handling of the above exception, another exception occurred:
  </code></pre>
  <hr />
  <p>related:</p>
  <ul>
  <li><a href="https://github.com/pytorch/pytorch/issues/9696" rel="noreferrer">https://github.com/pytorch/pytorch/issues/9696</a></li>
  <li><a href="https://discuss.pytorch.org/t/dist-init-process-group-hangs-silently/55347/2" rel="noreferrer">https://discuss.pytorch.org/t/dist-init-process-group-hangs-silently/55347/2</a></li>
  <li><a href="https://forums.developer.nvidia.com/t/imagenet-hang-on-dgx-1-when-using-multiple-gpus/61919" rel="noreferrer">https://forums.developer.nvidia.com/t/imagenet-hang-on-dgx-1-when-using-multiple-gpus/61919</a></li>
  <li>apex suggestion: <a href="https://discourse.mozilla.org/t/hangs-on-dist-init-process-group-in-distribute-py/44686" rel="noreferrer">https://discourse.mozilla.org/t/hangs-on-dist-init-process-group-in-distribute-py/44686</a></li>
  <li><a href="https://github.com/pytorch/pytorch/issues/15638" rel="noreferrer">https://github.com/pytorch/pytorch/issues/15638</a></li>
  <li><a href="https://github.com/pytorch/pytorch/issues/53395" rel="noreferrer">https://github.com/pytorch/pytorch/issues/53395</a></li>
  </ul>