Pytorch "NCCL error": unhandled system error, NCCL version 2.4.8"
I use pytorch to distributed training my model.I have two nodes and two gpu for each node, and I run the code for one node:
python train_net.py  --config-file configs/InstanceSegmentation/pointrend_rcnn_R_50_FPN_1x_coco.yaml  --num-gpu 2  --num-machines 2 --machine-rank 0 --dist-url tcp://192.168.**.***:8000

and the other:
python train_net.py  --config-file configs/InstanceSegmentation/pointrend_rcnn_R_50_FPN_1x_coco.yaml  --num-gpu 2  --num-machines 2 --machine-rank 1 --dist-url tcp://192.168.**.***:8000

However the other has RuntimeError problem
global_rank 3 machine_rank 1 num_gpus_per_machine 2 local_rank 1
global_rank 2 machine_rank 1 num_gpus_per_machine 2 local_rank 0
Traceback (most recent call last):
  File &quot;train_net.py&quot;, line 109, in &lt;module&gt;
    args=(args,),
  File &quot;/root/detectron2_repo/detectron2/engine/launch.py&quot;, line 49, in launch
    daemon=False,
  File &quot;/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py&quot;, line 171, in spawn
    while not spawn_context.join():
  File &quot;/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py&quot;, line 118, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File &quot;/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py&quot;, line 19, in _wrap
    fn(i, *args)
  File &quot;/root/detectron2_repo/detectron2/engine/launch.py&quot;, line 72, in _distributed_worker
    comm.synchronize()
  File &quot;/root/detectron2_repo/detectron2/utils/comm.py&quot;, line 79, in synchronize
    dist.barrier()
  File &quot;/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py&quot;, line 1489, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:410, unhandled system error, NCCL version 2.4.8

IF I change mask-rank = 1 to mask-rank = 0, then no error will be reported, but can't distributed training,Does anyone know why this error may occur?

A number of things can cause this issue, see for example 1, 2. Adding the line
import os
os.environ[&quot;NCCL_DEBUG&quot;] = &quot;INFO&quot;

to your script will log more specific debug info leading up to the error, giving you a more helpful error message to google.
