Torch: Nccl available but not used (?)
I use PyTorch 1.9.0 but get the following error when trying to run a distributed version of a model:
File &quot;/home/ferdiko/fastmoe/examples/transformer-xl/train.py&quot;, line 315, in &lt;module&gt;
    para_model = DistributedGroupedDataParallel(model).to(device)
  File &quot;/home/ferdiko/anaconda3/envs/fastmoe/lib/python3.9/site-packages/fastmoe-0.2.1-py3.9-linux-x86_64.egg/fmoe/distributed.py&quot;, line 45, in __init__
    self.comms[&quot;dp&quot;] = get_torch_default_comm()
  File &quot;/home/ferdiko/anaconda3/envs/fastmoe/lib/python3.9/site-packages/fastmoe-0.2.1-py3.9-linux-x86_64.egg/fmoe/utils.py&quot;, line 30, in get_torch_default_comm
    raise RuntimeError(&quot;Unsupported PyTorch version&quot;)

if I run torch.cuda.nccl.version() I get 2708. The developers suggested to run:
x = torch.rand(10).cuda() 
print(torch.cuda.nccl.is_available(x))


which gives me False. Does this actually mean that there's a problem with PyTorch and NCCL?

torch.cuda.nccl.is_available takes a sequence of tensors, and if they are on different devices, there is hope that you'll get a True:
    In [1]: import torch
    
    In [2]: x = torch.rand(1024, 1024, device='cuda:0')
    
    In [3]: y = torch.rand(1024, 1024, device='cuda:1')
    
    In [4]: torch.cuda.nccl.is_available([x, y])
    Out[4]: True


If you give it just one tensor, torch.cuda.nccl.is_available will iterate through it instead, but different parts of the same tensor are always on the same device, so you'll always get a False:
    In [5]: torch.cuda.nccl.is_available(x)
    Out[5]: False


    In [6]: torch.cuda.nccl.is_available([x])
    Out[6]: True

