Error: Some NCCL operations have failed or timed out
While running a distributed training on 4 A6000 GPUs, I get the following error:
[E ProcessGroupNCCL.cpp:630] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1803710 milliseconds before timing out.       
                                                                                                                                                        [E ProcessGroupNCCL.cpp:390] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.                                                                                 

terminate called after throwing an instance of 'std::runtime_error'                                                                                                        
what():  [Rank 2] Watchdog caught collective operation timeout: 
WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1804406 milliseconds before timing out.        

[E ProcessGroupNCCL.cpp:390] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.

I use standard NVidia PyTorch docker. Interesting thing is that training works fine for small datasets but for the bigger datasets, I get this error. So I can confirm that the training code is correct and does work.
There is no actual runtime error or any other information to get an actual error messages anywhere.

Following two have solved the issue:

Increase default SHM (shared memory) for CUDA to 10g (I think 1g would have worked as well). You can do this in docker run command by passing --shm-size=10g. I also pass --ulimit memlock=-1.
export NCCL_P2P_LEVEL=NVL.

Debugging Tips
To check current SHM,
df -h
# see the row for shm

To see NCCL debug messages:
export NCCL_DEBUG=INFO

Run p2p bandwidth test for GPU to GPU communication link:
cd /usr/local/cuda/samples/1_Utilities/p2pBandwidthLatencyTest
sudo make
./p2pBandwidthLatencyTest

For A6000 4 GPU box this prints:

The matrix shows bandwith betweeb each pair of GPU and with P2P, it should be high.
