can lead to wrong results or DDP backward hang.
Implementation
Below are pointers to the DDP implementation components. The stacked
graph shows the structure of the code.
ProcessGroup


ProcessGroup.hpp: contains the abstract API of all process group
  implementations. The "c10d" library provides 3 implementations out
  of the box, namely, ProcessGroupGloo, ProcessGroupNCCL, and
  ProcessGroupMPI. "DistributedDataParallel" uses
  "ProcessGroup::broadcast()" to send model states from the process
  with rank 0 to others during initialization and
  "ProcessGroup::allreduce()" to sum gradients.


Store.hpp: assists the rendezvous service for process group
  instances to find each other.


DistributedDataParallel

distributed.py: is the Python entry point for DDP. It implements the
  initialization steps and the "forward" function for the
  "nn.parallel.DistributedDataParallel" module which call into C++
