libraries. Its "_sync_param" function performs intra-process
  parameter synchronization when one DDP process works on multiple
  devices, and it also broadcasts model buffers from the process with
  rank 0 to all other processes. The inter-process parameter
  synchronization happens in "Reducer.cpp".


comm.h: implements the coalesced broadcast helper function which is
  invoked to broadcast model states during initialization and
  synchronize model buffers before the forward pass.


reducer.h: provides the core implementation for gradient
  synchronization in the backward pass. It has three entry point
  functions:


"Reducer": The constructor is called in "distributed.py" which
    registers "Reducer::autograd_hook()" to gradient accumulators.


"autograd_hook()" function will be invoked by the autograd engine
    when a gradient becomes ready.


"prepare_for_backward()" is called at the end of DDP forward pass
    in "distributed.py". It traverses the autograd graph to find

