parameters as ready for reduction. During the backward pass, the
  "Reducer" would only wait for unready parameters, but it would still
  reduce all buckets. Marking a parameter gradient as ready does not
  help DDP skip buckets as for now, but it will prevent DDP from
  waiting for absent gradients forever during the backward pass. Note
  that traversing the autograd graph introduces extra overheads, so
  applications should only set "find_unused_parameters" to "True" when
  necessary.

Backward Pass: The "backward()" function is directly invoked on
  the loss "Tensor", which is out of DDP's control, and DDP uses
  autograd hooks registered at construction time to trigger gradients
  synchronizations. When one gradient becomes ready, its corresponding
  DDP hook on that grad accumulator will fire, and DDP will then mark
  that parameter gradient as ready for reduction. When gradients in
  one bucket are all ready, the "Reducer" kicks off an asynchronous
