gradients to become ready during the backward pass in approximately
  that order. The figure below shows an example. Note that, the
  "grad0" and "grad1" are in "bucket1", and the other two gradients
  are in "bucket0". Of course, this assumption might not always be
  true, and when that happens it could hurt DDP backward speed as the
  "Reducer" cannot kick off the communication at the earliest possible
  time. Besides bucketing, the "Reducer" also registers autograd hooks
  during construction, one hook per parameter. These hooks will be
  triggered during the backward pass when the gradient becomes ready.

Forward Pass: The DDP takes the input and passes it to the local
  model, and then analyzes the output from the local model if
  "find_unused_parameters" is set to "True". This mode allows running
  backward on a subgraph of the model, and DDP finds out which
  parameters are involved in the backward pass by traversing the
  autograd graph from the model output and marking all unused
