"allreduce" on that bucket to calculate mean of gradients across all
  processes. When all buckets are ready, the "Reducer" will block
  waiting for all "allreduce" operations to finish. When this is done,
  averaged gradients are written to the "param.grad" field of all
  parameters. So after the backward pass, the grad field on the same
  corresponding parameter across different DDP processes should be the
  same.

Optimizer Step: From the optimizer's perspective, it is
  optimizing a local model. Model replicas on all DDP processes can
  keep in sync because they all start from the same state and they
  have the same averaged gradients in every iteration.

[image: ddp_grad_sync.png][image]
Note:
DDP requires "Reducer" instances on all processes to invoke
  "allreduce" in exactly the same order, which is done by always
  running "allreduce" in the bucket index order instead of actual
  bucket ready order. Mismatched "allreduce" order across processes