unused parameters when "find_unused_parameters" is set to "True"
    in DDP constructor.
[image: ddp_code.png][image]
TorchDynamo DDPOptimizer
DDP's performance advantage comes from overlapping allreduce
collectives with computations during backwards. AotAutograd prevents
this overlap when used with TorchDynamo for compiling a whole forward
and whole backward graph, because allreduce ops are launched by
autograd hooks after the whole optimized backwards computation
finishes.
TorchDynamo's DDPOptimizer helps by breaking the forward graph at the
logical boundaries of DDP's allreduce buckets during backwards.  Note:
the goal is to break the graph during backwards, and the simplest
implementation is to break the forward graphs and then call
AotAutograd and compilation on each section.  This allows DDP's
allreduce hooks to fire in-between sections of backwards, and schedule
communications to overlap with compute.