os.environ["MASTER_ADDR"] = "localhost"
       os.environ["MASTER_PORT"] = "29500"
       main()
DDP works with TorchDynamo.  When used with TorchDynamo, apply the DDP
model wrapper before compiling the model, such that torchdynamo can
apply "DDPOptimizer" (graph-break optimizations) based on DDP bucket
sizes.  (See TorchDynamo DDPOptimizer for more information.)
TorchDynamo support for DDP currently requires setting
static_graph=False, due to interactions between the graph tracing
process and DDP's mechanism for observing operations happening on its
module, but this should be fixed ultimately.
ddp_model = DDP(model, device_ids=[rank])
   ddp_model = torch.compile(ddp_model)
Internal Design
This section reveals how it works under the hood of
"torch.nn.parallel.DistributedDataParallel" by diving into details of
every step in one iteration.

Prerequisite: DDP relies on c10d "ProcessGroup" for
  communications. Hence, applications must create "ProcessGroup"
