Distributed Data Parallel
Warning:
The implementation of "torch.nn.parallel.DistributedDataParallel"
  evolves over time. This design note is written based on the state as
  of v1.4.
"torch.nn.parallel.DistributedDataParallel" (DDP) transparently
performs distributed data parallel training. This page describes how
it works and reveals implementation details.
Example
Let us start with a simple "torch.nn.parallel.DistributedDataParallel"
example. This example uses a "torch.nn.Linear" as the local model,
wraps it with DDP, and then runs one forward pass, one backward pass,
and an optimizer step on the DDP model. After that, parameters on the
local model will be updated, and all models on different processes
should be exactly the same.
import torch
   import torch.distributed as dist
   import torch.multiprocessing as mp
   import torch.nn as nn
   import torch.optim as optim
   from torch.nn.parallel import DistributedDataParallel as DDP