Note:
 Parameters are never broadcast between processes. The module
 performs an all-reduce step on gradients and assumes that they
 will be modified by the optimizer in all processes in the same
 way. Buffers (e.g. BatchNorm stats) are broadcast from the module
 in process of rank 0, to all other replicas in the system in
 every iteration.

Note:
 If you are using DistributedDataParallel in conjunction with the
 Distributed RPC Framework, you should always use
 "torch.distributed.autograd.backward()" to compute gradients and
 "torch.distributed.optim.DistributedOptimizer" for optimizing
 parameters.Example:

    >>> import torch.distributed.autograd as dist_autograd
    >>> from torch.nn.parallel import DistributedDataParallel as DDP
    >>> import torch
    >>> from torch import optim
    >>> from torch.distributed.optim import DistributedOptimizer
    >>> import torch.distributed.rpc as rpc
