DistributedDataParallel
class torch.nn.parallel.DistributedDataParallel(module, device_ids=None, output_device=None, dim=0, broadcast_buffers=True, process_group=None, bucket_cap_mb=25, find_unused_parameters=False, check_reduction=False, gradient_as_bucket_view=False, static_graph=False)
Implements distributed data parallelism that is based on
   "torch.distributed" package at the module level.
This container provides data parallelism by synchronizing gradients
   across each model replica. The devices to synchronize across are
   specified by the input "process_group", which is the entire world
   by default. Note that "DistributedDataParallel" does not chunk or
   otherwise shard the input across participating GPUs; the user is
   responsible for defining how to do so, for example through the use
   of a "DistributedSampler".
See also: Basics and Use nn.parallel.DistributedDataParallel
   instead of multiprocessing or nn.DataParallel. The same constraints