Variables:
      module (Module) -- the module to be parallelized.
Example:
  >>> torch.distributed.init_process_group(backend='nccl', world_size=4, init_method='...')
  >>> net = torch.nn.parallel.DistributedDataParallel(model)

join(divide_by_initial_world_size=True, enable=True, throw_on_early_termination=False)
  A context manager to be used in conjunction with an instance of
  "torch.nn.parallel.DistributedDataParallel" to be able to train
  with uneven inputs across participating processes.

  This context manager will keep track of already-joined DDP
  processes, and "shadow" the forward and backward passes by
  inserting collective communication operations to match with the
  ones created by non-joined DDP processes. This will ensure each
  collective call has a corresponding call by already-joined DDP
  processes, preventing hangs or errors that would otherwise
  happen when training with uneven inputs across processes.
