on input as in "torch.nn.DataParallel" apply.
Creation of this class requires that "torch.distributed" to be
   already initialized, by calling
   "torch.distributed.init_process_group()".
"DistributedDataParallel" is proven to be significantly faster than
   "torch.nn.DataParallel" for single-node multi-GPU data parallel
   training.
To use "DistributedDataParallel" on a host with N GPUs, you should
   spawn up "N" processes, ensuring that each process exclusively
   works on a single GPU from 0 to N-1. This can be done by either
   setting "CUDA_VISIBLE_DEVICES" for every process or by calling:



torch.cuda.set_device(i)



where i is from 0 to N-1. In each process, you should refer the
   following to construct this module:



torch.distributed.init_process_group(
    backend='nccl', world_size=N, init_method='...'
)
model = DistributedDataParallel(model, device_ids=[i], output_device=i)



In order to spawn up multiple processes per node, you can use