either "torch.distributed.launch" or "torch.multiprocessing.spawn".
Note:
 Please refer to PyTorch Distributed Overview for a brief
 introduction to all features related to distributed training.

Note:
 "DistributedDataParallel" can be used in conjunction with
 "torch.distributed.optim.ZeroRedundancyOptimizer" to reduce per-
 rank optimizer states memory footprint. Please refer to
 ZeroRedundancyOptimizer recipe for more details.

Note:
 "nccl" backend is currently the fastest and highly recommended
 backend when using GPUs. This applies to both single-node and
 multi-node distributed training.

Note:
 This module also supports mixed-precision distributed training.
 This means that your model can have different types of parameters
 such as mixed types of "fp16" and "fp32", the gradient reduction
 on these mixed types of parameters will just work fine.

Note:
 If you use "torch.save" on one process to checkpoint the module,
