


                    output_device=args.local_rank)




classmethod convert_sync_batchnorm(module, process_group=None)
  Helper function to convert all "BatchNorm*D" layers in the model
  to "torch.nn.SyncBatchNorm" layers.

  Parameters:
     * **module** (*nn.Module*) -- module containing one or more
       "BatchNorm*D" layers

     * **process_group** (*optional*) -- process group to scope
       synchronization, default is the whole world

  Returns:
     The original "module" with the converted
     "torch.nn.SyncBatchNorm" layers. If the original "module" is
     a "BatchNorm*D" layer, a new "torch.nn.SyncBatchNorm" layer
     object will be returned instead.

  Example:

     >>> # Network with nn.BatchNorm layer
     >>> module = torch.nn.Sequential(
     >>>            torch.nn.Linear(20, 100),
     >>>            torch.nn.BatchNorm1d(100),
     >>>          ).cuda()
     >>> # creating process group (optional)
