class_docstr: Applies Batch Normalization over a N-Dimensional input.
  
  The N-D input is a mini-batch of [N-2]D inputs with additional channel dimension) as described in the paper
  `Batch Normalization: Accelerating Deep Network Training by Reducing
  Internal Covariate Shift <https://arxiv.org/abs/1502.03167>`__ .
  
  .. math::
  
      y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
  
  The mean and standard-deviation are calculated per-dimension over all
  mini-batches of the same process groups. :math:`\gamma` and :math:`\beta`
  are learnable parameter vectors of size `C` (where `C` is the input size).
  By default, the elements of :math:`\gamma` are sampled from
  :math:`\mathcal{U}(0, 1)` and the elements of :math:`\beta` are set to 0.
  The standard-deviation is calculated via the biased estimator, equivalent to
  `torch.var(input, unbiased=False)`.
  
  Also by default, during training this layer keeps running estimates of its
  computed mean and variance, which are then used for normalization during
  evaluation. The running estimates are kept with a default :attr:`momentum`
  of 0.1.
  
  If :attr:`track_running_stats` is set to ``False``, this layer then does not
  keep running estimates, and batch statistics are instead used during
  evaluation time as well.
  
  .. note::
      This :attr:`momentum` argument is different from one used in optimizer
      classes and the conventional notion of momentum. Mathematically, the
      update rule for running statistics here is
      :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`,
      where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the
      new observed value.
  
  Because the Batch Normalization is done for each channel in the ``C`` dimension, computing
  statistics on ``(N, +)`` slices, it's common terminology to call this Volumetric Batch
  Normalization or Spatio-temporal Batch Normalization.
  
  Currently :class:`SyncBatchNorm` only supports
  :class:`~torch.nn.DistributedDataParallel` (DDP) with single GPU per process. Use
  :meth:`torch.nn.SyncBatchNorm.convert_sync_batchnorm()` to convert
  :attr:`BatchNorm*D` layer to :class:`SyncBatchNorm` before wrapping
  Network with DDP.
  
  Args:
      num_features: :math:`C` from an expected input of size
          :math:`(N, C, +)`
      eps: a value added to the denominator for numerical stability.
          Default: ``1e-5``
      momentum: the value used for the running_mean and running_var
          computation. Can be set to ``None`` for cumulative moving average
          (i.e. simple average). Default: 0.1
      affine: a boolean value that when set to ``True``, this module has
          learnable affine parameters. Default: ``True``
      track_running_stats: a boolean value that when set to ``True``, this
          module tracks the running mean and variance, and when set to ``False``,
          this module does not track such statistics, and initializes statistics
          buffers :attr:`running_mean` and :attr:`running_var` as ``None``.
          When these buffers are ``None``, this module always uses batch statistics.
          in both training and eval modes. Default: ``True``
      process_group: synchronization of stats happen within each process group
          individually. Default behavior is synchronization across the whole
          world
  
  Shape:
      - Input: :math:`(N, C, +)`
      - Output: :math:`(N, C, +)` (same shape as input)
  
  .. note::
      Synchronization of batchnorm statistics occurs only while training, i.e.
      synchronization is disabled when ``model.eval()`` is set or if
      ``self.training`` is otherwise ``False``.
  
  Examples::
  
      >>> # xdoctest: +SKIP
      >>> # With Learnable Parameters
      >>> m = nn.SyncBatchNorm(100)
      >>> # creating process group (optional)
      >>> # ranks is a list of int identifying rank ids.
      >>> ranks = list(range(8))
      >>> r1, r2 = ranks[:4], ranks[4:]
      >>> # Note: every rank calls into new_group for every
      >>> # process group created, even if that rank is not
      >>> # part of the group.
      >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]]
      >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1]
      >>> # Without Learnable Parameters
      >>> m = nn.BatchNorm3d(100, affine=False, process_group=process_group)
      >>> input = torch.randn(20, 100, 35, 45, 10)
      >>> output = m(input)
  
      >>> # network is nn.BatchNorm layer
      >>> sync_bn_network = nn.SyncBatchNorm.convert_sync_batchnorm(network, process_group)
      >>> # only single gpu per process is currently supported
      >>> ddp_sync_bn_network = torch.nn.parallel.DistributedDataParallel(
      >>>                         sync_bn_network,
      >>>                         device_ids=[args.local_rank],
      >>>                         output_device=args.local_rank)
class_docstr_tok: ["Applies","Batch","Normalization","over","a","N-Dimensional","input",".","The","N-D","input","is","a","mini-batch","of","[","N-2","]","D","inputs","with","additional","channel","dimension",")","as","described","in","the","paper","`","Batch","Normalization",":","Accelerating","Deep","Network","Training","by","Reducing","Internal","Covariate","Shift","<","https",":","\/\/arxiv.org\/abs\/1502.03167",">","`","__",".","..","math",":",":","y","=","\\frac","{","x","-","\\mathrm","{","E","}","[","x","]","}","{","\\sqrt","{","\\mathrm","{","Var","}","[","x","]","+","\\epsilon","}","}","*","\\gamma","+","\\beta","The","mean","and","standard-deviation","are","calculated","per-dimension","over","all","mini-batches","of","the","same","process","groups",".",":","math",":","`","\\gamma","`","and",":","math",":","`","\\beta","`","are","learnable","parameter","vectors","of","size","`","C","`","(","where","`","C","`","is","the","input","size",")",".","By","default",",","the","elements","of",":","math",":","`","\\gamma","`","are","sampled","from",":","math",":","`","\\mathcal","{","U","}","(","0",",","1",")","`","and","the","elements","of",":","math",":","`","\\beta","`","are","set","to","0",".","The","standard-deviation","is","calculated","via","the","biased","estimator",",","equivalent","to","`","torch.var","(","input",",","unbiased=False",")","`",".","Also","by","default",",","during","training","this","layer","keeps","running","estimates","of","its","computed","mean","and","variance",",","which","are","then","used","for","normalization","during","evaluation",".","The","running","estimates","are","kept","with","a","default",":","attr",":","`","momentum","`","of","0.1",".","If",":","attr",":","`","track_running_stats","`","is","set","to","``","False","``",",","this","layer","then","does","not","keep","running","estimates",",","and","batch","statistics","are","instead","used","during","evaluation","time","as","well",".","..","note",":",":","This",":","attr",":","`","momentum","`","argument","is","different","from","one","used","in","optimizer","classes","and","the","conventional","notion","of","momentum",".","Mathematically",",","the","update","rule","for","running","statistics","here","is",":","math",":","`","\\hat","{","x","}","_\\text","{","new","}","=","(","1","-","\\text","{","momentum","}",")","\\times","\\hat","{","x","}","+","\\text","{","momentum","}","\\times","x_t","`",",","where",":","math",":","`","\\hat","{","x","}","`","is","the","estimated","statistic","and",":","math",":","`","x_t","`","is","the","new","observed","value",".","Because","the","Batch","Normalization","is","done","for","each","channel","in","the","``","C","``","dimension",",","computing","statistics","on","``","(","N",",","+",")","``","slices",",","it","'s","common","terminology","to","call","this","Volumetric","Batch","Normalization","or","Spatio-temporal","Batch","Normalization",".","Currently",":","class",":","`","SyncBatchNorm","`","only","supports",":","class",":","`","~torch.nn.DistributedDataParallel","`","(","DDP",")","with","single","GPU","per","process",".","Use",":","meth",":","`","torch.nn.SyncBatchNorm.convert_sync_batchnorm","(",")","`","to","convert",":","attr",":","`","BatchNorm","*","D","`","layer","to",":","class",":","`","SyncBatchNorm","`","before","wrapping","Network","with","DDP",".","Args",":","num_features",":",":","math",":","`","C","`","from","an","expected","input","of","size",":","math",":","`","(","N",",","C",",","+",")","`","eps",":","a","value","added","to","the","denominator","for","numerical","stability",".","Default",":","``","1e-5","``","momentum",":","the","value","used","for","the","running_mean","and","running_var","computation",".","Can","be","set","to","``","None","``","for","cumulative","moving","average","(","i.e",".","simple","average",")",".","Default",":","0.1","affine",":","a","boolean","value","that","when","set","to","``","True","``",",","this","module","has","learnable","affine","parameters",".","Default",":","``","True","``","track_running_stats",":","a","boolean","value","that","when","set","to","``","True","``",",","this","module","tracks","the","running","mean","and","variance",",","and","when","set","to","``","False","``",",","this","module","does","not","track","such","statistics",",","and","initializes","statistics","buffers",":","attr",":","`","running_mean","`","and",":","attr",":","`","running_var","`","as","``","None","``",".","When","these","buffers","are","``","None","``",",","this","module","always","uses","batch","statistics",".","in","both","training","and","eval","modes",".","Default",":","``","True","``","process_group",":","synchronization","of","stats","happen","within","each","process","group","individually",".","Default","behavior","is","synchronization","across","the","whole","world","Shape",":","-","Input",":",":","math",":","`","(","N",",","C",",","+",")","`","-","Output",":",":","math",":","`","(","N",",","C",",","+",")","`","(","same","shape","as","input",")","..","note",":",":","Synchronization","of","batchnorm","statistics","occurs","only","while","training",",","i.e",".","synchronization","is","disabled","when","``","model.eval","(",")","``","is","set","or","if","``","self.training","``","is","otherwise","``","False","``",".","Examples",":",":",">",">",">","#","xdoctest",":","+SKIP",">",">",">","#","With","Learnable","Parameters",">",">",">","m","=","nn.SyncBatchNorm","(","100",")",">",">",">","#","creating","process","group","(","optional",")",">",">",">","#","ranks","is","a","list","of","int","identifying","rank","ids",".",">",">",">","ranks","=","list","(","range","(","8",")",")",">",">",">","r1",",","r2","=","ranks","[",":4","]",",","ranks","[","4",":","]",">",">",">","#","Note",":","every","rank","calls","into","new_group","for","every",">",">",">","#","process","group","created",",","even","if","that","rank","is","not",">",">",">","#","part","of","the","group",".",">",">",">","process_groups","=","[","torch.distributed.new_group","(","pids",")","for","pids","in","[","r1",",","r2","]","]",">",">",">","process_group","=","process_groups","[","0","if","dist.get_rank","(",")","<","=","3","else","1","]",">",">",">","#","Without","Learnable","Parameters",">",">",">","m","=","nn.BatchNorm3d","(","100",",","affine=False",",","process_group=process_group",")",">",">",">","input","=","torch.randn","(","20",",","100",",","35",",","45",",","10",")",">",">",">","output","=","m","(","input",")",">",">",">","#","network","is","nn.BatchNorm","layer",">",">",">","sync_bn_network","=","nn.SyncBatchNorm.convert_sync_batchnorm","(","network",",","process_group",")",">",">",">","#","only","single","gpu","per","process","is","currently","supported",">",">",">","ddp_sync_bn_network","=","torch.nn.parallel.DistributedDataParallel","(",">",">",">","sync_bn_network",",",">",">",">","device_ids=","[","args.local_rank","]",",",">",">",">","output_device=args.local_rank",")"]
docstr: Converts all :attr:`BatchNorm*D` layers in the model to :class:`torch.nn.SyncBatchNorm` layers.
  
  Args:
      module (nn.Module): module containing one or more :attr:`BatchNorm*D` layers
      process_group (optional): process group to scope synchronization,
          default is the whole world
  
  Returns:
      The original :attr:`module` with the converted :class:`torch.nn.SyncBatchNorm`
      layers. If the original :attr:`module` is a :attr:`BatchNorm*D` layer,
      a new :class:`torch.nn.SyncBatchNorm` layer object will be returned
      instead.
  
  Example::
  
      >>> # Network with nn.BatchNorm layer
      >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA)
      >>> module = torch.nn.Sequential(
      >>>            torch.nn.Linear(20, 100),
      >>>            torch.nn.BatchNorm1d(100),
      >>>          ).cuda()
      >>> # creating process group (optional)
      >>> # ranks is a list of int identifying rank ids.
      >>> ranks = list(range(8))
      >>> r1, r2 = ranks[:4], ranks[4:]
      >>> # Note: every rank calls into new_group for every
      >>> # process group created, even if that rank is not
      >>> # part of the group.
      >>> # xdoctest: +SKIP("distributed")
      >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]]
      >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1]
      >>> sync_bn_module = torch.nn.SyncBatchNorm.convert_sync_batchnorm(module, process_group)
code: def convert_sync_batchnorm(cls, module, process_group=None):
      r"""Converts all :attr:`BatchNorm*D` layers in the model to :class:`torch.nn.SyncBatchNorm` layers.
  
      Args:
          module (nn.Module): module containing one or more :attr:`BatchNorm*D` layers
          process_group (optional): process group to scope synchronization,
              default is the whole world
  
      Returns:
          The original :attr:`module` with the converted :class:`torch.nn.SyncBatchNorm`
          layers. If the original :attr:`module` is a :attr:`BatchNorm*D` layer,
          a new :class:`torch.nn.SyncBatchNorm` layer object will be returned
          instead.
  
      Example::
  
          >>> # Network with nn.BatchNorm layer
          >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA)
          >>> module = torch.nn.Sequential(
          >>>            torch.nn.Linear(20, 100),
          >>>            torch.nn.BatchNorm1d(100),
          >>>          ).cuda()
          >>> # creating process group (optional)
          >>> # ranks is a list of int identifying rank ids.
          >>> ranks = list(range(8))
          >>> r1, r2 = ranks[:4], ranks[4:]
          >>> # Note: every rank calls into new_group for every
          >>> # process group created, even if that rank is not
          >>> # part of the group.
          >>> # xdoctest: +SKIP("distributed")
          >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]]
          >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1]
          >>> sync_bn_module = torch.nn.SyncBatchNorm.convert_sync_batchnorm(module, process_group)
  
      """
      module_output = module
      if isinstance(module, torch.nn.modules.batchnorm._BatchNorm):
          module_output = torch.nn.SyncBatchNorm(
              module.num_features,
              module.eps,
              module.momentum,
              module.affine,
              module.track_running_stats,
              process_group,
          )
          if module.affine:
              with torch.no_grad():
                  module_output.weight = module.weight
                  module_output.bias = module.bias
          module_output.running_mean = module.running_mean
          module_output.running_var = module.running_var
          module_output.num_batches_tracked = module.num_batches_tracked
          module_output.training = module.training
          if hasattr(module, "qconfig"):
              module_output.qconfig = module.qconfig
      for name, child in module.named_children():
          module_output.add_module(
              name, cls.convert_sync_batchnorm(child, process_group)
          )
      del module
      return module_output
  
imports: from typing import Optional, Any
  import torch
  from torch import Tensor
  from torch.nn.parameter import Parameter, UninitializedParameter, UninitializedBuffer
  from ..None import functional
  from ..None import init
  from ._functions import SyncBatchNorm
  from .lazy import LazyModuleMixin
  from .module import Module
label_desc: Use image node_id 5 for calling the SyncBatchNorm obj's underlying member method code with example usage: obj.convert_sync_batchnorm(cls, module, process_group) and returns: module_output