class_docstr: synchronized batch normalization module extented from ``torch.nn.BatchNormNd``
  with the added stats reduction across multiple processes.
  
  When running in training mode, the layer reduces stats across process groups
  to increase the effective batchsize for normalization layer. This is useful
  in applications where batch size is small on a given process that would
  diminish converged accuracy of the model.
  
  When running in evaluation mode, the layer falls back to
  ``torch.nn.functional.batch_norm``.
  
  Args:
      num_features: :math:`C` from an expected input of size
          :math:`(N, C, L)` or :math:`L` from input of size :math:`(N, L)`
      eps: a value added to the denominator for numerical stability.
          Default: 1e-5
      momentum: the value used for the running_mean and running_var
          computation. Can be set to ``None`` for cumulative moving average
          (i.e. simple average). Default: 0.1
      affine: a boolean value that when set to ``True``, this module has
          learnable affine parameters. Default: ``True``
      track_running_stats: a boolean value that when set to ``True``, this
          module tracks the running mean and variance, and when set to ``False``,
          this module does not track such statistics and always uses batch
          statistics in both training and eval modes. Default: ``True``
  
  Example::
  
      >>> sbn = apex.contrib.GroupBatchNorm2d(100).cuda()
      >>> inp = torch.randn(10, 100, 14, 14).cuda()
      >>> out = sbn(inp)
      >>> inp = torch.randn(3, 100, 20).cuda()
      >>> out = sbn(inp)
class_docstr_tok: ["synchronized","batch","normalization","module","extented","from","``","torch.nn.BatchNormNd","``","with","the","added","stats","reduction","across","multiple","processes",".","When","running","in","training","mode",",","the","layer","reduces","stats","across","process","groups","to","increase","the","effective","batchsize","for","normalization","layer",".","This","is","useful","in","applications","where","batch","size","is","small","on","a","given","process","that","would","diminish","converged","accuracy","of","the","model",".","When","running","in","evaluation","mode",",","the","layer","falls","back","to","``","torch.nn.functional.batch_norm","``",".","Args",":","num_features",":",":","math",":","`","C","`","from","an","expected","input","of","size",":","math",":","`","(","N",",","C",",","L",")","`","or",":","math",":","`","L","`","from","input","of","size",":","math",":","`","(","N",",","L",")","`","eps",":","a","value","added","to","the","denominator","for","numerical","stability",".","Default",":","1e-5","momentum",":","the","value","used","for","the","running_mean","and","running_var","computation",".","Can","be","set","to","``","None","``","for","cumulative","moving","average","(","i.e",".","simple","average",")",".","Default",":","0.1","affine",":","a","boolean","value","that","when","set","to","``","True","``",",","this","module","has","learnable","affine","parameters",".","Default",":","``","True","``","track_running_stats",":","a","boolean","value","that","when","set","to","``","True","``",",","this","module","tracks","the","running","mean","and","variance",",","and","when","set","to","``","False","``",",","this","module","does","not","track","such","statistics","and","always","uses","batch","statistics","in","both","training","and","eval","modes",".","Default",":","``","True","``","Example",":",":",">",">",">","sbn","=","apex.contrib.GroupBatchNorm2d","(","100",")",".cuda","(",")",">",">",">","inp","=","torch.randn","(","10",",","100",",","14",",","14",")",".cuda","(",")",">",">",">","out","=","sbn","(","inp",")",">",">",">","inp","=","torch.randn","(","3",",","100",",","20",")",".cuda","(",")",">",">",">","out","=","sbn","(","inp",")"]
code: def __init__(
      self,
      num_features,
      group_size,
      eps=1e-5,
      momentum=0.1,
      affine=True,
      track_running_stats=True,
  ):
      super(GroupBatchNorm2d, self).__init__(
          num_features,
          eps=eps,
          momentum=momentum,
          affine=affine,
          track_running_stats=track_running_stats,
      )
      self.group_size = group_size
      rank = torch.distributed.get_rank()
      self.group_id = rank // group_size
      self.group_rank = rank % group_size
      self.fwd_peer_buffers = self.get_peer_buffers(num_features)
      self.bwd_peer_buffers = self.get_peer_buffers(num_features)
      self.minibatch_mean = torch.cuda.FloatTensor(num_features)
      self.minibatch_inv_var = torch.cuda.FloatTensor(num_features)
  
file_path: apex/apex/contrib/cudnn_gbn/batch_norm.py
imports: import torch
  from torch.nn.modules.batchnorm import _BatchNorm
  from torch.nn import functional
  from torch import Tensor
  import peer_memory_cuda
  import cudnn_gbn_lib
  from torch.cuda.amp import custom_fwd, custom_bwd
label_desc: Use image node_id 1 to create a new GroupBatchNorm2d object from inherited base classes: _BatchNorm with example: obj = GroupBatchNorm2d(num_features, group_size, eps, momentum, affine, track_running_stats)