SyncBatchNorm
class torch.nn.SyncBatchNorm(num_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, process_group=None, device=None, dtype=None)
Applies Batch Normalization over a N-Dimensional input (a mini-
   batch of [N-2]D inputs with additional channel dimension) as
   described in the paper Batch Normalization: Accelerating Deep
   Network Training by Reducing Internal Covariate Shift .
  y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}}
  * \gamma + \beta

The mean and standard-deviation are calculated per-dimension over
   all mini-batches of the same process groups. \gamma and \beta are
   learnable parameter vectors of size C (where C is the input
   size). By default, the elements of \gamma are sampled from
   \mathcal{U}(0, 1) and the elements of \beta are set to 0. The
   standard-deviation is calculated via the biased estimator,
   equivalent to torch.var(input, unbiased=False).