pytorch batch normalization in distributed train
wondering how distributed pytorch handle batch norm, when I add a batch norm layer, will pytorch engine use the same allreduce call to sync the data cross node? or the batch norm only happen on local node.

Similarly to DataParallel (check the first Warning box). It will compute the norm separately for each node (or, more precisely, each GPU). It will not sync the rolling estimates of the norm either, but it will keep the values from one of the GPUs in the end. So assuming the examples are distributed across your cluster randomly, your BatchNorm will work roughly as expected, except its estimates of the normalization factors will have higher variance due to smaller effective sample sizes.
