Inconsistency in SyncBatchNormalization/BatchNormalization(synchronized=True) Results during Distributed Training on CPUs
Hi @Liqi1003 ,

Thanks for reporting. As the `SyncBatchNormalization` API endpoint is deprecated and might removed in Future release, I tried to replicate the `BatchNormalization` issue with `synchronized=True`. I have replicated the difference in outputs wrt 1- CPU and 2-CPU as reported and the value might not be ignorable and needs to have a look into it.Reference [gist](https://colab.research.google.com/gist/SuryanarayanaY/fbcc86cf6730918960ef275a1fa3ddfa/syncbatchnorm-inconsistency-tf-nightly.ipynb).


> while for the model removing batchnorm layer it’s Linf = 6.212488e-36

Could you please submit code snippet for this also.

Thanks! @SuryanarayanaY 
Thanks for the quick reproduction! The reproduction code without SyncBatchNormalization is in the same colab here: https://colab.research.google.com/drive/1N-aXPcfckVb8fPDSmghMlzBOFEGq6RzM?usp=sharing

In the last code block, uncomment the second line and execute the code block again, and you should be able to see the difference without SyncBatchNormalization layers. @Liqi1003 ,

I have done some changes in code to check the difference in outputs with 1 CPU and 2 CPUs with `BatchNormalization(synchronised=True)` and `BatchNormalization(synchronised=False)` and the difference in results wrt 1-CPU and 2-CPU came around same at `0.00010111928`. Could you please refer attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/fc3d4fee3ec732665af2b1d86f403995/60438-syncbatchnorm-inconsistency-tf-nightly.ipynb).

In your code for `synchronised=False` it is omitting `batch_normalization` and hence there is large difference in results.


Thanks!



 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. @SuryanarayanaY 

Thanks for the debugging effort! 
However, the key point here is that BatchNormalization layer is introducing big variances when training on CPUs. On 1-GPU vs 2-GPU setting, we see very small differences (~1e-7), but on 1-CPU vs 2-CPU setting, we can see a much larger difference (~1e-4).

Given our effort to control variance by limiting training steps, a difference of 1e-4 should be considered big. Although with larger training size, the difference could either become bigger or cancle out, we do see the difference in one-step training.

I suspect there is something special with CPU execution of BatchNormalization layers, which caused the difference. @Liqi1003 ,

Will escalate the issue to Dev team for their comments. Thanks!
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

tf 2.11.0

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 20.04.5 LTS

### Mobile device

_No response_

### Python version

3.8.10

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

11.7

### GPU model and memory

_No response_

### Current Behaviour?

I am reporting an issue encountered during the distributed training of a model with `SyncBatchNormalization` layers on CPUs. The issue presents as a relatively large inconsistency between the results produced when training the same model on a single device versus multiple devices.

We've prepared a reproduction script demonstrating this issue. We set the weights of the `SyncBatchNormalization`/`BatchNormalization` layer to reproduce this bug. After conducting a one-step training, the predictions from the identical model trained on a single CPU and two CPUs exhibited relatively large differences (measured by L-infinity distance, Linf = 0.00074467063). As a comparison, we build another model with exactly the same architecture but removing the `SyncBatchNormalization` layer, and the prediction difference from this model trained on a single CPU and two CPUs is Linf = 1.0131771e-09.

Another comparison experiment is on GPUs. We trained the same model containing `SyncBatchNormalization` layer on a single GPU and two GPUs. The Linf of model’s prediction result is 2.5331974e-07. It’s much smaller than 1CPU vs 2CPU (Linf = 0.00074467063). This is weird because I would expect the CPU executions to be more deterministic than GPUs, and I would expect larger inconsistencies in the GPU runs if the inconsistencies are caused by variance.

To ensure a controlled environment, we used the same training data for one-step training on both setups, and then evaluated the model. To guarantee that the distributed trainings are expected to produce the same results given different number of devices, we use `MirroredStrategy`, keep the global batch size the same, and use `tf.keras.losses.Reduction.AUTO` reduction. To make the difference more apparent given a limited amount of training data, we deliberately chose a relatively high learning rate (lr=10).

It's noteworthy that `SyncBatchNormalization` has been deprecated as of TensorFlow version 2.12. Therefore, we also conducted the same experiment using TensorFlow's nightly build (2.13.0-dev20230428), replacing `SyncBatchNormalization()` with `BatchNormalization(synchronized=True)`. This experiment still manifested the same inconsistency. The prediction from the model containing batchnorm layer trained on a single CPU and two CPUs exhibits a substantial difference of Linf = 0.00010111928, while for the model removing batchnorm layer it’s Linf = 6.212488e-36.


### Standalone code to reproduce the issue

```shell
For TF 2.11.0 CPU experiments, the model contains `SyncBatchNormalization` layer:
https://colab.research.google.com/drive/11EseXxq_uHweY7omt7JCr1mc-42Kf1H5?usp=sharing

For TF 2.11.0 GPU experiments, the model contains `SyncBatchNormalization` layer (fix seed to generate same inputs):
https://colab.research.google.com/drive/1m_gCWzb_OKVTYAMMs8YbdKUYM3aINNPV?usp=sharing 

For TF nightly 2.13.0-dev20230428, the model contains `BatchNormalization(synchronized=True)` layer:
https://colab.research.google.com/drive/1N-aXPcfckVb8fPDSmghMlzBOFEGq6RzM?usp=sharing
```


### Relevant log output

For TF 2.11.0:

```shell
contain syncbatchnorm:  True
1CPU vs 2CPU: 0.00074467063

contain syncbatchnorm:  False
1CPU vs 2CPU: 1.0131771e-09

contain syncbatchnorm:  True
1GPU vs 2GPU: 2.5331974e-07
```

For TF nightly 2.13.0-dev20230428:

```shell
contain syncbatchnorm:  True
1CPU vs 2CPU: 0.00010111928

contain syncbatchnorm:  False
1CPU vs 2CPU: 6.212488e-36
```

</details>