The return value when LayerNormalization takes zero vectors as input
Hi @PhyllisJi,

There is a new parameter called momentum in Batch Normalization which makes a difference during inference.

For Batch Normalization - **During inference (i.e. when using evaluate() or predict() or when calling the layer/model with the argument training=False (which is the default), the layer normalizes its output using a moving average of the mean and standard deviation of the batches it has seen during training. That is to say, it returns gamma * (batch - self.moving_mean) / sqrt(self.moving_var+epsilon) + beta.

self.moving_mean and self.moving_var are non-trainable variables that are updated each time the layer in called in training mode, as such:

moving_mean = moving_mean * momentum + mean(batch) * (1 - momentum)
moving_var = moving_var * momentum + var(batch) * (1 - momentum)
As such, the layer will only normalize its inputs during inference after having been trained on data that has similar statistics as the inference data.**
Check the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization) for your reference

We can see by default, training = False, when training= True, we get NAN values for batch normalization.

Please find the [gist](https://colab.research.google.com/gist/Varsha-anjanappa/0f23b826355921b6831423de356b7bf5/61287.ipynb) for your reference.

Thank you!!



 > Hi @PhyllisJi,
> 
> There is a new parameter called momentum in Batch Normalization which makes a difference during inference.
> 
> For Batch Normalization - **During inference (i.e. when using evaluate() or predict() or when calling the layer/model with the argument training=False (which is the default), the layer normalizes its output using a moving average of the mean and standard deviation of the batches it has seen during training. That is to say, it returns gamma * (batch - self.moving_mean) / sqrt(self.moving_var+epsilon) + beta.
> 
> self.moving_mean and self.moving_var are non-trainable variables that are updated each time the layer in called in training mode, as such:
> 
> moving_mean = moving_mean * momentum + mean(batch) * (1 - momentum) moving_var = moving_var * momentum + var(batch) * (1 - momentum) As such, the layer will only normalize its inputs during inference after having been trained on data that has similar statistics as the inference data.** Check the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization) for your reference
> 
> We can see by default, training = False, when training= True, we get NAN values for batch normalization.
> 
> Please find the [gist](https://colab.research.google.com/gist/Varsha-anjanappa/0f23b826355921b6831423de356b7bf5/61287.ipynb) for your reference.
> 
> Thank you!!

Hi, 
First, using tensorflow version 2.12.0, BatchNormalization outputs 0 regardless of whether trainable is False or True, and LayerNormalization outputs nan, and I'll try it on 2.13.0 next; Second, we want to express that epsilon is a bias term designed to prevent division by zero errors, so we don't think it can accept a value of 0.

<img width="888" alt="截屏2023-07-20 22 44 31" src="https://github.com/tensorflow/tensorflow/assets/127826463/7342df1e-9d29-4dfd-a6c7-47333053a5ef"> Hi @BiophiliaSWDA,

Please check the gist provided in the previous comment. We've used the parameter training and not trainable. When training = False, we get 0 values and when training= True, we get NAN values for batch normalization.

Thank you.
 > Hi @BiophiliaSWDA,
> 
> Please check the gist provided in the previous comment. We've used the parameter training and not trainable. When training = False, we get 0 values and when training= True, we get NAN values for batch normalization.
> 
> Thank you.


Sorry for using the wrong parameter. But I want to emphasize these:

1) When I use `training=True` for two functions, I get NAN, but when I use `training=False` I get the different output then. So `training=True` doesn't seem to be a good solution;

2) The epsilon parameter, according to tensorflow's official documentation, is defined as (Small float added to variance to avoid dividing by zero. Defaults to 1e-3), `0` should not be supported as a parameter to prevent division by zero problems;

3) The function we are testing is tf.keras.layers.LayerNormalization not tf.keras.layers.BatchNormalization.

```python
# My code is as follows:
import tensorflow as tf
data = tf.zeros(shape=(3, 4))
print(data)

layer1 = tf.keras.layers.LayerNormalization(axis=1, epsilon=0)
output1 = layer1(data, training=True)
print("LayerNormalization: training=True: \n{}".format(output1))

layer2 = tf.keras.layers.LayerNormalization(axis=1, epsilon=0)
output2 = layer2(data, training=False)
print("LayerNormalization: training=False: \n{}".format(output2))

layer3 = tf.keras.layers.BatchNormalization(axis=1, epsilon=0)
output3 = layer3(data, training=True)
print("BatchNormalization: training=True: \n{}".format(output3))

layer4 = tf.keras.layers.BatchNormalization(axis=1, epsilon=0)
output4 = layer4(data, training=False)
print("BatchNormalization: training=False: \n{}".format(output4))

```

```python
# The output is as follows:
tf.Tensor(
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]], shape=(3, 4), dtype=float32)
LayerNormalization: training=True: 
[[nan nan nan nan]
 [nan nan nan nan]
 [nan nan nan nan]]
LayerNormalization: training=False: 
[[nan nan nan nan]
 [nan nan nan nan]
 [nan nan nan nan]]
BatchNormalization: training=True: 
[[nan nan nan nan]
 [nan nan nan nan]
 [nan nan nan nan]]
BatchNormalization: training=False: 
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]]
```

 Hi @BiophiliaSWDA 

During Inference BatchNormalization works different. Please check the call function in source code that takes training, mask, inputs as additional parameters.

https://github.com/keras-team/keras/blob/b3ffea6602dbbb481e82312baa24fe657de83e11/keras/layers/normalization/batch_normalization.py#L547

When `training=False` which is the default value for batch normalization during inference the calculations in batch normalization changes .Please refer the source code here.

https://github.com/keras-team/keras/blob/b3ffea6602dbbb481e82312baa24fe657de83e11/keras/layers/normalization/batch_normalization.py#L645

Whereas in layer normalization there is no specific implementation during inference, by default it considers as training and performs the normalization 
Please refer the source code for layer normalization below :

https://github.com/keras-team/keras/blob/b3ffea6602dbbb481e82312baa24fe657de83e11/keras/layers/normalization/layer_normalization.py#L253

Here we can see mean and variance is calculated by using tf.nn.moments
Refer the below link:
https://github.com/keras-team/keras/blob/b3ffea6602dbbb481e82312baa24fe657de83e11/keras/layers/normalization/layer_normalization.py#L290C25-L290C25

Please go through the source code of [LayerNormalization](https://github.com/keras-team/keras/blob/v2.13.1/keras/layers/normalization/layer_normalization.py) and [BatchNormalization](https://github.com/keras-team/keras/blob/v2.13.1/keras/layers/normalization/batch_normalization.py) for better understanding.

Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61287">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61287">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

tf2.12.0

### Custom code

Yes

### OS platform and distribution

MacOs

### Mobile device

_No response_

### Python version

3.9

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

#### Output

```
tf.Tensor(
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]], shape=(3, 4), dtype=float32)
tf.Tensor(
[[nan nan nan nan]
 [nan nan nan nan]
 [nan nan nan nan]], shape=(3, 4), dtype=float32)
tf.Tensor(
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]], shape=(3, 4), dtype=float32)
```

#### Document

epsilon is used to avoid division by zero errors, but when LayerNormalization takes a zero vector as input, it returns nan, and BatchNormalization returns 0

（p.s.epsilon is documented as a small floating-point number.In our experiments, we found that epsilon works with larger floating-point numbers.Is this a bit vague?）

### Standalone code to reproduce the issue

```shell
#### Standalone code


import tensorflow as tf
data = tf.zeros(shape=(3, 4))
print(data)

layer1 = tf.keras.layers.LayerNormalization(axis=1, epsilon=0)
output1 = layer1(data)
print(output1)

layer2 = tf.keras.layers.BatchNormalization(axis=1, epsilon=0)
output2 = layer2(data)
print(output2)
```


### Relevant log output

_No response_