title: keras batchnorm layer moving average for variance
body: <p>I've been trying to understand the Keras BatchNorm layer behavior in my Keras NN model. One question I encountered was how the BN layer is calculating the moving average of the 'variance'. My understanding is Keras is using exponential-weighted-average method to calculate the moving average for both mean and variance from the training mini-batches. But regardless of this, after a really large number of epochs, this moving average should approach the mean/variance of the training data set. But in my simple example, the 'variance' moving average is always different from the training data 'variance'. Below is my code and output:</p>
  
  <pre><code>from keras.layers import Input, BatchNormalization
  from keras.models import Model
  from keras.optimizers import Adam, RMSprop
  
  import numpy as np
  
  X_input = Input(shape=(6,))
  X = BatchNormalization(axis=-1)(X_input)
  
  model = Model(inputs=X_input, outputs=X)
  
  model.compile(optimizer=RMSprop(), loss='mean_squared_error')
  
  np.random.seed(3)
  train_data = np.random.random((5,6))
  train_label = np.random.random((5,6))
  
  model.fit(x=train_data, y=train_label, epochs=10000, batch_size=6, verbose=False)
  
  bn_gamma, bn_beta, bn_mean, bn_var = model.layers[1].get_weights()
  train_mean = np.mean(train_data, axis=0)
  train_var = np.var(train_data, axis=0)
  
  print("train_mean: {}".format(train_mean))
  print("moving_mean: {}".format(bn_mean))
  print("train_var: {}".format(train_var))
  print("moving_var: {}".format(bn_var))
  </code></pre>
  
  <p>Below is the output:</p>
  
  <pre><code>train_mean: [0.42588575 0.47785879 0.32170309 0.49151921 0.355046   0.60104636]
  moving_mean: [0.4258843  0.47785735 0.32170165 0.49151778 0.35504454 0.60104346]
  train_var: [0.03949981 0.05228663 0.04027516 0.02522536 0.10261097 0.0838988 ]
  moving_var: [0.04938692 0.06537427 0.05035637 0.03153942 0.12829503 0.10489936]
  </code></pre>
  
  <p>If you see, the train_mean is the same as the moving average mean of BN layer, but train_var (variance) is not. Can anyone please help here? Thanks.</p>