title: keras BatchNormalization axis clarification
body: <p>The keras <a href="https://keras.io/layers/normalization/" rel="noreferrer"><code>BatchNormalization</code> layer</a> uses <code>axis=-1</code> as a default value and states that the feature axis is typically normalized. Why is this the case?</p>
  
  <p>I suppose this is surprising because I'm more familiar with using something like <a href="http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html" rel="noreferrer"><code>StandardScaler</code></a>, which would be equivalent to using <code>axis=0</code>. This would normalize the features individually.</p>
  
  <p>Is there a reason why samples are individually normalized by default (i.e. <code>axis=-1</code>) in keras as opposed to features?</p>
  
  <p><strong>Edit: example for concreteness</strong></p>
  
  <p>It's common to transform data such that each feature has zero mean and unit variance. Let's just consider the "zero mean" part with this mock dataset, where each row is a sample:</p>
  
  <pre><code>&gt;&gt;&gt; data = np.array([[   1,   10,  100, 1000],
                       [   2,   20,  200, 2000],
                       [   3,   30,  300, 3000]])
  
  &gt;&gt;&gt; data.mean(axis=0)
  array([    2.,    20.,   200.,  2000.])
  
  &gt;&gt;&gt; data.mean(axis=1)
  array([ 277.75,  555.5 ,  833.25])
  </code></pre>
  
  <p>Wouldn't it make more sense to subtract the <code>axis=0</code> mean, as opposed to the <code>axis=1</code> mean? Using <code>axis=1</code>, the units and scales can be completely different.</p>
  
  <p>Edit 2:</p>
  
  <p>The first equation of section 3 in <a href="https://arxiv.org/pdf/1502.03167.pdf" rel="noreferrer">this paper</a> seems to imply that <code>axis=0</code> should be used for calculating expectations and variances for each feature individually, assuming you have an (m, n) shaped dataset where m is the number of samples and n is the number of features.</p>
  
  <p>Edit 3: another example</p>
  
  <p>I wanted to see the dimensions of the means and variances <code>BatchNormalization</code> was calculating on a toy dataset:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  from sklearn.datasets import load_iris
  
  from keras.optimizers import Adam
  from keras.models import Model
  from keras.layers import BatchNormalization, Dense, Input
  
  
  iris = load_iris()
  X = iris.data
  y = pd.get_dummies(iris.target).values
  
  input_ = Input(shape=(4, ))
  norm = BatchNormalization()(input_)
  l1 = Dense(4, activation='relu')(norm)
  output = Dense(3, activation='sigmoid')(l1)
  
  model = Model(input_, output)
  model.compile(Adam(0.01), 'categorical_crossentropy')
  model.fit(X, y, epochs=100, batch_size=32)
  
  bn = model.layers[1]
  bn.moving_mean  # &lt;tf.Variable 'batch_normalization_1/moving_mean:0' shape=(4,) dtype=float32_ref&gt;
  </code></pre>
  
  <p>The input X has shape (150, 4), and the <code>BatchNormalization</code> layer calculated 4 means, which means it operated over <code>axis=0</code>.</p>
  
  <p>If <code>BatchNormalization</code> has a default of <code>axis=-1</code> then shouldn't there be 150 means?</p>