title: Why do we need to explicitly update the moving_mean and moving_variance in TensorFlow's Batch normalization in tf.contrib.layers.batch_norm?
body: <p><strong>To Long To Read</strong>: How can I use Batch Normalization with <code>tf.contrib.layers.batch_norm</code> without having to explicitly tell session to update the moving_statistics (<code>moving_mean</code> and <code>moving_variance</code>) or not?</p>
  
  <p>A few months ago I provided an answer to <a href="https://stackoverflow.com/questions/33949786/how-could-i-use-batch-normalization-in-tensorflow/38320613#38320613">How could I use Batch Normalization in TensorFlow?</a> and noticed a few weird details that I wanted to address. First it seems that the implementation that I provide seems repetitive with respect to the <code>is_training</code> variable. Recall my suggested code:</p>
  
  <pre><code>from tensorflow.contrib.layers.python.layers import batch_norm as batch_norm
  
  def batch_norm_layer(x,train_phase,scope_bn):
      bn_train = batch_norm(x, decay=0.999, center=True, scale=True,
      updates_collections=None,
      is_training=True,
      reuse=None, # is this right?
      trainable=True,
      scope=scope_bn)
      bn_inference = batch_norm(x, decay=0.999, center=True, scale=True,
      updates_collections=None,
      is_training=False,
      reuse=True, # is this right?
      trainable=True,
      scope=scope_bn)
      z = tf.cond(train_phase, lambda: bn_train, lambda: bn_inference)
      return z
  </code></pre>
  
  <p>in it I have a <code>train_phase</code> variable that just holds a tf boolean <code>tf.placeholder(tf.bool, name='phase_train')</code>. As you can see, it is used to decide if the batch norm layer should be in inference mode or not. However, the variable seemed a little redundant, since it seems I have two variables that specify the same thing twice. i.e. once in <code>train_phase</code> and another in <code>is_training</code>. Is that really necessary?</p>
  
  <p>I thought about it a bit and it seems I might to be able to remove the hard coded (<code>is_training=True/False</code>) with the (pseudo)code:</p>
  
  <pre><code>from tensorflow.contrib.layers.python.layers import batch_norm as batch_norm
  
  def batch_norm_layer(x,train_phase,scope_bn):
      bn = batch_norm(x, decay=0.999, center=True, scale=True,
      updates_collections=None,
      is_training=get_bool(train_phase),
      reuse=None, # is this right?
      trainable=True,
      scope=scope_bn)
      z = tf.cond(train_phase, lambda: bn, lambda: bn)
      return z
  </code></pre>
  
  <p>which seems to make the <code>train_phase</code> variable completely redundant/silly. This actually highlights my most important point, is the <code>train_phase</code> variable and <code>tf.cond(train_phase, lambda: bn_train, lambda: bn_inference)</code> even necessary? Which actually brings up my biggest complaint about the code (though I think this code might not even run because when defining the graph the placeholder train_phase might not even have a value but you get the idea).</p>
  
  <p>Honestly I find having to even explicitly define <code>train_phase</code> very dangerous because it seems very unnecessary for users to have to handle the inference/training mode of Batch Norm this explicitly. Though, "normal" users of Batch Norm should <strong>always</strong> update the <code>moving_mean</code>,<code>moving_variance</code> with the train data and any standard user of Batch Norm should not be updating <code>moving_mean</code>,<code>moving_variance</code> with test statistics at any time. Since the user is required to do:</p>
  
  <pre><code>sess.run(fetches=train_step, feed_dict={x: batch_xs, y_: batch_ys, phase_train=True})
  </code></pre>
  
  <p>it can bring cause really bad bugs for users that shouldn't even exist in the first place (at least in my opinion). Furthermore, it seems weird to have to explicitly say what the <code>phase_train</code> is because whenever one trains, one uses an optimizer, so it should be incredibly clear when that code is called that it should be true. Maybe this is a terrible idea but it feels like the optimizer or the session should be setting that to true automatically rather than relying on the user to do it right.</p>
  
  <p>I understand that sometimes users are allowed more flexibility to be more creative but I can't really appreciate how this (even for a researcher) be a good feature. Maybe I am just using the library incorrectly or being paranoic, but should the user really be forced to be so explicit when using batch norm? Is there some way around this?</p>
  
  <p>As a side point, having the <code>phase_train</code> be part of the model also makes the code be a bit more ugly and confusing than it feels necessary because it seems to me that its unavoidable to have a line of code where the session is being used to check if the batch norm flag is on or not. The code I am trying to avoid writing is the logic:</p>
  
  <pre><code>if batch_norm:
      # during training
      sess.run(fetches=train_step, feed_dict={x: batch_xs, y_: batch_ys, phase_train=True})
  else:
      # with no batch norm
      sess.run(fetches=train_step, feed_dict={x: batch_xs, y_: batch_ys})
  </code></pre>
  
  <p>it just feels totally unnecessary. It feels the during training the model should know if it should be updating the variables or not. </p>
  
  <p>As quick (really ugly) solution to the last problem with the if condition in the session, one can always define <code>phase_train</code> as part of the model (or at least as part of the graph) and accordingly set it equal to true and/or false when appropriate but when one doesn't actually use the batch norm layer, one actually <strong>does not use</strong> the <code>phase_train</code> placeholder in the model even if we set it have a value in the session.run. i.e. the sessions sets it to true or false, but when BN is not being used, it doesn't even matter what one sets it equal to since its not actually being used. Obviously, this makes the code really confusing (since one is defining some variable one doesn't even need), but I can't seem to find a way to hide the <code>phase_train</code> variable. For the moment this is what I am going for because it seems really ugly to have to split (or duplicate) my code between lines that have:</p>
  
  <pre><code>sess.run(fetches=..., feed_dict={...,phase_train=False})
  </code></pre>
  
  <p>and the ones that don't have it all:</p>
  
  <pre><code>sess.run(fetches=..., feed_dict={...})
  </code></pre>
  
  <p>Ideally I want the second solution and have batch norm work regardless if I use the silly <code>phase_train</code> variable.</p>