title: How could I use batch normalization with tensorflow slim?
body: <p>I've found a few questions on using batch normalization in TensorFlow, but none of them are about it's wrapper in slim.</p>
  
  <p>I'm trying to use batch normalization to train a MNIST digit classifier. While the training performance is high enough, the validation or test performance is poor.</p>
  
  <p>I built only one graph, and passed <code>is_training</code> as a tf.placeholder, just like this (BN is used in every conv and fc layer):</p>
  
  <pre><code>is_training = tf.placeholder(tf.bool, [])
  x_image = tf.reshape(x, [-1, 28, 28, 1])
  with slim.arg_scope([slim.conv2d, slim.fully_connected],
                      normalizer_fn=slim.batch_norm,
                      normalizer_params={'is_training': is_training}):
      conv1 = slim.conv2d(x_image, 32, [5, 5], scope='conv1')
      pool1 = slim.max_pool2d(conv1, [2, 2], scope='pool1')
      conv2 = slim.conv2d(pool1, 64, [5, 5], scope='conv2')
      pool2 = slim.max_pool2d(conv2, [2, 2], scope='pool2')
      flatten = slim.flatten(pool2)
      fc = slim.fully_connected(flatten, 1024, scope='fc1')
      drop = slim.dropout(fc, keep_prob=keep_prob)
      logits = slim.fully_connected(drop, 10, activation_fn=None, scope='logits')
  </code></pre>
  
  <p>I also added control dependencies as following:</p>
  
  <pre><code>update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
  if update_ops:
      updates = tf.group(*update_ops)
      cross_entropy = control_flow_ops.with_dependencies([updates], cross_entropy)
  </code></pre>
  
  <p>For the training phase, I use:</p>
  
  <pre><code>sess.run([net['cross_entropy'], net['accuracy']],
                                  feed_dict={net['x']: batch_xs,
                                             net['y_']: batch_ys,
                                             net['keep_prob']: 1.0,
                                             net['is_training']: True})
  </code></pre>
  
  <p>For the validation phase, I use:</p>
  
  <pre><code>sess.run(net['accuracy'], feed_dict={net['x']: batch_xs,
                                                  net['y_']: batch_ys,
                                                  net['keep_prob']: 1.0,
                                                  net['is_training']: False})
  </code></pre>
  
  <p>For test purpose, I dump the trained model to a checkpoint, then pass <code>is_training</code> as False. Again, it's performance is poor.</p>
  
  <p>So what's wrong with it? Is it about the <code>reuse</code> parameter? Or I need to maintain <code>gamma</code> and <code>beta</code> variables in BN layer myself?</p>
  
  <p>For ease of reproducity, this is my code(set <code>phase</code> to <code>train</code> to train a model and validate, <code>test</code> to restore from checkpoint and test):
  <a href="https://github.com/soloice/mnist-bn/blob/master/mnist_bn.py" rel="nofollow noreferrer">https://github.com/soloice/mnist-bn/blob/master/mnist_bn.py</a></p>