title: How to correctly implement dropout for convolution in TensorFlow
body: <p>According to the original paper on Dropout said regularisation method can be applied to convolution layers often improving their performance. TensorFlow function <code>tf.nn.dropout</code> supports that by having a <code>noise_shape</code> parameter to allow the user to choose which parts of the tensors will drop out independently. However, neither the paper nor the documentation give a clear explanation of which dimensions should be kept independently, and the TensorFlow explanation of how <code>noise_shape</code> works is rather unclear.</p>
  
  <blockquote>
    <p>only dimensions with noise_shape[i] == shape(x)[i] will make independent decisions.</p>
  </blockquote>
  
  <p>I would assume that for a typical CNN layer output of the shape <code>[batch_size, height, width, channels]</code> we don't want individual rows or columns to drop out by themselves, but rather whole channels (which would be equivalent to a node in a fully connected NN) independently of the examples (i.e. different channels could be dropped for different examples in a batch). Am I correct in this assumption? </p>
  
  <p>If so, how would one go about implementing dropout with such specificity using the <code>noise_shape</code> parameter? Would it be:</p>
  
  <pre><code>noise_shape=[batch_size, 1, 1, channels]
  </code></pre>
  
  <p>or:</p>
  
  <pre><code>noise_shape=[1, height, width, 1]
  </code></pre>