title: tensorflow: understanding cross entropy calculation with reduce_mean and reduce_sum
body: <p>I was looking at the Tensorflow's basic neural network for beginners [1]. I am having trouble understanding the calculation of the entropy value and how its used. In the example a place holder is created to hold the correct labels:</p>
  
  <pre><code>y_ = tf.placeholder(tf.float32, [None, 10])
  </code></pre>
  
  <p>and the cross-entropy, sum y'.log(y), is calculated as follows:</p>
  
  <pre><code>reduct = -tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1])
  cross_entropy = tf.reduce_mean( reduct )
  </code></pre>
  
  <p>Looking at the dimensions I assume we have (element wise multiplication):</p>
  
  <p>y_ * log(y) = [batch x classes ] x [batch x classes ]</p>
  
  <p>y_ * log(y) = [batch x classes ] </p>
  
  <p>And a quick check confirms this:</p>
  
  <pre><code>y_ * tf.log(y)
  &lt;tf.Tensor 'mul_8:0' shape=(?, 10) dtype=float32&gt;
  </code></pre>
  
  <p>Now here is what I don't understand. My understanding is that for cross-entropy we need to consider the distributions of <code>y</code> (predicted) and <code>y_</code> (oracle). So I assume that we first need to <strong><code>reduce_mean</code></strong> of the y and the <code>y_</code> by their columns (by class). I would then get 2 vectors of size:</p>
  
  <p>y_ = [classes x 1 ]</p>
  
  <p>y = [classes x 1 ]</p>
  
  <p>Since y_ is the "correct" distribution, we then do a (notice that in the example the vectors are flipped):</p>
  
  <p>log(y_) = [ classes x 1 ]</p>
  
  <p>And now we do an element wise multiplication:</p>
  
  <p>y x log(y_)</p>
  
  <p>Which gives us a vector with the length of the classes. And finally we simply sum this vector to get a single value:</p>
  
  <p>Hy(y_) = sum( y x log(y_) )</p>
  
  <p>However, this does not seem to be the calculations that are being performed. Can anyone explain were my error is? Maybe point me to some page with a good explanation. In addition to this we are using one-hot encoding. So log(1) = 0 and log(0) = -infinity so this will cause errors in the calculations. I understand that the optimizer will calculate the derivatives, but isn't the cross-entropy still calculated?</p>
  
  <p>TIA.</p>
  
  <p>[1] <a href="https://www.tensorflow.org/versions/r0.9/tutorials/mnist/beginners/index.html" rel="nofollow">https://www.tensorflow.org/versions/r0.9/tutorials/mnist/beginners/index.html</a></p>