title: Tensorflow: What exact formula is applied in `tf.nn.sparse_softmax_cross_entropy_with_logits`?
body: <p>I tried to manually recompute the outputs of this function so I created a minimal example:  </p>
  
  <pre><code>logits = tf.pack(np.array([[[[0,1,2]]]],dtype=np.float32)) # img of shape (1, 1, 1, 3)
  labels = tf.pack(np.array([[[1]]],dtype=np.int32)) # gt of shape (1, 1, 1)
  
  softmaxCrossEntropie = tf.nn.sparse_softmax_cross_entropy_with_logits(logits,labels)
  softmaxCrossEntropie.eval() # --&gt; output is [1.41]
  </code></pre>
  
  <p>Now according to my own calculation I only get [1.23]
  When manually calculating, I'm simply applying softmax</p>
  
  <p><a href="https://i.stack.imgur.com/rFFsi.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/rFFsi.jpg" alt="enter image description here"></a></p>
  
  <p>and cross-entropy:  </p>
  
  <p><a href="https://i.stack.imgur.com/HYciw.png" rel="noreferrer"><img src="https://i.stack.imgur.com/HYciw.png" alt="enter image description here"></a></p>
  
  <p>where <code>q(x) = sigma(x_j) or (1-sigma(x_j))</code> depending whether j is the correct ground truth class or not  and <code>p(x) = labels</code> which are then one-hot-encoded</p>
  
  <p>I'm not sure where the difference might originate from. I cannot really imagine that some epsilon causes such a big difference. Does someone know where I can lookup, which exact formula is used by tensorflow?
  Is the source code of that exact part available?<br>
  I could only find <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/nn_ops.py" rel="noreferrer"><code>nn_ops.py</code></a>, but it only uses another function called <code>gen_nn_ops._sparse_softmax_cross_entropy_with_logits</code> which I couldn't find on github...</p>