title: About tf.nn.softmax_cross_entropy_with_logits_v2
body: <p>I have noticed that <code>tf.nn.softmax_cross_entropy_with_logits_v2(labels, logits)</code> mainly performs 3 operations:</p>
  
  <ol>
  <li><p>Apply softmax to the logits (y_hat) in order to normalize them: <code>y_hat_softmax = softmax(y_hat)</code>.</p></li>
  <li><p>Compute the cross-entropy loss: <code>y_cross = y_true * tf.log(y_hat_softmax)</code></p></li>
  <li><p>Sum over different class for an instance: <code>-tf.reduce_sum(y_cross, reduction_indices=[1])</code></p></li>
  </ol>
  
  <p>The code borrowed from <a href="https://stackoverflow.com/questions/34240703/whats-the-difference-between-softmax-and-softmax-cross-entropy-with-logits">here</a> demonstrates this perfectly. </p>
  
  <pre><code>y_true = tf.convert_to_tensor(np.array([[0.0, 1.0, 0.0],[0.0, 0.0, 1.0]]))
  y_hat = tf.convert_to_tensor(np.array([[0.5, 1.5, 0.1],[2.2, 1.3, 1.7]]))
  
  # first step
  y_hat_softmax = tf.nn.softmax(y_hat)
  
  # second step
  y_cross = y_true * tf.log(y_hat_softmax)
  
  # third step
  result = - tf.reduce_sum(y_cross, 1)
  
  # use tf.nn.softmax_cross_entropy_with_logits_v2
  result_tf = tf.nn.softmax_cross_entropy_with_logits_v2(labels = y_true, logits = y_hat)
  
  with tf.Session() as sess:
      sess.run(result)
      sess.run(result_tf)
      print('y_hat_softmax:\n{0}\n'.format(y_hat_softmax.eval()))
      print('y_true: \n{0}\n'.format(y_true.eval()))
      print('y_cross: \n{0}\n'.format(y_cross.eval()))
      print('result: \n{0}\n'.format(result.eval()))
      print('result_tf: \n{0}'.format(result_tf.eval()))
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>y_hat_softmax:
  [[0.227863   0.61939586 0.15274114]
  [0.49674623 0.20196195 0.30129182]]
  
  y_true: 
  [[0. 1. 0.]
  [0. 0. 1.]]
  
  y_cross: 
  [[-0.         -0.4790107  -0.        ]
  [-0.         -0.         -1.19967598]]
  
  result: 
  [0.4790107  1.19967598]
  
  result_tf: 
  [0.4790107  1.19967598]
  </code></pre>
  
  <p>However, the one hot labels includes either 0 or 1, thus the cross entropy for such binary case is formulated as follows shown in <a href="https://datascience.stackexchange.com/questions/9302/the-cross-entropy-error-function-in-neural-networks">here</a> and <a href="https://sefiks.com/2017/12/17/a-gentle-introduction-to-cross-entropy-loss-function/" rel="noreferrer">here</a>:</p>
  
  <p><img src="https://latex.codecogs.com/gif.latex?-%5Csum_%7Bn%3D1%7D%5E%7BN%7D%5Csum_%7Bi%3D1%7D%5E%7BK%7D%28y_%7Btrue%7D%5Ei%20log%20y_%7Bpred%7D%5Ei%20&plus;%20%281-y_%7Btrue%7D%5Ei%29log%281-y_%7Bpred%7D%5Ei%29%29" alt="binary_cross_entropy"></p>
  
  <p>I write code for this formula in the next cell, the result of which is different from above. My question is which one is better or right? Does tensorflow has function to compute the cross entropy according to this formula also?</p>
  
  <pre><code>y_true = np.array([[0.0, 1.0, 0.0], [0.0, 0.0, 1.0]])
  y_hat_softmax_from_tf = np.array([[0.227863, 0.61939586, 0.15274114], 
                                [0.49674623, 0.20196195, 0.30129182]])
  comb = np.dstack((y_true, y_hat_softmax_from_tf))
  #print(comb)
  
  print('y_hat_softmax_from_tf: \n{0}\n'.format(y_hat_softmax_from_tf))
  print('y_true: \n{0}\n'.format(y_true))
  
  def cross_entropy_fn(sample):
      output = []
      for label in sample:
          if label[0]:
              y_cross_1 = label[0] * np.log(label[1])
          else:
              y_cross_1 = (1 - label[0]) * np.log(1 - label[1])
          output.append(y_cross_1)
      return output
  
  y_cross_1 = np.array([cross_entropy_fn(sample) for sample in comb])
  print('y_cross_1: \n{0}\n'.format(y_cross_1))
  
  result_1 = - np.sum(y_cross_1, 1)
  print('result_1: \n{0}'.format(result_1))
  </code></pre>
  
  <p>output</p>
  
  <pre><code>y_hat_softmax_from_tf: 
  [[0.227863   0.61939586 0.15274114]
  [0.49674623 0.20196195 0.30129182]]
  
  y_true: 
  [[0. 1. 0.]
  [0. 0. 1.]]
  
  y_cross_1: 
  [[-0.25859328 -0.4790107  -0.16574901]
  [-0.68666072 -0.225599   -1.19967598]]
  
  result_1: 
  [0.90335299 2.11193571]
  </code></pre>