title: What is the problem with my implementation of the cross-entropy function?
body: <p>I am learning the neural network and I want to write a function <code>cross_entropy</code> in python. Where it is defined as</p>
  
  <p><a href="https://i.stack.imgur.com/GKdbq.png" rel="noreferrer"><img src="https://i.stack.imgur.com/GKdbq.png" alt="cross entropy"></a></p>
  
  <p>where <code>N</code> is the number of samples, <code>k</code> is the number of classes, <em><code>log</code></em> is the natural logarithm, <code>t_i,j</code> is 1 if sample <code>i</code> is in class <code>j</code> and <code>0</code> otherwise, and <code>p_i,j</code> is the predicted probability that sample <code>i</code> is in class <code>j</code>.
  To avoid numerical issues with logarithm, clip the predictions to <code>[10^{−12}, 1 − 10^{−12}]</code> range.</p>
  
  <p>According to the above description, I wrote down the codes by clipping the predictions to <code>[epsilon, 1 − epsilon]</code> range, then computing the cross_entropy based on the above formula.</p>
  
  <pre><code>def cross_entropy(predictions, targets, epsilon=1e-12):
      """
      Computes cross entropy between targets (encoded as one-hot vectors)
      and predictions. 
      Input: predictions (N, k) ndarray
             targets (N, k) ndarray        
      Returns: scalar
      """
      predictions = np.clip(predictions, epsilon, 1. - epsilon)
      ce = - np.mean(np.log(predictions) * targets) 
      return ce
  </code></pre>
  
  <p>The following code will be used to check if the function <code>cross_entropy</code> are correct.</p>
  
  <pre><code>predictions = np.array([[0.25,0.25,0.25,0.25],
                          [0.01,0.01,0.01,0.96]])
  targets = np.array([[0,0,0,1],
                    [0,0,0,1]])
  ans = 0.71355817782  #Correct answer
  x = cross_entropy(predictions, targets)
  print(np.isclose(x,ans))
  </code></pre>
  
  <p>The output of the above codes is False, that to say my codes for defining the function <code>cross_entropy</code> is not correct. Then I print the result of <code>cross_entropy(predictions, targets)</code>. It gave <code>0.178389544455</code> and the correct result should be <code>ans = 0.71355817782</code>. Could anybody help me to check what is the problem with my codes?</p>