title: Vectorized softmax gradient
body: <p>I have a softmax layer (only the activation itself, without the linear part of multiplying inputs by weights), and I want to make for it a backward pass.</p>
  
  <p>I have found many tutorials/answers on SO that deals with it, but they all seemed to use <code>X</code> as <code>(1, n_inputs)</code> vector. I want to use it as <code>(n_samples, n_inputs)</code> array, and still, have a correct vectorized implementation of the forward/backward pass.</p>
  
  <p>I have written a following forward pass, normalizing the output for each row/sample (is it correct?):</p>
  
  <pre><code>import numpy as np
  
  X = np.asarray([
      [0.0, 0.0],
      [0.0, 1.0],
      [1.0, 0.0],
      [1.0, 1.0]], dtype=np.float32)
  
  def prop(self, X):
      s = np.exp(X)
      s = s.T / np.sum(s, axis=1)
      return s.T
  </code></pre>
  
  <p>It gives me the final result of forward propagation (including other layers) as:</p>
  
  <pre><code>Y = np.asarray([
         [0.5       , 0.5       ],
         [0.87070241, 0.12929759],
         [0.97738616, 0.02261384],
         [0.99200957, 0.00799043]], dtype=np.float32))
  </code></pre>
  
  <p>So, this is the output of the softmax, if it is correct. Now, how should I write the backward pass?</p>
  
  <p>I have derived the derivative of the softmax to be:</p>
  
  <p>1) if <code>i=j</code>: <code>p_i*(1 - p_j)</code>,</p>
  
  <p>2) if <code>i!=j</code>: <code>-p_i*p_j</code>,</p>
  
  <p>where <a href="https://i.stack.imgur.com/4mOlF.gif" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4mOlF.gif" alt="equation"></a></p>
  
  <p>I've tried to compute the derivative as:</p>
  
  <pre><code>ds = np.diag(Y.flatten()) - np.outer(Y, Y) 
  </code></pre>
  
  <p>But it results in the 8x8 matrix which does not make sense for the following backpropagation... What is the correct way to write it?</p>