title: How to implement the Softmax function in Python
body: <p>From the <a href="https://www.udacity.com/course/viewer#!/c-ud730/l-6370362152/m-6379811820">Udacity's deep learning class</a>, the softmax of y_i is simply the exponential divided by the sum of exponential of the whole Y vector:</p>
  
  <p><a href="https://i.stack.imgur.com/iP8Du.png"><img src="https://i.stack.imgur.com/iP8Du.png" alt="enter image description here"></a></p>
  
  <p>Where <code>S(y_i)</code> is the softmax function of <code>y_i</code> and <code>e</code> is the exponential and <code>j</code> is the no. of columns in the input vector Y.</p>
  
  <p>I've tried the following:</p>
  
  <pre><code>import numpy as np
  
  def softmax(x):
      """Compute softmax values for each sets of scores in x."""
      e_x = np.exp(x - np.max(x))
      return e_x / e_x.sum()
  
  scores = [3.0, 1.0, 0.2]
  print(softmax(scores))
  </code></pre>
  
  <p>which returns:</p>
  
  <pre><code>[ 0.8360188   0.11314284  0.05083836]
  </code></pre>
  
  <p>But the suggested solution was:</p>
  
  <pre><code>def softmax(x):
      """Compute softmax values for each sets of scores in x."""
      return np.exp(x) / np.sum(np.exp(x), axis=0)
  </code></pre>
  
  <p>which produces the <strong>same output as the first implementation</strong>, even though the first implementation explicitly takes the difference of each column and the max and then divides by the sum.</p>
  
  <p><strong>Can someone show mathematically why? Is one correct and the other one wrong?</strong></p>
  
  <p><strong>Are the implementation similar in terms of code and time complexity? Which is more efficient?</strong></p>