title: Gradient of a Loss Function for an SVM
body: <p>I'm working on <a href="http://cs231n.github.io/" rel="noreferrer">this</a> class on convolutional neural networks.  I've been trying to implement the gradient of a loss function for an svm and (I have a copy of the solution) I'm having trouble understanding why the solution is correct.</p>
  
  <p>On <a href="http://cs231n.github.io/optimization-1/#gradcompute" rel="noreferrer">this</a> page it defines the gradient of the loss function to be as follows:
  <a href="https://i.stack.imgur.com/Ot1nE.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Ot1nE.png" alt="Class course notes of cs231n"></a>
  In my code I my analytic gradient matches with the numeric one when implemented in code as follows:</p>
  
  <pre><code> dW = np.zeros(W.shape) # initialize the gradient as zero
  
    # compute the loss and the gradient
    num_classes = W.shape[1]
    num_train = X.shape[0]
    loss = 0.0
    for i in xrange(num_train):
      scores = X[i].dot(W)
      correct_class_score = scores[y[i]]
      for j in xrange(num_classes):
        if j == y[i]:
          if margin &gt; 0:
              continue
        margin = scores[j] - correct_class_score + 1 # note delta = 1
        if margin &gt; 0:
          dW[:, y[i]] += -X[i]
          dW[:, j] += X[i] # gradient update for incorrect rows
          loss += margin
  </code></pre>
  
  <p>However, it seems like, from the notes, that <code>dW[:, y[i]]</code> should be changed every time <code>j == y[i]</code> since we subtract the the loss whenever <code>j == y[i]</code>.  I'm very confused why the code is not:</p>
  
  <pre><code>  dW = np.zeros(W.shape) # initialize the gradient as zero
  
    # compute the loss and the gradient
    num_classes = W.shape[1]
    num_train = X.shape[0]
    loss = 0.0
    for i in xrange(num_train):
      scores = X[i].dot(W)
      correct_class_score = scores[y[i]]
      for j in xrange(num_classes):
        if j == y[i]:
          if margin &gt; 0:
              dW[:, y[i]] += -X[i]
              continue
        margin = scores[j] - correct_class_score + 1 # note delta = 1
        if margin &gt; 0:
          dW[:, j] += X[i] # gradient update for incorrect rows
          loss += margin
  </code></pre>
  
  <p>and the loss would change when <code>j == y[i]</code>.  Why are they both being computed when <code>J != y[i]</code>?</p>