title: Stochastic Gradient Descent Convergence Criteria
body: <p>Currently my convergence criteria for SGD checks whether the MSE error ratio is within a specific boundary.</p>
  
  <pre><code>def compute_mse(data, labels, weights):
      m = len(labels)
      hypothesis = np.dot(data,weights)
      sq_errors = (hypothesis - labels) ** 2
      mse = np.sum(sq_errors)/(2.0*m)
      return mse
  
  cur_mse = 1.0
  prev_mse = 100.0
  m = len(labels)
  while cur_mse/prev_mse &lt; 0.99999:
      prev_mse = cur_mse
  
      for i in range(m):
          d = np.array(data[i])
          hypothesis = np.dot(d, weights)
          gradient = np.dot((labels[i] - hypothesis), d)/m
          weights = weights + (alpha * gradient)
  
      cur_mse = compute_mse(data, labels, weights)
      if cur_mse &gt; prev_mse:
          return 
  </code></pre>
  
  <p>The weights are update w.r.t. to a single data point in the training set.</p>
  
  <p>With an alpha of 0.001, the model is supposed to have converged within a few iterations however I get no convergence. Is this convergence criteria too strict?</p>