title: What is the issue with this implementation of gradient descent?
body: <p>I tried to implement linear regresion with gradient descent, but my error diverges to infinity. I've read over my code and still cannot figure out where I went wrong. I'm hoping someone can help me debug why this implementation of linear regression isn't working.  </p>
  
  <p>When <code>N=100</code> then there are no problems, but when <code>N=1000</code> then divergence to infinity is observed. </p>
  
  <pre><code>import numpy as np
  
  class Regression:
      def __init__(self, xs, ys, w,alpha):
          self.w = w
          self.xs = xs
          self.ys = ys
          self.a = alpha
          self.N = float(len(xs))
  
      def error(self, ys, yhat):
          return (1./self.N)*np.sum((ys-yhat)**2)
  
      def propagate(self):
          yhat = xs*w[0]+w[1]
          loss = yhat - self.ys
  
          r1 = (2./self.N)*np.sum(loss*self.xs)
          r2 = (2./self.N)*np.sum(loss)
  
          self.w[0] -= self.a*r1
          self.w[1] -= self.a*r2
  
  
  N = 600
  xs = np.arange(0,N)
  bias = np.random.sample(size=N)*10
  ys = xs * 2. + 2. + bias
  ws = np.array([0.,0.])
  
  regressor = Regression(
      xs, ys, ws,
      0.00001)
  
  for i in range(1000):
      regressor.propagate()
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>...
  2.71623180177e+286
  5.27841816362e+286
  1.02574818143e+287
  1.99332318715e+287
  3.87359919362e+287
  7.52751526171e+287
  1.46281231441e+288
  2.84266426942e+288
  5.52411274435e+288
  1.07349369184e+289
  2.0861064206e+289
  4.05390365232e+289
  7.87789858657e+289
  1.5309018532e+290
  2.97498179035e+290
  5.78124367308e+290
  1.12346161297e+291
  2.18320843611e+291
  4.24260074438e+291
  8.2445912074e+291
  1.6021607564e+292
  3.11345829619e+292
  6.05034327761e+292
  1.17575539141e+293
  2.28483026006e+293
  4.4400811218e+293
  8.62835227315e+293
  </code></pre>