title: Python, GD and SGD implementation on Linear Regression
body: <p>I try to understand and implement these algorithms in a simple example of linear regression. It is clear to me that the full batch gradient descent uses all the data to calculate the gradient and that the stochastic gradient descent uses only one.</p>
  
  <p>Full Batch Gradient Descent:</p>
  
  <pre><code>import pandas as pd
  from math import sqrt
  
  df = pd.read_csv("data.csv")
  df = df.sample(frac=1)
  X = df['X'].values
  y = df['y'].values
  
  m_current=0
  b_current=0
  
  epochs=100000
  learning_rate=0.0001
  N = float(len(y))
  
  for i in range(epochs):
      y_current = (m_current * X) + b_current
      cost = sum([data**2 for data in (y-y_current)]) / N
      rmse = sqrt(cost)
  
      m_gradient = -(2/N) * sum(X * (y - y_current))
      b_gradient = -(2/N) * sum(y - y_current)
  
      m_current = m_current - (learning_rate * m_gradient)
      b_current = b_current - (learning_rate * b_gradient)
  
  print("RMSE: ", rmse)
  </code></pre>
  
  <p>Full Batch Gradient Descent output <code>RMSE:  10.597894381512043</code></p>
  
  <p>Now I tried to implement Stochastic Gradient Descent on this code, and it looks like this:</p>
  
  <pre><code>import pandas as pd
  from math import sqrt
  
  df = pd.read_csv("data.csv")
  df = df.sample(frac=1)
  X = df['X'].values
  y = df['y'].values
  
  m_current=0
  b_current=0
  
  epochs=100000
  learning_rate=0.0001
  N = float(len(y))
  
  mini = df.sample(n=1) # get one random row from dataset
  
  X_mini = mini['X'].values
  y_mini = mini['y'].values
  
  for i in range(epochs):
      y_current = (m_current * X) + b_current
      cost = sum([data**2 for data in (y-y_current)]) / N
      rmse = sqrt(cost)
  
      m_gradient = -(2/N) * (X_mini * (y_mini - y_current))
      b_gradient = -(2/N) * (y_mini - y_current)
  
      m_current = m_current - (learning_rate * m_gradient)
      b_current = b_current - (learning_rate * b_gradient)
  
  print("RMSE: ", rmse)
  </code></pre>
  
  <p>Outputs: <code>RMSE:  27.941268469783633</code>, <code>RMSE:  20.919246260939282</code>, <code>RMSE:  31.100985268167648</code>, <code>RMSE:  21.023479528518386</code>, <code>RMSE:  19.920972478204785</code>...</p>
  
  <p>The results I got using sklearn SGDRegressor (with the same settings): </p>
  
  <pre><code>import pandas as pd
  from sklearn import linear_model
  from sklearn.metrics import mean_squared_error
  from math import sqrt
  
      data= pd.read_csv('data.csv')
  
      x = data.X.values.reshape(-1,1)
      y = data.y.values.reshape(-1,1).ravel()
  
      Model = linear_model.SGDRegressor(alpha = 0.0001, shuffle=True, max_iter = 100000)
      Model.fit(x,y)
      y_predicted = Model.predict(x)
  
      mse = mean_squared_error(y, y_predicted)
      print("RMSE: ", sqrt(mse))
  </code></pre>
  
  <p>Otuputs: <code>RMSE:  10.995881334048224</code>, <code>RMSE:  11.75907544873036</code>, <code>RMSE:  12.981134247509486</code>, <code>RMSE:  12.298263437187988</code>, <code>RMSE:  12.549948073154608</code>...</p>
  
  <p>The results obtained by the above algorithm are worse than scikit model results.. I wonder where I made a mistake? Also my algorithm is quite slower (few seconds)..</p>