title: SGD diverging after changing learning rate
body: <p>I am creating a function for stochastic gradient descent with ridge regression. I am keeping the step size constant for 1800 iterations and then changing it to 1/n or 1/sqrt(n). When I use 1/sqrt(n) my loss is decreasing and almost converging. However when I use 1/n my loss decreases and then starts to increase! Can someone please help me. The code for SGC is below and also the function which I use to compute the loss for the entire batch after every update.</p>
  
  <pre><code>def stochastic_grad_descent_steptrial(x,y,thetha,alpha,num_iter,lambda_reg):
  
      loss_log=[]
      theta_log=[]
      ridge_log=[]
  
      total_loss_log=[]  # total loss for updates(for plotting)
      total_iter_count=[]
      for j in range(num_iter): # j epochs
          for i in range(x.shape[0]): 
  
  
              diff=np.dot(x.iloc[i,:],thetha)-y[i]  # loss of each i
              loss=np.sum(diff**2)+lambda_reg*np.sum(thetha**2)
              loss_log.append(loss) # append to log
  
              diff=np.dot(x.iloc[i,:],thetha)-y[i] # specific i
              grad=(2/N)*np.dot(x.iloc[i,:].T,diff)+2*lambda_reg*thetha # grad for i
  
  
              total_iter=((j+1)*(i+1)) # Total step number till now 
              total_iter_count.append(total_iter)
  
  
              if total_iter&lt;1800:        # change to step size function only after n steps 
                  step=alpha             # Can use function and step condition as hyper parameter
              else:
                  step=1/(total_iter)
  #                 step=1/np.sqrt(total_iter)
  
              thetha=thetha-step*grad #update
  
  
              #appends
              theta_log.append(thetha) 
              ridge_log.append(lambda_reg*np.sum(thetha**2))  
  
  
              # compute loss on entire data 
              total_loss=ridge_loss(x,y,thetha,lambda_reg) 
              total_loss_log.append(total_loss) # append
  
      normal_loss=cost(x,y,thetha) #final loss(without ridge)
  
      loss_log=np.array(loss_log)  # conversions to np,pd
      theta_log=pd.DataFrame(theta_log)
      ridge_log=np.array(ridge_log)
  
      return(loss_log,theta_log,ridge_log,thetha,normal_loss,total_loss_log)
  </code></pre>
  
  <p>My loss function(which is then plotted) is: </p>
  
  <pre><code>def ridge_loss(x,y,thetha,lambda_reg):
  
      diff=np.dot(x,thetha)-y
      cost=(1/N)*np.sum(diff**2)+lambda_reg*np.sum(thetha**2)
      return(cost)
  </code></pre>
  
  <p>Why is the loss increasing? (See graphs below) How can the loss increase, shouldn't it just slowly decrease anyways?</p>
  
  <p><a href="https://github.com/neha-duggirala/100DaysOfMLCode/blob/master/linear_regression/1.txt" rel="nofollow noreferrer">Link to dataset</a></p>
  
  <p>Image 1 is with constant rate, image 2 is with square root iteration 
  number and image 3 is 1 over iteration number. </p>
  
  <p><img src="https://i.stack.imgur.com/kh2aY.png" alt="enter image description here"></p>
  
  <p><img src="https://i.stack.imgur.com/Bkv5Z.png" alt="enter image description here"></p>
  
  <p><img src="https://i.stack.imgur.com/klMbV.png" alt="enter image description here"></p>