title: Is my implementation of stochastic gradient descent correct?
body: <p>I am trying to develop stochastic gradient descent, but I don't know if it is 100% correct.</p>
  
  <ul>
  <li>The cost generated by my stochastic gradient descent algorithm is sometimes very far from the one generated by FMINUC or Batch gradient descent.</li>
  <li>while batch gradient descent cost converge when I set a learning rate alpha of 0.2, I am forced to set a learning rate alpha of 0.0001 for my stochastic implementation for it not to diverge. Is this normal?  </li>
  </ul>
  
  <p>Here are some results I obtained with a training set of 10,000 elements and num_iter = 100 or 500</p>
  
  <pre><code>    FMINUC : 
      Iteration  #100 | Cost: 5.147056e-001
  
      BACTH GRADIENT DESCENT  500 ITER
      Iteration #500 - Cost = 5.535241e-001
  
      STOCHASTIC GRADIENT DESCENT 100 ITER
      Iteration #100 - Cost = 5.683117e-001  % First time I launched
      Iteration #100 - Cost = 7.047196e-001  % Second time I launched
  </code></pre>
  
  <p><strong>Gradient descent implementation for logistic regression</strong></p>
  
  <pre><code>J_history = zeros(num_iters, 1); 
  
  for iter = 1:num_iters 
  
      [J, gradJ] = lrCostFunction(theta, X, y, lambda);
      theta = theta - alpha * gradJ;
      J_history(iter) = J;
  
      fprintf('Iteration #%d - Cost = %d... \r\n',iter, J_history(iter));
  end
  </code></pre>
  
  <p><strong>Stochastic gradient descent implementation for logistic regression</strong></p>
  
  <pre><code>% number of training examples
  m = length(y);
  
  % STEP1 : we shuffle the data
  data = [y, X];
  data = data(randperm(size(data,1)),:);
  y = data(:,1);
  X = data(:,2:end);
  
  for iter = 1:num_iters 
  
       for i = 1:m
          x = X(i,:); % Select one example
          [J, gradJ] = lrCostFunction(theta, x, y(i,:), lambda);
          theta = theta - alpha * gradJ;
       end
  
       J_history(iter) = J;
       fprintf('Iteration #%d - Cost = %d... \r\n',iter, J);
  
  end
  </code></pre>
  
  <p><em>For reference, here is the logistic regression cost function used in my example</em></p>
  
  <pre><code>function [J, grad] = lrCostFunction(theta, X, y, lambda)
  
  m = length(y); % number of training examples
  
  % We calculate J    
  hypothesis = sigmoid(X*theta); 
  costFun = (-y.*log(hypothesis) - (1-y).*log(1-hypothesis));    
  J = (1/m) * sum(costFun) + (lambda/(2*m))*sum(theta(2:length(theta)).^2);
  
  % We calculate grad using the partial derivatives
  beta = (hypothesis-y); 
  grad = (1/m)*(X'*beta);
  temp = theta;  
  temp(1) = 0;   % because we don't add anything for j = 0  
  grad = grad + (lambda/m)*temp; 
  grad = grad(:);
  
  end
  </code></pre>