title: Why do higher learning rates in logistic regression produce NaN costs?
body: <h1>Summary</h1>
  
  <p>I am building a classifier for spam vs. ham emails using Octave and the Ling-Spam corpus; my method of classification is logistic regression.</p>
  
  <p>Higher learning rates lead to NaN values being calculated for the cost, yet it does not break/decrease the performance of the classifier itself.</p>
  
  <h1>My Attempts</h1>
  
  <p><strong>NB: My dataset is already normalised using mean normalisation.</strong>
  When trying to choose my learning rate, I started with it as 0.1 and 400 iterations. This resulted in the following plot:</p>
  
  <h1>1 - Graph 1</h1>
  
  <p><img src="https://puu.sh/CKgUN/5a6025f2e0.png" alt=""></p>
  
  <p>When he lines completely disappear after a few iterations, it is due to a NaN value being produced; I thought this would result in broken parameter values and thus bad accuracy, but when checking the accuracy, I saw it was 95% on the test set (meaning that gradient descent was apparently still functioning). I checked different values of the learning rate and iterations to see how the graphs changed:</p>
  
  <h1>2 - Graph 2</h1>
  
  <p><img src="https://puu.sh/CKh14/420932ef70.png" alt=""></p>
  
  <p>The lines no longer disappeared, meaning no NaN values, BUT the accuracy was 87% which is substantially lower.</p>
  
  <p>I did two more tests with more iterations and a slightly higher learning rate, and in both of them, the graphs both decreased with iterations as expected, but the accuracy was ~86-88%. No NaNs there either.</p>
  
  <p>I realised that my dataset was skewed, with only 481 spam emails and 2412 ham emails. I therefore calculated the FScore for each of these different combinations, hoping to find the later ones had a higher FScore and the accuracy was due to the skew. That was not the case either - I have summed up my results in a table:</p>
  
  <h1>3 - Table</h1>
  
  <p><img src="https://puu.sh/CL2CK/62e0367a9d.png" alt=""></p>
  
  <p>So there is no overfitting and the skew does not seem to be the problem; I don't know what to do now!</p>
  
  <p><del>The only thing I can think of is that my calculations for accuracy and FScore are wrong, or that my initial debugging of the line 'disappearing' was wrong.</del></p>
  
  <p><strong>EDIT: This question is crucially about <em>why</em> the NaN values occur for <em>those chosen learning rates</em>. So the temporary fix I had of lowering the learning rate did not really answer my question - I always thought that higher learning rates simply diverged instead of converging, <em>not</em> producing NaN values.</strong></p>
  
  <h1>My Code</h1>
  
  <p>My main.m code (bar getting the dataset from files):</p>
  
  <pre><code>numRecords = length(labels);
  
  trainingSize = ceil(numRecords*0.6);
  CVSize = trainingSize + ceil(numRecords*0.2);
  
  featureData = normalise(data);
  
  featureData = [ones(numRecords, 1), featureData];
  
  numFeatures = size(featureData, 2);
  
  featuresTrain = featureData(1:(trainingSize-1),:);
  featuresCV = featureData(trainingSize:(CVSize-1),:);
  featuresTest = featureData(CVSize:numRecords,:);
  
  labelsTrain = labels(1:(trainingSize-1),:);
  labelsCV = labels(trainingSize:(CVSize-1),:);
  labelsTest = labels(CVSize:numRecords,:);
  
  paramStart = zeros(numFeatures, 1);
  
  learningRate = 0.0001;
  iterations = 400;
  
  [params] = gradDescent(featuresTrain, labelsTrain, learningRate, iterations, paramStart, featuresCV, labelsCV);
  
  threshold = 0.5;
  [accuracy, precision, recall] = predict(featuresTest, labelsTest, params, threshold);
  fScore = (2*precision*recall)/(precision+recall);
  </code></pre>
  
  <p>My gradDescent.m code:</p>
  
  <pre><code>function [optimParams] = gradDescent(features, labels, learningRate, iterations, paramStart, featuresCV, labelsCV)
  
  x_axis = [];
  J_axis = [];
  J_CV = [];
  
  params = paramStart;
  
  for i=1:iterations,
    [cost, grad] = costFunction(features, labels, params);
    [cost_CV] = costFunction(featuresCV, labelsCV, params);
  
    params = params - (learningRate.*grad);
  
    x_axis = [x_axis;i];
    J_axis = [J_axis;cost];
    J_CV = [J_CV;cost_CV];
  endfor
  
  graphics_toolkit("gnuplot")
  plot(x_axis, J_axis, 'r', x_axis, J_CV, 'b');
  legend("Training", "Cross-Validation");
  xlabel("Iterations");
  ylabel("Cost");
  title("Cost as a function of iterations");
  
  optimParams = params;
  endfunction
  </code></pre>
  
  <p>My costFunction.m code:</p>
  
  <pre><code>function [cost, grad] = costFunction(features, labels, params)
    numRecords = length(labels);
  
    hypothesis = sigmoid(features*params);
  
    cost = (-1/numRecords)*sum((labels).*log(hypothesis)+(1-labels).*log(1-hypothesis));
  
    grad = (1/numRecords)*(features'*(hypothesis-labels));
  endfunction
  </code></pre>
  
  <p>My predict.m code:</p>
  
  <pre><code>function [accuracy, precision, recall] = predict(features, labels, params, threshold)
  numRecords=length(labels);
  
  predictions = sigmoid(features*params)&gt;threshold;
  
  correct = predictions == labels;
  
  truePositives = sum(predictions == labels == 1);
  falsePositives = sum((predictions == 1) != labels);
  falseNegatives = sum((predictions == 0) != labels);
  
  precision = truePositives/(truePositives+falsePositives);
  recall = truePositives/(truePositives+falseNegatives);
  
  accuracy = 100*(sum(correct)/numRecords);
  endfunction
  </code></pre>