title: DerivativeCheck fails with minFunc
body: <p>I'm trying to train a single layer of an autoencoder using minFunc, and while the cost function appears to decrease, when enabled, the DerivativeCheck fails. The code I'm using is  as close to textbook values as possible, though extremely simplified.</p>
  
  <p>The loss function I'm using is the squared-error:</p>
  
  <p>$ J(W; x) = \frac{1}{2}||a^{l} - x||^2 $</p>
  
  <p>with $a^{l}$ equal to $\sigma(W^{T}x)$, where $\sigma$ is the sigmoid function. The gradient should therefore be:</p>
  
  <p>$ \delta = (a^{l} - x)*a^{l}(1 - a^{l}) $</p>
  
  <p>$ \nabla_{W} = \delta(a^{l-1})^T $</p>
  
  <p>Note, that to simplify things, I've left off the bias altogether. While this will cause poor performance, it shouldn't affect the gradient check, as I'm only looking at the weight matrix. Additionally, I've tied the encoder and decoder matrices, so there is effectively a single weight matrix.</p>
  
  <p>The code I'm using for the loss function is (<strong>edit:</strong> I've vectorized the loop I had and cleaned code up a little):</p>
  
  <pre><code>% loss function passed to minFunc
  function [ loss, grad ] = calcLoss(theta, X, nHidden)
    [nInstances, nVars] = size(X);
  
    % we get the variables a single vector, so need to roll it into a weight matrix
    W = reshape(theta(1:nVars*nHidden), nVars, nHidden);
    Wp = W; % tied weight matrix
  
    % encode each example (nInstances)
    hidden = sigmoid(X*W);
  
    % decode each sample (nInstances)
    output = sigmoid(hidden*Wp);
  
    % loss function: sum(-0.5.*(x - output).^2)
    % derivative of loss: -(x - output)*f'(o)
    % if f is sigmoid, then f'(o) = output.*(1-output)
    diff = X - output;
    error = -diff .* output .* (1 - output);
    dW = hidden*error';
  
    loss = 0.5*sum(diff(:).^2, 2) ./ nInstances;
  
    % need to unroll gradient matrix back into a single vector
    grad = dW(:) ./ nInstances;
  end
  </code></pre>
  
  <p>Below is the code I use to run the optimizer (for a single time, as the runtime is fairly long with all training samples):</p>
  
  <pre><code>examples = 5000;
  fprintf('loading data..\n');
  images = readMNIST('train-images-idx3-ubyte', examples) / 255.0;
  
  data = images(:, :, 1:examples);
  
  % each row is a different training sample
  X = reshape(data, examples, 784);
  
  % initialize weight matrix with random values
  % W: (R^{784} -&gt; R^{10}), W': (R^{10} -&gt; R^{784})
  numHidden = 10; % NOTE: this is extremely small to speed up DerivativeCheck
  numVisible = 784;
  low = -4*sqrt(6./(numHidden + numVisible));
  high = 4*sqrt(6./(numHidden + numVisible));
  W = low + (high-low)*rand(numVisible, numHidden);
  
  % run optimization
  options = {};
  options.Display = 'iter';
  options.GradObj = 'on';
  options.MaxIter = 10;
  mfopts.MaxFunEvals = ceil(options.MaxIter * 2.5);
  options.DerivativeCheck = 'on';
  options.Method = 'lbfgs';    
  [ x, f, exitFlag, output] = minFunc(@calcLoss, W(:), options, X, numHidden);
  </code></pre>
  
  <p>The results I get with the DerivitiveCheck on are generally less than 0, but greater than 0.1. I've tried similar code using batch gradient descent, and get slightly better results (some are &lt; 0.0001, but certainly not all).</p>
  
  <p>I'm not sure if I made either a mistake with my math or code. Any help would be greatly appreciated!</p>
  
  <p><strong>update</strong></p>
  
  <p>I discovered a small typo in my code (which doesn't appear in the code below) causing exceptionally bad performance. Unfortunately, I'm still getting getting less-than-good results. For example, comparison between the two gradients:</p>
  
  <pre><code>calculate     check
  0.0379        0.0383
  0.0413        0.0409
  0.0339        0.0342
  0.0281        0.0282
  0.0322        0.0320
  </code></pre>
  
  <p>with differences of up to 0.04, which I'm assuming is still failing.</p>