MNIST 2-layer NN failing to recognize certain digits Recently I've been taking a look into some basic multilayer neural networks, and I decided to try tackling the MNIST database of handwritten digits using a 2 layer neural network with 100 neurons in the hidden layer.
My network has a hidden layer using ReLU, and the output layer uses the sigmoid function. The cost is calculated by MSE, and weights are updated using SGD. Below is the code written in MATLAB for training the network.
When training this network, I discovered that it has somehow failed to recognize every 8 and 9 in the data set. It has some problems recognizing other numbers, but not nearly as much as 8 or 9.
How can I fix this issue?
Edit: The results when training a network from a random set of weights are quite inconsistent and oftentimes have low accuracy. Is there something wrong with the backpropagation step or is the network structure not the best for digit recognition?
*dsigmoid and dReLU are the derivatives of their respective functions.
function [trainedWeights1, trainedWeights2] = twoLayerSGD(weights1, weights2, inputs, outputs, alpha, shuffle)
% Trains a 2 layer perceptron using stochastic gradient descent and MSE
%
% function [trainedWeights1, trainedWeights2] = twoLayerSGD(weights2, weights1, inputs, outputs, alpha, numHiddenLayers)
% Inputs:
%   weights1 = weights between input layer and hidden layer as a matrix
%              where row i and column j represents the weight between node
%              j of the input layer and node i of the hidden layer
%   weights2 = weights between hidden layer and output layer as a matrix
%              where row i and column j represents the weight between node j of the
%              input layer and node i of the hidden layer
%   inputs = inputs of the training data, where each set of inputs is
%            stored as a row vector
%   outputs = the correct outputs of the training data, where each set of
%             outputs is stored as a row vector
%   alpha = the learning rate, default is 0.005
%   shuffle = boolean value, determines whether the training cases will be
%             randomly ordered, default is false
%
% Outputs:
%   trainedWeights1 = the trained weight matrix for the weights between the
%                     input layer and hidden layer
%   trainedWeights2 = the trained weight matrix for the weights between the
%                     hidden layer and output layer

if ~exist('alpha','var') || isempty(alpha)
    alpha = 0.005;
end

if ~exist('shuffle','var') || isempty(shuffle)
    shuffle = false;
end

% Determining the size of the input data set
N = size(inputs,1);
fprintf('Data set size: %.0f\n', N);

% Shuffle input data
if shuffle
    fprintf('Shuffling data...\n')
    order = randperm(size(inputs, 1));
    inputs = inputs(order, :);
    outputs = outputs(order, :);
    fprintf('Shuffle complete\n')
end

% Train Model
fprintf('Starting training process...\n')
trainedWeights1 = weights1;
trainedWeights2 = weights2;

for dataSet = 1:N
    % Get inputs of current data set
    input = inputs(dataSet,:); % row vector
    
    % Calculate activations of hidden nodes and the output
    Z1 = trainedWeights1*input'; % column vector
    hiddenActivation = ReLU(Z1); % column vector
    Z2 = trainedWeights2*hiddenActivation; % column vector
    output = sigmoid(Z2)'; % row vector
    
    % Calculate the error/cost of the current weights
    correctOutput = outputs(dataSet,:); % row vector
    cost = sum((output - correctOutput).^2);
    
    % Update Weights
    trainedWeights2 = trainedWeights2 - alpha * 2*(output'-correctOutput') .* hiddenActivation' .* dsigmoid(Z2);
    S = sum(2*(output'-correctOutput') .* trainedWeights2 .* dsigmoid(Z2),1);
    trainedWeights1 = trainedWeights1 - alpha * S' .* input .* dReLU(Z1);
    
    % Display status
    if mod(dataSet, 1000) == 0
        fprintf('Data set %.0fk/%.0fk complete!\n',dataSet/1000,N/1000);
    end
end
end
Both sigmoid activation and MSE loss often don't work so well.  I would start by using standard methods, i.e., softmax and cross-entropy loss.