Neural network reaching local optima I was recently trying to train a convolutional neural network to classify people as Hispanic or white (for learning purposes). I couldn't find a good dataset of just those two races, so I had to manually scrape images from the web. I ended up with 48 total images, 24 of them Hispanic and 24 white. When I trained my network, I tried many different architectures and hyperparameters, but the accuracy stayed at exactly 50%. I learned this was because the network was outputting 1 every time, classifying every image as Hispanic because it found a local optima for the cost function. I believe this is because of my 50/50 data split, although I may be wrong. If if is because of the split, how does this not happen in the real world? For example, if I had 10 million images, half of them white and the other half Hispanic, how does the network avoid falling prey to the same trap?
First of all your dataset is very small to train a deep learning model you would need at least 10k images to train the model and produce good results. Secondly, if you still think that your cost function gets stuck in local minima then you might consider using Nesterov momentum.