How to correctly classify voice from other sound events?
It is very possible that you don't have enough data to train your classifier.  Although several thousand seems like a lot, it means that you only have a fraction of the actual thing you are trying to identify.  It may be helpful to expand your training data by slightly tweaking your samples to improve the robustness of your machine learning algorithm.  You could take the existing human voice samples you have and try slowing each one down slightly, increasing the speed slightly, introducing pink/white noise, etc...  This technique is commonly used with images in CNNs (by introducing small rotations and stretching different ways) to increase the size of the training set when training image classifiers.  

Also, don't forget to reduce your learning rate as learning proceeds.  With too large of a step size your gradient decent could actually overshoot the loss function minima and result in poor results or saturate learning prematurely.

Hope that is helpful.