How to correctly classify voice from other sound events? I have a huge labeled dataset of several thousand sound events, including human voice, dish washing, things falling to the ground, among others. 

I need to report when a human voice event takes place. Notice that it's not a simple voice activity detection (VAD), since there are other sound events competing with voice (in VAD, they normally only concern about well behaved noise).

My approach so far has been to train a binary SVM classifier (voice and non-voice classes) with MFCC features. Even after parameter optimization and tinkering about with different number of MFCC coefficients, the performance is awful for such a simple task...

Is there any heuristic or any thing that may help distinguish voice from non-voice events that I'm missing out on?

(This related question is similar, but I don't need to "eliminate" other sounds completely, and I'm looking, first, for possible heuristics to improve the classification. This article would be my last resort.)
It is very possible that you don't have enough data to train your classifier.  Although several thousand seems like a lot, it means that you only have a fraction of the actual thing you are trying to identify.  It may be helpful to expand your training data by slightly tweaking your samples to improve the robustness of your machine learning algorithm.  You could take the existing human voice samples you have and try slowing each one down slightly, increasing the speed slightly, introducing pink/white noise, etc...  This technique is commonly used with images in CNNs (by introducing small rotations and stretching different ways) to increase the size of the training set when training image classifiers.  

Also, don't forget to reduce your learning rate as learning proceeds.  With too large of a step size your gradient decent could actually overshoot the loss function minima and result in poor results or saturate learning prematurely.

Hope that is helpful.