Unable to overfit using MLP I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples). 

With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.

Now moving on to MLP results,

Exp 1:


Network Architecture: [67 40 5]
Optimizer: Adam
Learning Rate: exponential decay of base learning rate
Validation Accuracy: ~45%
Observation: Both training accuracy and validation accuracy stops improving.


Exp 2:
Repeated Exp 1 with batchnorm layer


Validation Accuracy: ~50%
Observation: Got 5% increase in accuracy.


Exp 3:

To overfit, increased the depth of MLP. A deeper version of Exp 1 network


Network Architecture: [67 40 40 40 40 40 40 5]
Optimizer: Adam
Learning Rate: exponential decay of base learning rate
Validation Accuracy: ~55%


Thoughts on what might be happening?
Some things that could be happening: 


You do not have enough parameters in your hidden Layer to learn. Try something like [67, 512, 5] or expand deeper with something like [67, 1024, 256, 5]. The idea is that your hidden layer may be too small to capture interactions between the attributes given the amount of training data you have.


A rule of thumb that I have seen is:

Nh=Ns/(α∗(Ni+No))

Nh = number of hidden neurons.
Ni = number of input neurons.
No = number of output neurons.
Ns = number of samples in training data set.
α = an arbitrary scaling factor usually 2-10.


Which in your case gets you to like 5000 neurons, but since you are oversampling your classes, I would also suggest under-sampling instead, and that gets you maybe to around 1024, which is fairly standard. The [67, 512, 5] configuration is pure gut reaction. 


Learning rate is too high. Try starting with lower LR (like 10e-5). Sometimes this works with Adam.
Weight Initialization : double check you are using Xavier.