Unable to overfit using MLP I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples). 

With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.

Now moving on to MLP results,

Exp 1:


Network Architecture: [67 40 5]
Optimizer: Adam
Learning Rate: exponential decay of base learning rate
Validation Accuracy: ~45%
Observation: Both training accuracy and validation accuracy stops improving.


Exp 2:
Repeated Exp 1 with batchnorm layer


Validation Accuracy: ~50%
Observation: Got 5% increase in accuracy.


Exp 3:

To overfit, increased the depth of MLP. A deeper version of Exp 1 network


Network Architecture: [67 40 40 40 40 40 40 5]
Optimizer: Adam
Learning Rate: exponential decay of base learning rate
Validation Accuracy: ~55%


Thoughts on what might be happening?
I guess you are using linear activation functions,  maybe you are not initializing your weights, or you are regularizing your model enough.  

Initialize weights with glorot, insert dropout layers in between, use Relu as your activation function, stop the training process based on Early Stopping. and just experiment with one hidden layer.

side note: if you use Adam, don't mess with the learning rate.
in SGD optimizer you could use decay because there is a single learning rate for all weight updates and the learning rate does not change during training.
In Adam, the learning rate is maintained for each network weight (parameter) and separately adopted as learning unfolds.