Final layer of neural network responsible for overfitting I am using a multi-layer perceptron with 2 hidden layers to solve a binary classification task on a noisy timeseries dataset with a class imbalance of 80/20. I have 30 million rows and 500 features in the training set. The dataset is structured, ie, not images. My original features were highly right skewed; I do my best to transform these into nicer distributions by either taking logs or categorising some of them. I use an architecture of 512->128->128->1, with relu activations in every layer except the last. My loss function is sigmoid cross entropy.

The validation set contains 10 million rows. Initially the validation error goes down, but then starts to go up again after a couple of epochs. On analysing the gradients and weights of each layer, I see that the overfitting coincides with the weights on the final layer only getting larger and larger. The final layer seems to go into overdrive while the rest of the network seems to do very little learning.

I can solve the overfitting problem by using l2 regularisation, but this hurts the validation error. I have yet to find a beta regularisation parameter which doesn't hurt the best validation error I've seen. Dropout makes things even worse.

Granted, the classification problem is very difficult, with probably a very weak signal, but I find that gradient boosted trees are able to generalize much better than a simple, say, 64x64 multi-layer perceptron (the log loss on the training set is the same for both network and gradient boosted tree).

Are there any words of wisdom on how to make this network generalize better given that I've already tried:


dropout of varying degrees
l1/l2/group lasso regularization
adding noise to inputs
adding noise to gradients and weights
feature-engineering so as to remove/re-represent highly skewed features
batch normalization
using a lower learning rate on the final layer
simply using a smaller network (this is the best solution I've found)


to some or all layers. All methods hurt the validation error so much that the performance is nowhere near how well the tree model does. I would have given up by now were it not for the fact that the tree model is able to do so much better out of sample, but the training log loss for both is the same.
Are you balancing your training dataset 50/50? It seems to me that you are making a lot of changes to account for 80/20 balance when it sounds like you have a deep enough dataset to balance your training to be 50/50 and still have a reasonable size for your training set. How about keeping it to a smaller network - as you and some notes above suggest - and with a 50/50 set on a smaller training sample?