Debugging Neural Networks I've built an artificial neural network in python using the scipy.optimize.minimize (Conjugate gradient) optimization function.

I've implemented gradient checking, double checked everything etc and I'm pretty certain it's working correctly.

I've run it a few times and it reaches 'Optimization terminated successfully' however when I increase the number of hidden layers, the cost of the hypothesis increases (everything else is kept the same) after it has successfully terminated.

Intuitively it feels as if the cost should decrease when the number of hidden layers is increased, as it is able to generate a more complex hypothesis which can fit the data better, however this appears not to be the case.

I'd be interested to understand what's going on here, or if I've implemented neural net incorrectly?
There are so many ways to go wrong with a neural net that it's going to be difficult to debug. Also, to address your intuition, each additional hidden layer makes learning much harder. With that said, here are some possibilities:


You have added weight decay. Adding more layers adds more weights which increases your regularization cost.
The problem is simple enough that a model with a single hidden layer is sufficient. Adding more hidden layers makes it harder for the network to learn (harder optimization problem).
The optimization method is not doing a great job (I prefer climin to scipy.optimize).
You are using the sigmoid/tanh activation function. The sigmoid function causes the vanishing gradient problem which makes learning hard with more layers. Try using the ReLu function.


Training neural nets takes a lot of practice, luck, and patience. Good luck.