How deep should my neural network be?
Answer


  Are there any heuristics for deciding on where to start with the number of layers for a neural network? 


No there isn't one in general. The architecture of the network (including depth) is considered to the new "feature engineering". It's more of an art than science at the moment. Anything above 3 layers worth considering. 


  I want to add a bunch of polynomial features 


You don't need to do this kind of feature engineering like you would in traditional machine learning. If you have a network that is sufficiently wide and deep, your network should be able to "engineer" those features itself. Just think about how the first hidden layer is a linear combination of input features. And the second hidden layers is a linear combination of the first hidden layer. It should be obvious to see how a neural network is capable of structuring any polynomial features itself.

Suggestion

Try using the original 60 features directly and start with 3 layers. Given the size of the network, the training should take less than 10 minutes on a GPU-enabled machine. Compare your result with your benchmark model (you should always have a benchmark model). Play around with the hidden size and layers, rinse and repeat.