Neural Network Shape / Structure I am attempting to learn neural networks using the Keras libary on the MNIST hand written digit dataset, using dense layers only. I am trying to figure out what the best shape for the network should be but I can't seem to find any literature or discussion on the subject.

By shape, I mean should the hidden layers have more nodes as I go deeper, less nodes, or the same? I've taken to calling them square, pyramid, and reverse-pyramid because I couldn't find a better name. See the end of this post for clarification on shape

I've tried all 3 and the reverse pyramid is giving me the best at 80% accuracy but I can't imagine why. I feel like I would just be losing data and introducing noise by expanding the number of neurons as I go deeper.

Is there any literature or discussions about this subject? Perhaps, I'm not using the right keywords to look, I just don't know the correct terminology. I've tried neural network shape, structure, hidden layer layout, etc. with very little luck.

Thanks in advance.

Square - Hidden layers stay same size until the end



Pyramid - less neurons at each additonial hidden layer (pretend they are all hidden layers)



Reverse Pyramid - more neurons at each additiona hidden layer
Are you using the Training set for testing ? What is your train test split ?

With Neural Nets, it's always the pyramid shape that works. This is because as you go deeper, data about the input is condensed and we get a proper representation of the input for outputting the required class or regression value.

The only reason reverse pyramid nets will work well is overfitting, which accommodates noise and does not generalize well. 

http://www.deeplearningbook.org/contents/mlp.html

Try the Bible of Deep Learning for more details.