How to set the number of neurons and layers in neural networks
The consideration of the number of neurons for each layer and number of layers in fully connected networks depends on the feature space of the problem. For illustrating what happens in the two dimensional cases in order to depict, I use 2-d space. I have used images from the works of a scientist. For understanding other nets like CNN I recommend you taking a look at here.

Suppose you have just a single neuron, in this case after learning the parameters of the network you will have a linear decision boundary which can separate the space to two individual classes. 





Suppose that you are asked to separate the following data. You will need d1 which specifies the upper decision boundary and somehow is doing AND operation to determine whether the input data is on the left side of it or on the right side. Line d2 is doing another AND operation which investigates whether the input data is upper than d2 or not. In this case d1 is trying to understand whether the input is on the left side of line to classify the input as circle, also d2 is trying to figure out whether the input is on the right side of the line to classify the input as circle. Now we need another AND operation to wrap up the results of the two lines which are constructed after training their parameters. If the input is on the left side of d1 and on the right side of d2, it should be classified as circle.



Now suppose that you have the following problem and you are asked to separate the classes. In this case the justification is exactly like the above's.



For the following data:



the decision boundary is not convex and is more complex than the previous boundaries. First you have to have a sub-net which finds the inner circles. Then you have to have another sub-net which finds the inner rectangular decision boundary which decides the inputs which are inside of the rectangle are not circle and if they are outside, they are circle. After these, you have to wrap up the results and say if the input data is inside the bigger rectangle and outside of the inner rectangle, it should be classified as circle. You need another AND operation for this purpose. The network would be like this:





Suppose that you are asked to find the following circled decision boundary.



In this case your network would be like the following network which was referred to but with much more neurons in the first hidden layer.