Depth of a Neural network
I think the short mathematical intuition would be to show that Deeper offers more flexibility.

Imagine we want to fit an impossibly complicated function, like this:



... but in an n-dimensional space - we obviously cannot visualise such a function. But we can agree is is complicated

Basic math analogy

The goal of the neural network would be to map the raw input data (e.g. images to a convolutional network) to some output, by approximating the complicated function.

So if we have some input, and apply a non-linear function $f$ to it, we transform it into something else:

$$output_1 = f(input)$$

Perhaps that gave us a curvy function, but it doesn't get close to matching the complicated function, so we endow the model with another chance, by applying a second non-linear function:

$$output_2 = g(output_1)$$

Giving a second function is almost like offering another degree of freedom (or flexibility) to the model.

We continue like this until the chain of non-linear functions is able to map out the output space sufficiently well. Perhaps we end up with this:

$$output_6 = k(j(i(h(g(f(input)))))$$

In this framework, imagine every single non-linear function as one of the layers in a deep network. The deeper the network gets, the more functions we are applying and the more we mould and transform the input to something else; perhaps in different ranges, by different magnitudes and so on.

This should (in a very hand-wavey manner) convince you that more functions applied gives more possibilities in the final output space. So more layers gives us more power to express more and more complicated functions.



A practical note: the more layers we add, the more powerful the model and the larger the tendency to either:


be very difficult to train
eventual overfit the training data completely
the longer the model takes to train
the greater the level of regularisation that is likely required to obtain a reasonable validation metric on unseen data




image source