Significance of depth of a deep neural network
Note: The following statements are about feed-forward neural networks. If you're interested in something else, please let me know.

There is a paper which proofs that one hidden layer is enough. So theoretically you could represent any network with many layers with a single hidden layer. I don't remember the title, though. Also I don't remember how it scaled (how many additional nodes you need per added layer).

In practice, it is a very different story anyway: 


You need to train the network in an acceptable wall-clock time,
able to fit stuff in memory
You have limitations on maximum time for one inference
You need to find the weights by a training algorithm


Anecdotally, I can say wider networks are usually have more problems with overfitting / memory limitation problems while deeper networks have more problems with maximum inference time.

There is a nice paper from Microsoft where they trained the first super deep network. Many papers followed to that one:


  He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.


One main claim was that deeper networks learn more complex features. I don't remember how much of a problem overfitting was. Another important insight is that skip connections are critical for really deep networks (> 20 learning layers).