How can I understand this statement about RNNs and hidden layers?
I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid.

Let's say we have an Elman recurrent neural network with one input neuron, one output neuron and one hidden layer with two neurons.

In total there are 10 connections. As the image shows, neuron A receives the combined previous output of both neuron A and B as input. The same goes for neuron B. 

This is not the case when we split the neurons up into multiple layers; the context neuron(s) are only used by neurons that are in the same layer. Let say we now use multiple hidden layers and keep the amount of neurons the same. In total there are 7 connections now (image below). That is 3 less than in the first example, which has only one hidden layer. So which connections do we miss? That is shown in the bottom image. (I had to paste these two images together in one image because my reputation only allows me to post 2 links)


Please note the cross; the connection between neuron A and B is not there in the first image, because it would be some kind of random recurrent connection.

The first and the last image are exactly the same. I think that if you compare the first and the last image that you agree that the statement is true.