Difference between Jordan, Elman and normal RNN
The architectures in your second and third pictures are identical because the link $I$ from the hidden units to the context units in the Elman's network is not trainable. It is always equal to 1.
In his paper [1], Elman first provides the Jordan's architecture, then introduces his own network but does not discuss why his one is better.
Anyway, this is what I think about why it may be preferred.
The neurons in the hidden layer learn features or internal representations of the inputs. Backward connections allow hidden units to learn features which encode sequential properties of the inputs.
In a deep network, the neurons in the layer that is further from the input learn more abstract features than the neurons in the layer closer to the input. Feeding the output from a layer back to the same layer allows learning sequential properties at the same level of abstraction. Feeding the output from a layer to the input of a different layer mixes the levels of abstractions.
In the Jordan's network, the output of the whole network directly influences the hidden layer's input, so the learned representation is split between the hidden layer and the output layer. Elman's network allows more separation of features by levels of abstraction.
Elman spends most of his paper trying to understand the features his neurons have learned even for those tasks where his network does not perform well. I am not completely sure but I guess that backward connections between different layers would make such analysis more difficult.

Elman, J.L. (1990), Finding Structure in Time. Cognitive Science, 14: 179-211. doi:10.1207/s15516709cog1402_1