Is this a correct visual representation of a recurrent neural network (RNN)?
I will answer this question, leaving it open to challenge by anyone more knowledgeable. 

The equations to update each layer of an RNN are

$y_t = \sigma(W_x x_t + b_x )$

and

$h_t = \sigma(W_hx_t + b_h)$

Where $h_t$ is the hidden layer (in blue in the picture), and $y_t$ is the output layer (red in picture). This equations says that every single component in the hidden layer vector, i.e. every unit in the hidden layer, is a function of the linear combination of the $x_t$ vector, which is the first yellow row along the depth axis. In other words, all the first yellow input nodes on the bottom left in the picture. 

Thus, technically this picture is not correct, all the yellow nodes should have points to all the blue nodes. Also, by similar reasoning, all the blue nodes in the subsequent steps of the hidden layer should be connected to all the blue nodes of the previous layer.

Of course, that would make for a much uglier/harder picture to make, so I don't blame the authors, although this has given me a few hours of confused research, which I guess still meets their educational goal.