What are the hidden states in the Transformer-XL? Also, how does the recurrence wiring look like?
By hidden states, they mean outputs of the layers, i.e., what you get after the feed-forward sub-layer. For Transformer-XL, it is important that these are also what you use as an input to the self-attention. Therefore, at inference time, if you want to compute the states recursively by segments (presumably because you cannot fit the entire input int he memory), this is the only thing you need to remember from the previous steps to continue the computation.
There is no encoder, you can imagine Transformer-XL as a decoder-only model. Transfering the states just means remembering them, so you can do the self-attention over them, but you can no longer back-propagate through them because you only remember the values and the entire graph telling how you got them.
The dots in the scheme correspond to the hidden states: one state per input subword and per layer. The lines between them are the self-attention links.