Why does an LSTM cycle on initialisation?
This doesn't necessarily answer the question, but it does give some possible solutions to mitigate the problem. 

Apparently, the emphasised looping behaviour above is a result of improper initialisation, in which I was initialising with only positive and 0 weights. The looping behavior is largely diminished by proper initialisation, however it is not totally removed. See below for examples:

djbgnuywjkfrrkrkrpueeeeeffrrdv
kmkkkkkkkkkkkrkkkkkkkkkkkkkkkk
clfwtpjbsqeeeeeeesssjjeeeeeeee
ldwgmbvcmmbvcmmmgmmmgmmmgmmmmg
ywzrfotntntnttntntunuwqzffotnt
lvvurrrrrrxafllvvurrrrrrxcfhla


(These are samples from 6 different uniquely initialised LSTM's on the exact same architecture).

I have also found that this common behaviour is typically combated by adding a certain degree of randomness in feeding in the LSTM's output. This is normally done by interpreting the output at time step $t$ of the LSTM as a probability distribution, used to pick the next value fed into the LSTM at $t+1$. This helps break non-confident looping behaviour which is where looping behaviour most commonly occurs through some quick experimentation, while still mostly retaining confident predictions.

I also tested this by randomising the input a little bit, using a probability distribution where 90% of the time the correct input is selected and 10% a random one is, and back-propagating as normal. This didn't seem to have much of an effect on the looping behaviour, although perhaps might work as a good form of regularisation. I am yet to test it.

You can also use the temperature method in LSTMs, which explained really well here: https://cs.stackexchange.com/q/79241/20691.