Query on unstable loss curves for RNN
There can be some other factors that affect this, such as using simulated annealing (in a NN context) or other learning rate schedules. Are you using a specific LR schedule?

A schedule might be that the LR decreases by 50%, every time the validation loss of 5 epochs in a row does not decrease. This will help get closer and closer to a minimum of the loss. However, we know it is possible to get stuck within a local minimum, which may be far from optimal, so we can shake things up by increasing the LR once again, which will essentially throw the algorithm our of tyhe local minima and on its way to a new minima (at least that is the hope). This kind of schedule often produces loss curves like the ones you see.

Another alternative is simple that your batch sizes are quite small, and every once ina  while, you get a batch that consists of example which your model really struggles with, so the loss for that batch (and so the epoch) would spike in comparison to other epochs.

A final idea - thinking more about your data - if it is time-series e.g. of stock prices or weather - there could be a regime change/shift. Meaning that the underlying function or system suddenly switches to a new pattern. Something like this could throw your model off the scent for a while, and so produce bumps in the loss curve.



A small point on terminology: LSTM and GRU architectures are themselves RNNs. A recurrent network is one in which connections do not only move forward in a network, but can also go sidewards across a layer or indeed backwards. So it is a more general term, whereas LSTM/GRU layers are specific examples of RNNs.

If you can say a little more about your three model architectures, perhaps it might be clearer which names to use - and maybe even better understand these loss curves  :-)