Is the working of RNNs, LSTM and GRU sequential or parallel?
Yes, you are correct and it was one of original motivations, which inspired the invention of the Attention mechanism in seq2seq problems https://arxiv.org/pdf/1706.03762.pdf.
There is a quote from this paper:

Recurrent models typically factor computation along the symbol
positions of the input and output sequences. Aligning the positions to
steps in computation time, they generate a sequence of hidden states $h_t$,
as a function of the previous hidden state $h_{t−1}$ and the input for
position $t$. This inherently sequential nature precludes parallelization
within training examples, which becomes critical at longer sequence
lengths, as memory constraints limit batching across examples.

On the other hand, Transformer architectures have a loot loom for parallelization, because they take the whole sequence at once, and multiple heads can be executed in parallel.