Is the working of RNNs, LSTM and GRU sequential or parallel? You take any blog or any example and all they tell you about is the given picture below.

It has 4 different matrices and 3 of whose weights are shared. So, I'm wondering how is this achieved in practice?
Please correct me:
I think the first word &quot;hello&quot; goes in as a one-hot encoded form and changes the Hidden matrix. And then after it, &quot;world&quot; goes and gets multiplied and then changes the matrix again and so on. What people make it look like is that all of the words going are in Parallel. It can't be the case because the Hidden matrix is dependent on the previous word and without changing the matric, you can not pass the current word. Please correct if my idea is wrong but I think the execution is in sequential order.
Yes, you are correct and it was one of original motivations, which inspired the invention of the Attention mechanism in seq2seq problems https://arxiv.org/pdf/1706.03762.pdf.
There is a quote from this paper:

Recurrent models typically factor computation along the symbol
positions of the input and output sequences. Aligning the positions to
steps in computation time, they generate a sequence of hidden states $h_t$,
as a function of the previous hidden state $h_{t−1}$ and the input for
position $t$. This inherently sequential nature precludes parallelization
within training examples, which becomes critical at longer sequence
lengths, as memory constraints limit batching across examples.

On the other hand, Transformer architectures have a loot loom for parallelization, because they take the whole sequence at once, and multiple heads can be executed in parallel.