What is the big fuzz about SHA-RNN versus Transformers?
Transformers can be trained with parallelization. So you might have 100 cores training the transformer for an hour (100 total hours of CPU time), whereas the LSTM has one core training for ten hours (10 total hours of CPU time). Even though the LSTM takes longer, it is less expensive to train.