How does transformer leverage GPU which trains faster than RNN?
The issue with Recurrent models is that they don't parallelization during training. 
Sequential models performs better with more memory but faces problem in learning long-term memory dependencies. 

On the other hand Transformers take into account of self attention which boosts the speed of how fast the model can translate from one sequence to another and establishes dependencies b/w input and output and focus on relevant parts of the input sequence, which in turn eliminates recurrence and convolution unlike RNNs where sequential computation inhibits parallelization.