Processing the entire input sequence at once allows transformers to capture dependencies across large contexts, but it comes at a cost. The attention mechanism's computational requirements grow in proportion to the square of the input sequence length. This is a disadvantage when compared to RNNs, which have a fixed computation per input token. There is also some evidence that transformers are more prone to overfitting when trained with lower-quality data.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence