Can the attention mechanism improve the performance in the case of short sequences?
They shouldn't have any issues with short sequences, as short dependencies are easier to learn. The only difficult cases are long dependencies which is where most of the research is geared at. However, this is assuming that by &quot;short sequence&quot; you mean a sequence of text that is fully contained within itself, i.e. there is no cross-sequence dependencies.
For example, if you have a really long paragraph that doesn't fit in a transformer model, you would have to break that paragraph into many &quot;short sequences&quot;, but each of these sequence may have a dependency that depends on another sequence, i.e. cross-sequence dependencies. For these cross-sequence dependencies, any model with recurrence should do better than ones without (e.g. RNN, LSTM, Transformer-XL).
If each short sequence is self-contained, then all of the models should perform pretty well.