t_product_attention is faster than xFormers on all the GPUs we tested.
Using torch.compile in addition to the accelerated transformers implementation can yield up to 19% performance improvement over xFormers in Ampere and Ada cards, and up to 20% (Ampere) or 28% (Ada) over vanilla attention.
GPU
Batch Size
Vanilla Attention
xFormers
PyTorch2.0 SDPA
SDPA + torch.compile
Speed over xformers (%)
Speed over vanilla (%)
A100
4
16.56
12.42
12.2
11.84