s (sometimes slightly faster/slower) on all the GPUs we tested.
And using torch.compile gives further speed-up of up of 10% over xFormers, but it’s mostly noticeable on the A100 GPU.
The time reported is in seconds.
GPU
Batch Size
Vanilla Attention
xFormers
PyTorch2.0 SDPA
SDPA + torch.compile
Speed over xformers (%)
A100
1
2.69
2.7
1.98
2.47
8.52
A100
2
3.21
3.04
2.38
2.78
8.55
A100
4
5.27
3.91
3.89
3.53
9.72
A100
8
9.74
7.03
7.04
6.62
5.83
A1