mers, but it’s mostly noticeable on the A100 GPU.
The time reported is in seconds.
GPU
Batch Size
Vanilla Attention
xFormers
PyTorch2.0 SDPA
SDPA + torch.compile
Speed over xformers (%)
A100
10
12.02
8.7
8.79
7.89
9.31
A100
16
18.95
13.57
13.67
12.25
9.73
A100
32 (1)
OOM
26.56
26.68
24.08
9.34
A100
64

52.51
53.03
47.81
8.95







A10
4
13.94
9.81
10.01
9.35
4.69
A10
8
27.09
19
19.53
18.33
3.53
A10
10
33.69
23.53
24.19
22.52
4.29
A10
16
OOM
37