anilla attention.
GPU
Batch Size
Vanilla Attention
xFormers
PyTorch2.0 SDPA
SDPA + torch.compile
Speed over xformers (%)
Speed over vanilla (%)
A100
1
4.97
3.86
2.6
2.86
25.91
42.45
A100
2
9.03
6.76
4.41
4.21
37.72
53.38
A100
4
16.70
12.42
7.94
7.54
39.29
54.85
A100
10
OOM
29.93
18.70
18.46
38.32

A100
16

47.08
29.41
29.04
38.32

A100
32

92.89
57.55
56.67
38.99

A100
64

185.3
114.8
112.98
39.03









A10
1
10.59
8.81
7.51
7.35
16.57
30.59