

Runtimes for batch size 4



Configuration

P100

T4

A10

V100

A100




Original without xFormers

117.9s (-20.0%)
   
112.4s (-81.8%)
   
47.2s (-101.7%)
   
35.8s (-71.9%)
   
22.8s (-78.9%)
   


Original with xFormers

98.3s (0.0%)
   
61.8s (0.0%)
   
23.4s (0.0%)
   
20.8s (0.0%)
   
12.7s (0.0%)
   


Optimized with vanilla math attention, no compilation

101.1s (-2.9%)
   
73.0s (-18.0%)
   
28.3s (-21.0%)
   
23.3s (-11.9%)
   