The table below shows the improvement in runtime between the original implementation with xFormers installed and our optimized version with PyTorch-integrated memory efficient attention (originally developed for and released in the xFormers library)  and PyTorch compilation. The compilation time is excluded.
Runtime improvement in % compared to original+xFormers
See the absolute runtime numbers in section “Benchmarking setup and results summary”



GPU

Batch size 1

Batch size 2

Batch size 4




P100 (no compilation)

-3.8
   
0.44
   
5.47
   


T4

2.12
   
10.51
   