47.7s (0.0%)
   
31.3s (0.0%)
   
12.5s (0.0%)
   
11.4s (0.0%)
   
8.0s (0.0%)
   



Optimized with vanilla math attention, no compilation

49.3s (-3.5%)
   
37.9s (-21.0%)
   
17.8s (-42.2%)
   
12.7s (-10.7%)
   
7.8s (1.8%)
   


Optimized with mem. efficient attention, no compilation

47.5s (0.4%)
   
31.2s (0.5%)
   
12.2s (2.6%)
   
11.5s (-0.7%)
   
7.0s (12.6%)
   


Optimized with mem. efficient attention and compilation

-
   
28.0s (10.5%)
   
11.4s (9.0%)
   
10.7s (6.4%)
   
6.4s (20.3%)
   


Runtimes for batch size 4