unctional.scaled_dot_product_attention and torch.compile+torch.nn.functional.scaled_dot_product_attention.
For the benchmark we used the stable-diffusion-v1-4 model with 50 steps. The xFormers benchmark is done using the torch==1.13.1 version, while the accelerated transformers optimizations are tested using nightly versions of PyTorch 2.0. The tables below summarize the results we got.
Please refer to our featured blog post in the PyTorch site