is done using the torch==1.13.1 version, while the accelerated transformers optimizations are tested using nightly versions of PyTorch 2.0. The tables below summarize the results we got.
The Speed over xformers columns denote the speed-up gained over xFormers using the torch.compile+torch.nn.functional.scaled_dot_product_attention.

FP16 benchmark
	
The table below shows the benchmark results for inference using fp16. As we can see, torch.nn.fu