ct_attention.

FP16 benchmark
	
The table below shows the benchmark results for inference using fp16. As we can see, torch.nn.functional.scaled_dot_product_attention is as fast as xFormers (sometimes slightly faster/slower) on all the GPUs we tested.
And using torch.compile gives further speed-up of up of 10% over xFormers, but it’s mostly noticeable on the A100 GPU.
The time reported is in seconds.
GPU
Batch Size
Vanilla Attention
xFormers
PyT