 huge speed ups and gains in GPU memory usage. The most recent being Flash Attention from @tridao: code, paper.
Here are the speedups we obtain on a few Nvidia GPUs when running the inference at 512x512 with a batch size of 1 (one prompt):
GPU
Base Attention FP16
Memory Efficient Attention FP16
NVIDIA Tesla T4
3.5it/s
5.5it/s
NVIDIA 3060 RTX
4.6it/s
7.8it/s
NVIDIA A10G
8.88it/s
15.6it/s
NVIDIA RTX A6000
11.7it/s
21.09it/s
NVIDIA TITAN RTX
12.51