ined, but it doesn’t have to be with a few optimization tricks. One of the biggest (and easiest) optimizations you can enable is switching to memory-efficient attention. If you’re using PyTorch 2.0, scaled-dot product attention is automatically enabled and you don’t need to do anything else. For non-PyTorch 2.0 users, you can install and use xFormers’s implementation of memory-efficient attention. Both options reduce memory usage and accelerate