hts. This has a couple of advantages:
Previous pretrained weights are kept frozen so the model is not as prone to catastrophic forgetting.
Rank-decomposition matrices have significantly fewer parameters than the original model, which means that trained LoRA weights are easily portable.
LoRA matrices are generally added to the attention layers of the original model. 🧨 Diffusers provides the load_attn_procs() method to load the LoRA weights into 