del and they control to which extent the model is adapted toward new training images via a scale parameter.
Note that the usage of LoRA is not just limited to attention layers. In the original LoRA work, the authors found out that just amending
the attention layers of a language model is sufficient to obtain good downstream performance with great efficiency. This is why, it’s common
to just add the LoRA weights to the attention layers of a mode