illip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen.
In a nutshell, LoRA allows adapting pretrained models by adding pairs of rank-decomposition weight matrices (called update matrices)
to existing weights and only training those newly added weights. This has a couple of advantages:
Previous pretrained weights are kept frozen so that the model is not so prone to catastrophic forgetting. 
Rank-decomposition matrices have