All trainings used a cross-entropy loss with a AdamW 71 optimizer with ß 1 = 0.9, ß 2 = 0.999, and ε = 1.10 -8 . The LoRA fine-tunings required additional parameters like rank of the matrices, scaling factor alpha, and dropout probability for the LoRA layers. A fixed rank of 8 is chosen considering previous findings that higher ranks result in minor performance improvements and could result in increased memory requirements . The alpha is kept to be the same value as rank as suggested in the original LoRA paper . An initial set of experiments were performed to fix some hyper-parameters, selecting a batch size of 32 and cosine-with-restarts learning rate scheduler. A hyper-parameter