LoRA Dropout as a Sparsity Regularizer for Overfitting Reduction
Large Language Model;LoRA;Dropout
Yes
No ethics review needed.
1. Can the authors provide more clarity on the assumptions made for the theoretical analysis, specifically the \eta-Lipschitz continuity and the positive semi-definiteness of the Hessian? Are there common NLP models or settings where these conditions do not hold?
2. Could the authors provide more detailed quantitative data on the computational overhead introduced by sampling multiple dropout instances during training and testing? How significant is the increase in training and inference time compared to the original LoRA or other baselines?
3. Can the author provide more experiments on other variants of Lora to demonstrate the effectiveness of the proposed Loradropout framework? E.g., LoRA+, VeRA, LoRA- fa.
4. What is the difference between the author's Lora dropout method and other existing dropout methods, such as DropKey, DropAttention, and HiddenCut.
1. The paper is well-executed, with thorough theoretical analysis supporting the proposed method.
2. The paper is generally clear and well-structured, making it accessible to readers familiar with PEFT and regularization techniques.
The paper introduces LoRA Dropout, a refined dropout strategy applied to the rows and columns of LoRA's low-rank matrices, which enhances sparsity patterns without significant computational overhead. Additionally, the authors propose a test-time ensemble strategy that aggregates outputs from different dropout instances, further improving generalization. Experiments on NLP tasks like GLUE and SQuAD benchmarks demonstrate that LoRA Dropout significantly narrows the training-test loss gap and enhances model performance and generalization.
1. Theoretical Assumptions and Generalizability
The theoretical framework relies on specific assumptions such as η-Lipschitz continuity and positive semi-definiteness of the Hessian matrix. These conditions may not universally apply to all language models or downstream tasks, making the theoretical guarantees less generalizable
2. Computational Efficiency and Practicality
The paper acknowledges that sampling multiple dropout instances during training and testing increases computational costs but lacks quantitative evidence on the extent of this overhead. Without this data, it is difficult for practitioners to evaluate the feasibility of adopting LoRA Dropout in real-world applications.
3. Experimental diversity
Although relevant experiments have been conducted in Lora and AdaLora, their effectiveness in other Lora variants has not been demonstrated.
Yes
No ethics review needed.
- Can the authors specify how significantly can "closeness" affect the stability constant in Proposition 2.2? Is closeness small for LoRA Dropout?
- How large is $\eta$? Is this small for your pretrained models?
**Originality:**
This paper proposes the use of dropout for the rows or columns of the tunable low-rank matrices in LoRA, which is practical and overcomes the memory limitation of some prior dropout methods. Furthermore, an ensemble method is proposed for the inference phase to further boost quality of PEFT.

**Quality:** 
The experimental results suggest that the proposed method can work well.

**Clarity:**
The writing is quite easy to follow.

**Significance:**
The proposed method seems to work well, which potentially is significant to many practical applications.
This paper investigates effectiveness of dropout for parameter-efficient fine-tuning (PEFT) methods, specifically LoRA. This topic has been investigated before. The authors first present an analysis about the effect of dropout (with Bernoulli noises) to the loss and generalization error of the learning algorithm. The authors show that a suitable use of dropout rate can reduce generalization error, which is similar with the existing understanding about dropout in ML. The authors then propose a new way (called LoRA dropout) to enclose dropout in LoRA (and LoRA-based variants). Different with prior methods, LoRA dropout applies dropout for the rows or columns of the tunable low-rank matrices. Furthermore, the authors propose an ensemble method for the inference phase, borrowing the idea of MC dropout. Such an ensemble can boost performance of LoRA dropout. Finally, the authors did an extensive experiment to evaluate performance of their method, in comparison with related baselines. The empirical results suggest that their new method can often perform better than the baselines on different benchmarks and tasks. This is really encouraging.
Despite some encouraging empirical results, many concerns arise from the current presentation. Those concerns are mostly for their theoretical analysis and writing. Each will be discussed below.

**For the theory:**

- The authors use algorithmic stability to analyze generalization error of dropout training. They analyze a regularized loss, which uses Bernoulli dropout in the regularization term. I am not sure that LoRA dropout training can be easily formulated as minimizing this simple regularized loss. Hence, some of their theoretical understandings may not apply to LoRA dropout.

- The main theoretical understandings are mostly originated from Proposition 2.2 which analyzes point-wise hypothesis stablity of a learning algorithm. However, there are many concerns about correctness and meaning of this proposition: 

    - Closeness between $\mathbf{\theta}_L (\mathbf{S}^i)$ and $\mathbf{\theta}_L (\mathbf{S})$ is *not defined* clearly. Such quantity is very important to derive their result, and should play a significant part in the stability constant. The authors ignored this part completely in their proof, causing a big question about the goodness/meaning of their result.
    - The assumption about $\eta$-Lipschitzness of the loss is very important for their analysis and result. Such Lipschitzness seems to be w.r.t the model parameters. However, the authors did not clearly specify it. More importantly, the magnitude of $\eta$ should affect significantly to the bound on generalization error in Theorem 2.4. Nonetheless, the authors did not discuss how large $\eta$ is in practice. For big models, e.g., Llama2, $\eta$ might be very large. If so, their error bound in Theorem 2.4 will be vacuous or meaningless.
    - The assumptions about $\eta$-Lipschitzness and closeness should be translated to the training algorithm. Those assumptions are really strong, and may not fit well with practice. 
    
- Proposition 3.1 claims that their ensemble classifier should have smaller error than the average error of the whole classifier family. However, the statement and the proof of this proposition have a big mismatch. Their proof in Appendix A.2 assumes that the loss is convex w.r.t. the model parameters. Such an assumption is entirely different from that in Proposition 3.1. More importantly, such an assumption is **unrealistic** for deep neural networks, since it is well-known that the training loss for DNNs is often nonconvex. Therefore, their interpretation from this proposition seems not to be well-supported.

**For the writing:**

- Some notations are used without definition, e.g., $\mathbf{\theta}_L$
- Some important concepts or assumptions are not defined explicitly, e.g. "closeness" and $\eta$-Lipschitzness. 
- Some mistakes appear in Eq. (1) and (3). The authors misused the function and problem definitions.
- The loss function in Lemma 2.3 seems wrong.
- The empirical and expected losses in Lemma 2.3 and Theorem 2.4 seem wrong. Should they represent the loss for a learning algorithm?
Yes
No ethics review needed.
* Can the statement of theorem 2.4 depend on the presence of dropout (preferably in the form used in the experiments) and show that dropout contributes to reducing the generalization gap?

* What is the practical cost of the test time ensembling (in the experimental results)?

* In figure 4, what is the rank used for LORA-dropout?  This is important to grasp the meaning of the dropout rate.
* The idea is simple and makes a lot of sense.

* The empirical evaluation is comprehensive and convincing. In particular, I like figure 5 because it shows that the authors have identified a key question and are bringing a satisfactory answer.
The paper proposes a dropout method for LORA fine-tuning which amounts to using dropout on the implicit hidden layer that sits in the middle of the low rank adapters. This is supported by questionable theoretical arguments and solid empirical evaluation.
The theoretical part (section 2) is weak and confuses the message of the paper.  

* First, equation (1) is nonsensical. This constraint trivially implies that \delta\theta=0.  This can be fixed by using an inequality constraint instead (i.e. the constraint is less than an arbitrary constant C whose choice is related to the choice of lambda in the dual formulation). 

* Second, theorem 2.4 does not rely on the presence of dropout, and therefore does not say much about dropout at all. The actual argument (lines 189 to 193) could have been made without this theorem. Therefore the theoretical development of section 2 seems irrelevant to the question at hand and distracts rather than inform the reader.  

* Finally the dropout scheme discussed in section 2 (equation 3) is different from the scheme proposed in section 3.2 and used in the experiments (as explained by the authors in section 3.2). This further weakens the theory of section 2.  

The paper would be as readable and informative if section 2 were replaced by developing the argument of lines 189 to 193 in the context of the actual dropout algorithm (the one of section 3.2). This would make the paper lighter but better overall.  Alternatively it may be possible to make theorem 2.4 dependent on the presence of dropout, preferably in the form used in the experiments. That would save the theoretical section.
Yes
No ethics review needed.
* Why did you decide to only DeBERTaV3 and LLaMA2-7B (for just one task) to validate your claims?
* Why did you conduct the ablation analysis only on MRPC and CoLA?
* How would the runtime look like for larger, state-of-the-art language models? Do you expect a similar increase in runtime and is this feasible in practive?
* The idea is empirically and theoretically well motivated.
* The improvements of DeBERTaV3 and LLaMA2-7B on GLUE, SQuAD, MMLU, and Vicuna-Eval benchmarks are statistically significant.
* The presentation is clear and easy to follow.
The paper builds upon Low-Rank Adaption (LoRa) by introducing a dropout-based sparsity regulator to mitigate overfitting. The authors provide empirical evidence for signs of overfitting with LoRa and derive a generalisation error bound under sparsity regulation Their method drops rows and columns from both tuneable low-rank parameter matrices and maintains efficiency with an ensemble of losses with different dropout instances. Further, model inference is improved by an ensemble strategy compressing the error bound. Finally, all claims are empirically evaluated in Natural Language Understanding, Question Answering, and Instruction Tuning.
* The evaluation is focused on smaller and nowadays partially outdated models. The paper would benefir from more recent models.
* The ablation is missing test-statistics and standard deviations and is conducted on different, smaller datasets.
* The runtime increases drastically (For example, 18min -> 60min on CoLa).
* It is not clear, how model size further impacts runtime and efficiency as more parameters are involved. As inference time is very crucial it would be beneficial to explore this in more detail and give reccomendations.
Yes
No ethics review needed.
1. Line 48-49, “these models typically tend to maintain a relatively high rank to ensure sufficient expressive power”. Indeed, as seen in many papers, with rank of 2,  LoRA series usually can achieve comparable performance to fully fine-tuning. Even fine-tuning LLAMA-7B, some methods can achieve comparable performance with rank=4. 

2. What is the rank setting in NLU tasks? Additionally, how should the LoRA Dropout rate be configured? The original LoRA already includes a dropout rate—does it also play a role during training in this context?

3. Line 228, “which is equivalent to masking random columns..” LoRA Dropout indeed masks random columns and rows of Delta W. Why does masking rows improve performance? What would the outcome be if only rows were masked? 

4. Line 240-243, “Additionally, performing dropout…sparsity in our framework.” I have a question: while I acknowledge that sparsity is important for fine-tuning, why is it necessary to increase the sparsity of delta W when designing LoRA Dropout? After all, more sparsity does not always equate to better performance.
1. I endorse the decision to exclude dropout from the rank dimension in the design of the method.
2. LoRA Dropout can help existing methods like LoRA achieve better performance.
This paper proposes LoRA Dropout, a novel sparsity regularization technique that applies dropout to the low-rank matrices in LoRA-based fine-tuning methods to mitigate overfitting, offering both theoretical analysis and practical improvements across various NLP tasks.
1. Line 49-50, the sentence is written in bold. Does it relate to the motivation of this paper and LoRA Dropout? If so, what is the relationship between "high rank overfitting" and LoRA Dropout? Can LoRA Dropout alleviate the overfitting issues caused by a high-rank LoRA model?

2. Following up on the first question, I do not believe that AdaLoRA addresses the high-rank overfitting problem as stated. Instead, AdaLoRA aims to allocate different ranks to weights based on their importance, which is unrelated to the issue of high-rank overfitting. If the authors claim that AdaLoRA can resolve this issue, further explanation is required.

3. The motivation of this paper is unclear, and the introduction lacks logical coherence. In essence, the introduction's logic could be simplified to: high-rank causes overfitting, AdaLoRA mitigates this issue but is insufficient, and Dropout can address it, but without solid theoretical support. It appears that the fundamental problem this paper seeks to address is how to theoretically prove that Dropout can solve the high-rank overfitting issue. However, this is not the case. Could the authors clarify the true motivation behind this work?

4. I do not consider this manuscript to offer a theoretical analysis of the LoRA optimization process with dropout. First, the whole section 2 is quite similar to that in [1], the two main conclusions (Eqs. 5 and 7) differ from the formula 6 and 7 in [1] merely by a change in variables. The proofs in this paper is also heavily similar to those in [1]. Second, Eq. 1 in this paper can be transformed into Eq 1 in [1]. Therefore, they are essentially the same problem. Third, the conclusions drawn in Section 2 (lines 186-193) are, in fact, consistent with the analysis of the effects of sparsity in [1]. Dropout can be viewed as a specific form of sparsity, and therefore, the relationships proven in this section are already encompassed by the analysis in [1]. In conclusion, due to the similarity in proofs and the correlation between dropout and sparsity, I do not find Section 2 to be meaningful. The authors could have simply referenced the conclusions from [1] instead of reproducing the proofs from [1] and presenting them as their own contribution.

5.  While the method itself is not complex, it demands almost double the training time (Table C.2).

[1] On the effectiveness of parameter-efficient fine-tuning, AAAI 2023.

typos:
1. Line 13. faces
2. Line 212. bold delta W. Besides, the use of the symbol “delta W” in this paper is inconsistent.
3. Line 526. introduce
4. Table D.1 caption. dataset.
5. Table C.2 duplicated “on”
6. Table1 and 2 caption. Dropout
7. Line 436. between
8. Line 173, 909. Respectively

The writing of this paper gives the impression that it was completed rather hurriedly.
@inproceedings{
anonymous2024lora,
title={Lo{RA} Dropout as a Sparsity Regularizer for Overfitting Reduction},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=c4498OydLP},
note={under review}
}
Parameter-efficient fine-tuning methods, represented by LoRA, play an essential role in adapting large-scale pre-trained models to downstream tasks. 
However, fine-tuning LoRA-series models also face the risk of overfitting on small training datasets, and there's still a lack of theoretical guidance and practical mechanisms to control overfitting on LoRA-based PEFT methods. This paper introduces a novel dropout-based sparsity regularizer for LoRA, dubbed LoRA Dropout, which mitigates overfitting by applying refined dropout to LoRA's low-rank matrices.
We establish a theoretical framework that models dropout in LoRA as a sparse fine-tuning process and derive a generalization error bound under this sparsity regularization.
Theoretical results show that appropriate sparsity can tighten the gap between empirical and generalization risks and thereby control overfitting. We further enhance the sparsity patterns in conventional dropout methods and propose an innovative LoRA Dropout method for more precise sparsity regularization to achieve better overfitting reduction. 
Furthermore, we introduce a test-time ensemble strategy and provide theoretical evidence demonstrating that the ensemble method can further compress the error bound and lead to better performance. 
Extensive experiments on various NLP tasks validate the effectiveness of our LoRA Dropout framework in improving the model's performance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Model LoRA Dropout
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c943288f424a90f586920d61178342900b9253b4.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/52f03e9c48c3d79bf374e491887f48fc7780e256.zip
LoRA Dropout as a Sparsity Regularizer for Overfitting Reduction
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission