DPD-LoRA: Dynamic Prompt-Driven Low-Rank Adaptation for Improved Generalization
Vision-Language Models;PEFT;Prompt Learning
We thank the reviewer for many insightful comments. We answer the questions in what follows. Please let us know if further clarification is needed.

**Q:This can seem a bit cluttered and redundant; a more concise summary and consolidation of related content would be beneficial.**

Thank you for your insightful suggestion. Reviewer pw2m mentioned this as well, and we will revise the paper to focus on the main points. We want to emphasize that our main contributions are twofold: first, we are the first to prove that prompt learning can additionally provide task-specific guidance to LoRA (even in plain LoRA settings, as shown in Table 5 in the appendix); second, the proposed gating mechanism strengthens their connection.

**Q:Sec 3.2; the explanation seems to treat prompts and LoRA separately, although the appendix provides a detailed explanation of their combined effect. As this is a crucial part of the paper, more clarity and detail in the main body of the text would be necessary.**

They are actually not separate. Our intention in Section 3.2 was to demonstrate the combined effect of prompts and LoRA within the Transformer architecture. Specifically, we introduce learnable prompts in both the textual and visual branches, which are then incorporated into the input sequences. These prompted inputs interact with the MHA mechanism as shown in Equation (2). Then we adapt the standard LoRA formulation to this prompted framework in Equation (3), which is directly influenced by the prompted inputs $X$. This equation illustrates that the output h is a result of both the original weights and the prompt-guided LoRA adjustment, highlighting their interconnected roles. However, we acknowledge that we might merge more details from the appendix. Thank you for pointing this out.


**Q:The title of Section 3.3; The order of introduction and the content should correspond to the title.**

Thank you for bringing this to our attention; we will align the order of introduction and section contents.

**Q:Analysis of the synergistic effects between LoRA and prompt learning**

We include this part in our Appendix Section D, and the interpretation based on mathematical derivation is also included in Section D5.

**Q:Different weights allocation methods are used: α and 1-α for prompt tokens, and β and γ for LoRA layers.**

We thank you for pointing out this issue that many reviewers care about. We acknowledge it causes confusion that our 
 $\alpha,\beta,\gamma$  are the same and have the same values (0.1 for $(l-1)$ and 0.9 for $l$, as you can tell from our Table 4(a)). The reason we chose different alphabetical symbols is that we wanted to separate prompt-token side annotations and LoRA side annotations. We will change them to the same representation to prevent confusion.

**Q:Hyper-parameters configurations; An explanation for these fixed values would be necessary**

* To ensure a fair comparison, we followed previous methods[1,2] for all prompt token settings, including deep prompt tokens, N_CTX, and learning rates.

* Regarding the LoRA component, we initially experimented with rank settings commonly used in conventional LoRA[3], specifically r = 1, 4, 8. Higher ranks such as r = 32, 64 were not considered due to our aim to minimize the number of parameters. We observed that performance improved with increasing ranks in the set r = {1, 4, 8}. We then incrementally increased the rank until r = 12, where we found the performance to be better than at r = 13, prompting us to stop at r = 12.

* For the quantity $m$, we have very limited choices when we have fixed rank. With r = 12 from our previous experiments, we tested all divisor combinations of r and m (i.e., r × m = 12 × 1; 6 × 2; 4 × 3; 3 × 4). Based on these experiments performance, we set the sub-LoRA matrices with r = 4 and m = 3 across all benchmarks.

* Finally, for different $\lambda$, we empirically defined them to ensure that each loss is within a similar magnitude.

1.Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., & Khan, F. S. (2023). Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19113-19122).

2.Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M. H., & Khan, F. S. (2023). Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 15190-15200).

3.Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., ... & Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
Response to Reviewer zPfS
We thank the reviewer for many insightful comments. We answer the questions in what follows. Please let us know if further clarification is needed.

**Q:I strongly suggest the authors consider identifying and focusing on the core components of their method.**

Thank you for your insightful suggestion. Our proposed methods mainly focus on two things: first, prompts can additionally provide task-specific guidance to LoRA (even in plain LoRA settings, as shown in Table 5 in the appendix); second, the proposed gating mechanism strengthens their connection.

**Q:It is unclear how Eqn (4) is optimized. Are $si$ and $AiBi$ learned simultaneously?**

Yes, $AiBi$ and $si$ are learned simultaneously. We provide our algorithm on page 18 of the appendix. 

**Q:How many sub-LoRAs are used, and why is it imperative to decompose a single LoRA into multiple sub-LoRAs essential? Do the learnable and share any functional overlap?**

We provided all hyperparameters in Table 4(a); we use a fixed 3 sub-LoRA matrices in all evaluations.

This LORSS idea is inspired by MoE-LoRA [1], but our approach is more parameter-efficient in terms of learnable parameters. We decompose the LoRA matrix into sub-LoRA matrices under the same parameter budget, while MoE-LoRA duplicates the LoRA matrix into several LoRA matrices. For example, if we have n sub-LoRA matrices with a  fixed rank r and $W \in \mathbb{R}^{d \times k}$ their parameters increase to $n×(d×r+r×k)$, whereas our parameters remain at $(d×r+r×k)$. Another difference is that MoE uses a Network to select the importance of matrices A/B. In contrast, we employ a single learnable parameter (the scaling factor) for each sub-LoRA matrix, which is clearly more efficient. Finally, our downstream tasks are completely different, highlighting the distinct applicability of our method. From our observation, we found that under the same parameters (i.e., 3*r=3 sub-LoRA sitting V.S. r=12 plain LoRA sitting), LORSS always outperforms the plain setting.

The scaling factors $si$ and the gating $G(\cdot)$ are totally different. The $si$ are the weights of different LoRA sub-matrices, while the gating provides a confidence score to the total sum of all LoRA sub-matrices. If we rewrite our euqation 6,7, you can see the scaling factor $s_i$ of LoRA matrices $A_i$ and $B_i$ are independent of the gating prediction:

$$
\Delta W = \left( \beta \sum_{i=1}^{m} \left( s_{i}^{(l)} \times A_{i}^{(l)} B_{i}^{(l)} \right) + \gamma \sum_{i=1}^{m} \left( s_{i}^{(l-1)} \times A_{i}^{(l-1)} B_{i}^{(l-1)} \right) \right) \times G(P_l)
$$
 
where the $\Delta W$ is the final updated matrix and will be added like in normal LoRA.

**Q:Why doesn't the weighting form in Eqn (6) match that in Eqn (5)? This discrepancy should be clarified.**

We thank you for pointing out this issue that many reviewers care about. There is actually a misunderstanding that our 
 $\alpha,\beta,\gamma$  are the same thing and have the same value (0.1 for $(l-1)$ and 0.9 for $l$, as you can tell from our Table 4(a)). The reason we chose different alphabetical symbols is that we want to separate prompt-token side annotations and LoRA side annotations. We will change them to the same representation to prevent confusion.

**Q:DPD-LoRA requires storing LoRAs per layer (Eqn (4)) and also duplicates each encoder in both branches while retaining unprompted inputs, which appears to impose a substantial memory cost.**

In fact, if you look at our Tables 4(a) and (b), where we provide computational complexity among different prompting methods, we only add a few parameters (or even fewer than previous methods!), and our proposed methods show better performance. As previously mentioned in our appendix algorithm, we actually follow a two-step training strategy which does not require much memory cost.


1.Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models. arXiv preprint arXiv:2402.12851, 2024.

2.Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., ... & Zhao, T. (2023). AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512.
Response to Reviewer pw2m
We thank the reviewer for many insightful comments. We answer the questions in what follows. Please let us know if further clarification is needed.


**Q:The motivation for using prompts to guide LoRA learning is not entirely intuitive.**

As we states in introduction. Prompt learning does not contribute to attention weights(lines 78-80), while solely using LoRA cannot provide task-specific guidance due to updating only pre-trained weights (lines 45-47). Therefore, we propose a new method that can integrate task-specific guidance directly into the adaptation mechanism.

**Q:The authors should clarify why applying a weight to each in the LoRA layer solely through gating prompt tokens is expected to be effective.**

Gating is commonly utilized in various deep learning tasks. In our case, it acts like a dynamic weight predictor; it assigns weights between [0,1] to help prevent updating unreliable LoRA matrices. Intuitively, you can consider the output of gating as a confidence score. Meanwhile, we have proven that even without gating (or the confidence score), our proposed method is valid (see $1^{st}$ column pf ablation study in Table 3).

**Q:The explanation of the Gating function requires clarification and potentially overlap in function.**
1. Actually, The LoRA matrices $A_i$ and $B_i$ are independent of the gating prediction. As explained in the previous question, our gating mechanism takes the prompt as input and then predicts weights/confidence scores for these LoRA matrices. The scaling factors si and the gating are totally different. The si are the weights of different LoRA sub-matrices, while the gating provides a confidence score to the total sum of all LoRA sub-matrices. Thus, as show in (eq 4), the prompt token differs across layers, which makes the weights of different layers differ. Therefore, $G(p_l)!=G(p_{l-1})$. 

**Q:Additionally, it is unclear how interacts with the Hierarchical Interaction—does it apply weighting to $A_{i}B_{i}$ at layer $l-1$ as well?**

Hierarchical Interaction and LORSS are actually affected by the gating in every layers, exactly as shown in our Equations (6) and (7). For your convenience, we rewrite it here:

$$
\Delta W = \left( \beta \sum_{i=1}^{m} \left( s_{i}^{(l)} \times A_{i}^{(l)} B_{i}^{(l)} \right) + \gamma \sum_{i=1}^{m} \left( s_{i}^{(l-1)} \times A_{i}^{(l-1)} B_{i}^{(l-1)} \right) \right) \times G(P_l)
$$
 
where the $\Delta W$ is the final updated matrix and will be added like in normal LoRA. We thank you for pointing this out, and we will add additional annotations to this equation.


**Q:Given the complexity of the proposed method and its multiple components, the current ablation study feels insufficient. For example, what is the rationale for decomposing a single LoRA into multiple sub-LoRAs?**

There are actually two additional tables in the appendix you might have overlooked. Table 4(a) provides the full setting, while Table 5 offers an additional ablation study. As you can see from Table 5, which uses only plain LoRA, it actually performs worse than our proposed LORSS. The reason is simple: we found that under the same parameters (i.e., 3*R=3 sub-LoRA sitting V.S. R=12 plain LoRA sitting), LORSS always outperforms the plain setting.

**Q:How do authors define hyper-parameters**

We thank you for pointing out this issue that many reviewers care about. There is actually a misunderstanding that our 
 $\alpha,\beta,\gamma$  are the same thing and have the same value (0.1 for $(l-1)$; 0.9 for $l$, as you can tell from our Table 4(a)). The reason we chose different alphabetical symbols is that we want to separate prompt-token side annotations and LoRA side annotations. We will change them to the same representation to prevent confusion. For different $\lambda$, we empirically defined them to ensure that each loss is within a similar magnitude.

**Q:How does the addition of orthogonal regularization prevent overfitting? More details on this would clarify the choice and its benefits.**

Orthogonal regularization is commonly used in LoRA research, such as in [1]. It can prevent redundancy: orthogonality ensures that the rows (or columns) of the matrices are linearly independent. This reduces redundancy in the learned features, allowing the model to capture more diverse and informative representations.



1.Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., ... & Zhao, T. (2023). AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512.
Response to Reviewer Tgia
Yes
No ethics review needed.
Please refer to the weakness section.
+ The integration of prompts with LoRA represents an innovative exploration in this domain.
+ The authors conducted extensive experiments to substantiate the performance improvements of the proposed algorithm.
This paper presents a dynamic prompt-guided LoRA approach that integrates several key modules: Hierarchical Interaction, a Prompt-Conditioned Gating Mechanism (PCGM), and a Self-Regularized Lower-Rank Subspace. The proposed method is evaluated on 11 benchmark datasets, demonstrating its effectiveness.
+ The motivation for using prompts to guide LoRA learning is not entirely intuitive. The authors should clarify why applying a weight to each $A_i B_i$​ in the LoRA layer solely through gating prompt tokens is expected to be effective.

+ The explanation of the Gating function requires clarification. Does $G(P)$ apply a weight before each $iA_i B_i$? How does this differ from directly learning $S_i$, and could it potentially overlap in function? Additionally, it is unclear how $G(P)$ interacts with the Hierarchical Interaction—does it apply weighting to $A_i B_i$ at layer $l−1$ as well?

+ Given the complexity of the proposed method and its multiple components, the current ablation study feels insufficient. For example, what is the rationale for decomposing a single LoRA into multiple sub-LoRAs? How are hyperparameters $\alpha$, $\beta$, $\gamma$, $\lambda_1$, $\lambda_2$​, and $\lambda_3$ set, and what is their impact on the final performance?

+ How does the addition of orthogonal regularization prevent overfitting? More details on this would clarify the choice and its benefits.
Yes
No
No ethics review needed.
See weakness 2-5
- The paper is well-structured and relatively easy to understand.
- Detailed experiments provide convincing evidence of the effectiveness of the proposed algorithm.
This paper proposes the DPD-LoRA algorithm, which integrates prompt learning to guide the LoRA learning distribution. By incorporating modules such as Hierarchical Interaction, the Prompt-Conditioned Gating Mechanism (PCGM), and the Self-Regularized Lower-Rank Subspace (LoRSS), the proposed DPD-LoRA achieves strong performance across 11 benchmark datasets.
- Overall, the proposed algorithm involves numerous modules. I strongly suggest the authors consider identifying and focusing on the core components of their method.
- It is unclear how Eqn (4) is optimized. Are $s_i$ and $A_iB_i$ learned simultaneously? How many sub-LoRAs $m$ are used, and why is it imperative to decompose a single LoRA into multiple sub-LoRAs essential? Do the learnable $S_i$ and $G(P)$ share any functional overlap?
- Why doesn't the weighting form in Eqn (6) match that in Eqn (5) (e.g., setting $\gamma=1-\beta$) ? This discrepancy should be clarified.
- In Eqn (8), why does the orthogonal regularization prevent overfitting and encourage diversity in the learned LoRA? If this assertion is based on findings from other studies, supporting citations would strengthen the claim.
- I would like to see a memory cost comparison between the DPD-LoRA and SOTA methods. DPD-LoRA requires storing $m$ LoRAs per layer (Eqn (4)) and also duplicates each encoder in both branches while retaining unprompted inputs, which appears to impose a substantial memory cost.
Yes
No ethics review needed.
(1): The phrase “without any additional models prior” in lines 110-111 is somewhat ambiguous. Typically, Parameter-Efficient Fine-Tuning (PEFT) builds on pre-trained models, so it is recommended to clarify whether this refers to the absence of model priors or additional model parameters.

(2): The abbreviation “PEFT” is used for both Prompt-based Efficient Fine-Tuning and Parameter-Efficient Fine-Tuning, which may lead to confusion. It is advisable to select distinct abbreviations to improve clarity.

(3): The term “PLoRA” in line 378 is confusing, as its specific reference is unclear. Further definition or clarification of this acronym is recommended for improved reader comprehension.
(1) The proposed methods are relatively comprehensive, using several points to improve existing problems.

(2) The writing is clear and easy to understand.
DPD-LoRA uses task-specific prompts to dynamically influence the low-rank updates of model parameters, enhancing the model's adaptability across diverse tasks and mitigating forgetting issues. By decomposing the standard low-rank adaptation into multiple low-rank sub-matrices, the method retains flexibility without adding additional parameters, thus improving the model’s learning capacity. An adaptive loss function is introduced to ensure alignment between the adapted distribution and the pre-trained model, thereby enhancing learning effectiveness and stability. A self-regulating mechanism is used to further improve model stability, along with a soft-gating mechanism to determine when to activate adaptation modules, ensuring improved performance on new categories.
(1): In line 041, “LLaVa” should be revised to “LLaVA” for consistent terminology throughout the document, avoiding unnecessary visual inconsistency.

(2): The related work section lacks references to significant LoRA extensions, such as DoRA, SVFT, PISSA, and LoRA-XS. It is recommended to include these studies and discuss how the proposed method compares to or builds upon these prior approaches. Specifically, it would be helpful to highlight the innovations of this work and the advantages it has over these extensions.

(3): The method incorporates a distillation-like Self-Constrain Loss, but there is no evaluation of training time, GPU resource consumption, or other efficiency-related metrics. Providing specific efficiency metrics, such as training time per epoch, peak GPU memory usage, and FLOPs, would substantiate the claims of being resource-efficient. Including a comparison of these metrics to baseline methods would further support the efficiency claims.

(4): The ablation study section only presents the individual performance of each component without evaluating the performance of their combinations. Adding experiments that evaluate different component combinations (e.g., two, three, and all four components) would provide a more comprehensive view of the model's performance. Including a table or figure showing these combinations or using an approach like forward selection to systematically evaluate the synergies between components would be very informative.

(5): The comparative experiments do not include related LoRA methods, such as DoRA and VeRA. Including comparisons with these methods would more clearly demonstrate the advantages of the proposed approach. It is suggested to add a specific experiment or table comparing the proposed method to DoRA, VeRA, and other relevant LoRA variants on key metrics or datasets to provide a clearer demonstration of its benefits.
Yes
No ethics review needed.
1. In the hierarchical interaction section, both prompt tokens and LoRA layers establish connections between the current layer and the previous one to prevent information loss across layers. However, different weight allocation methods are used: α and 1-α for prompt tokens, and β and γ for LoRA layers. It would be beneficial to explain the rationale and necessity for using different methods when their purposes are aligned.
2. The paper sets a considerable number of hyperparameters, including learning rates, weight factors, deep prompt tokens, etc., and uses a fixed rank r and quantity m for LoRSS configurations across three different tasks. The paper does not seem to discuss the rationale behind these settings or how different ranks and quantities might impact the results. An explanation for these fixed values would be necessary.
1. The proposal of the DPD-LoRA framework is innovative as it integrates prompt learning and low-rank adaptation to enhance the model's generalization capabilities. The introduction of adaptive loss functions and soft-gated selection mechanisms (PCGM) adds to the novelty of the approach.
2. The method designed by the authors has been applied to three different tasks: Base-to-novel class generalization, Cross-dataset evaluation, and Few-shot learning, showing promising results across the board, which speaks to the effectiveness of the approach.
3. The authors have conducted extensive experiments on 11 benchmark datasets, which helps to substantiate the effectiveness of the proposed method.
4. The overall structure of the paper is relatively clear, with proper introductions to various techniques, facilitating the reader's understanding of the content.
This manuscript introduces DPD-LoRA, a novel framework that aims to improve the generalization capability of large models by integrating dynamic prompt-driven low-rank adaptation. This method combines hierarchical prompt tokens and parameter-efficient adaptation to incorporate task-specific guidance, demonstrating superior performance over existing techniques across multiple benchmark datasets.
1. The paper employs a variety of techniques and methods, including prompt learning, LoRA, gating mechanisms, and loss design, with five points listed in the INTRODUCTION under contributions and five in the METHOD section. This can seem a bit cluttered and redundant; a more concise summary and consolidation of related content would be beneficial.

2. Section 3.2 is titled "PROMPT LEARNING WITH LOW RANK ADAPTATION IN TRANSFORMERS," yet the explanation seems to treat prompts and LoRA separately, although the appendix provides a detailed explanation of their combined effect. As this is a crucial part of the paper, more clarity and detail in the main body of the text would be necessary.

3. The title of Section 3.3 is "HIERARCHICAL INTERACTION AND EXPANDED SUBSPACES," but the content first introduces expanded subspaces and then hierarchical interactions. The order of introduction and the content should correspond to the title.

4. The paper slightly lacks in-depth analysis of the synergistic effects between LoRA and prompt learning. Although an ablation study is conducted, showing experimental results under different conditions, a deeper analysis of how these components interact and contribute to performance improvements is needed, especially considering this is the core and key of the paper.
We propose DPD-LoRA, a method that uses learned prompt tokens to guide enhanced low-rank feature spaces, achieving improved performance in both adaptation and generalization for downstream tasks.
@inproceedings{
anonymous2024dpdlora,
title={{DPD}-Lo{RA}: Dynamic Prompt-Driven Low-Rank Adaptation for Improved Generalization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Dci14asFPV},
note={under review}
}
Fine-tuning large models presents technical challenges such as catastrophic forgetting and parameter inefficiency. Low-rank Adaptation (LoRA) and Propmt Learning can help address some of these challenges by providing more compact and flexible representations. However, Low-rank approximation is susceptible to outliers and relies on the assumption of a global low-rank structure, which can be suboptimal. Additionally, Prompt learning can overfit to specific downstream tasks, reducing its effectiveness when adapting to new tasks. In this paper, we introduce $\textbf{Dynamic Prompt-Driven Low-Rank Adaptation (DPD-LoRA)}$, a novel framework that seamlessly integrates task-specific guidance using hierarchical prompt tokens and parameter-efficient adaptation. Unlike traditional methods, task-aware prompts in the DPD-LoRA dynamically influences low-rank updates in the model's parameters, thus enabling robust adaptation and generalization across diverse tasks and mitigating the forgetting issues. We further improve the learning capabilities of the model by breaking down the standard LoRA into multiple low-rank sub-matrices, without adding additional parameters. Further, we use an adaptive loss function to guarantee alignment with the distribution of the pre-trained model. Specifically, we introduce a self-regulated mechanism to improve stability, and a soft-gated selection mechanism to decide when to activate adaptation modules to improve performance on unseen categories. Extensive experiments on 11 benchmark datasets demonstrate that DPD-LoRA significantly outperforms state-of-the-art methods in both accuracy and generalization, offering a comprehensive solution to the challenges of fine-tuning large-scale models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Vision-Language Models PEFT Prompt Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a0c802f48899f849fd11ea1cdc86112e26899298.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
DPD-LoRA: Dynamic Prompt-Driven Low-Rank Adaptation for Improved Generalization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission