Divergence-enhanced Knowledge-guided Context Optimization for Visual-Language Prompt Tuning
visual-language prompt tuning;few-shot learning;zero-shot learning
Yes
No ethics review needed.
see weakness
1. This paper is well-organized and easy to follow. Figure 2 effectively illustrates the main idea by clarifying the roles of each loss function: the \( L_{CE} \) loss enforces alignment between text and vision embeddings, the \( L_{kg} \) loss encourages the learnable prompts to align closely with the CLIP textual embeddings, and the core \( L_{HSIC} \) loss ensures independence within the learnable prompt embeddings.

2. The experiments are comprehensive, covering base-to-new generalization, cross-dataset generalization, and few-shot classification. The proposed DeKgTCP method achieves superior results across most datasets.
This paper introduces a simple yet effective knowledge-based prompt tuning method that leverages the Hilbert-Schmidt Independence Criterion (HSIC) to regularize learnable prompts. By reducing the reliance on prior general knowledge, this approach enables the prompts to better align with task-specific knowledge. The method is versatile and can be easily integrated into other frameworks. When applied to the TCP method, it demonstrates superior performance across most datasets.
1. Why was the proposed method applied to KgCoOp and TCP rather than other state-of-the-art methods, such as PromptSRC, which performs even better than KgCoOp? Is it more challenging to integrate with PromptSRC, or are the results less effective? Providing additional clarification on this choice would enhance the paper.

2. Figure 4 provides an insight into how the proposed method balances dependence and independence; however, the paper lacks further analysis on this. Expanding on this point would strengthen the reader’s understanding of the method's underlying mechanics.
Yes
No ethics review needed.
Please see #Weaknesses
+Using the Hilbert-Schmidt Independence Criterion (HSIC) is an interesting topic for encouraging independence between learnable and crafted prompts, which can boost performance in the seen classes.

+Evaluation shows the effectiveness of the proposed method.

+The proposed DeKg integrates seamlessly with existing knowledge-guided methods.
This paper tackles the inherent issue of knowledge-guided context optimization, which overly biases general knowledge in pre-training. It proposes a novel HISC-based regularization method, DeKg, for encouraging independence between the learnable and the crafted prompts. Extensive experiments demonstrate the superiority of the proposed method in three challenging benchmarks:
-As shown in Figure 1, the proposed DeKg obtains a higher performance than the performance of CoOp for base classes and the zero-shot CLIP for new classes. However, the Hilbert-Schmidt Independence Criterion (HSIC) contained in DeKg is a constraint between the learnable and crafted prompts without injecting additional information. Why can the proposed DeKg obtain a better performance?

-L221: The proposed L_{kd} involves two terms: intra-class relations and inter-class relations. Moreover, the author claims that  penalizing
L_{kd} encourages both intra-class and inter-class independence. Furthermore, the intra-class consistency is formulated between w_i and w_{i}^{clip}, which is the same as the L_{kg}. In other words, the proposed HSIC has contained the knowledge consistency L_{kg}. Therefore, the final objective of Eq.(5) should not contain L_{kg} because L_{kd} has been constrained by the intra-class consistency. However, the results in Table 4 are inconclusive with the above conclusion. Even more unfortunate, L_{kd} performs worse than L_{kg}. Why?

-It is recommended to provide a code.

-Since the proposed HSIC is model-independent, it is suggested that the module's generalization and plug-and-play be verified using more CoOp-based methods.
Yes
No ethics review needed.
1. Could you explain why the accuracy for each dataset in Table 3 is around 70%? This seems somewhat unusual; is there a specific reason for this phenomenon?
2. In the base-to-novel generalization experiments, did you use full data or few-shot data? This detail appears to be missing from the paper; please clarify.
3. The three novel loss functions mentioned in the paper seem to be applicable as plug-and-play modules in existing prompt tuning methods. If possible, please include corresponding experimental results to validate this.
4. Please analyze the main differences between this method and prompt tuning approaches, as well as the amout of parameters fine-tuned under each experimental setting.
5. Method 3.3 does not appear to be a novel loss function design, as similar design principles have already been presented in the prompt tuning-related papers KgCoOp and PromptSRC. Please discuss this further.
1. CLIP-CITE offers a promising new direction for model fine-tuning in data-scarce settings, holding high research and practical value.
2. The paper provides thorough experimental validation of CLIP-CITE, demonstrating its effectiveness across tasks such as few-shot learning, domain generalization, and cross-domain generalization. Comprehensive experimental setups and detailed ablation studies highlight the contributions of each component.
This paper introduces CLIP-CITE, a fully fine-tuning method tailored to enhance Vision-Language Models (VLMs), such as CLIP, in data-limited scenarios. Unlike prompt tuning approaches, which often face challenges with generalization and flexibility, CLIP-CITE revisits full fine-tuning and leverages three specific loss functions: a discriminative visual-text alignment loss, supervised contrastive loss, and vision-language similarity distillation loss. These loss functions aim to improve semantic alignment, preserve learned knowledge, and mitigate overfitting, allowing the model to perform effectively on specific tasks while maintaining versatility across domains. Experimental results demonstrate that CLIP-CITE consistently outperforms prompt tuning and parameter-efficient methods across various settings, establishing it as a robust approach for domain-specific fine-tuning with limited supervision.
1. The paper primarily compares CLIP-CITE against a limited set of existing methods (e.g., prompt tuning approaches). 
2. While the paper asserts that CLIP-CITE achieves superior performance with fewer training iterations, it lacks a detailed discussion of the computational efficiency compared to other methods. 
3. The ablation studies could be enhanced. For example, exploring more granular combinations of the proposed loss functions or varying the hyperparameters across a wider range could provide deeper insights into their interactions and the sensitivity of the model to these changes.
Yes
No ethics review needed.
Please refer to the concern in the Weakness section.
1. I think this paper has a reasonable motivation to maximize the independence between learnable prompt and manual prompt. 
2. This paper has a very extensive experiment analysis on varies clip model adaptation task and show good results.
This paper proposes to adapt HSIC as an extra regularization term, which achieves a better trade-off between the performance on base and new classes. The experiments show the effectiveness of this regularization on varies experiment settings. The paper is well written and easy to understand.
1. More analysis is needed to discuss why HISC is chosen as the metrics to measure the prompt independence. Other methods like information bottleneck can do that too. 
2. L_kd and L_kg seems to be a pair of totally contradictive losses. I wonder if this will cause the model to be difficult to converge. It would be better to provide more analysis on how the loss weight of these two losses affect the model convergence. 
3. More performance comparison and analysis on other state-of-the-art prompt tuning method like:
Yubin, et.al, Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models
@inproceedings{
anonymous2024divergenceenhanced,
title={Divergence-enhanced Knowledge-guided Context Optimization for Visual-Language Prompt Tuning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6wOmHdwCC4},
note={under review}
}
Prompt tuning vision-language models like CLIP has shown great potential in learning transferable representations for various downstream tasks. The main issue is how to mitigate the over-fitting problem on downstream tasks with limited training samples. While knowledge-guided context optimization (Yao et al.,2023; 2024) has been proposed by constructing consistency constraints to handle catastrophic forgetting in the pre-trained backbone, it also introduces a potential bias toward pre-training. This paper proposes a novel and simple Divergence-enhanced Knowledge-guided Prompt Tuning (DeKg) method to address this issue. The key insight is that the bias toward pre-training can be alleviated by encouraging the independence between the learnable and the crafted prompt. Specifically, DeKg employs the Hilbert-Schmidt Independence Criterion (HSIC) to regularize the learnable prompts, thereby reducing their dependence on prior general knowledge, and enabling divergence induced by target knowledge. Comprehensive evaluations demonstrate that DeKg serves as a plug-and-play module can seamlessly integrate with existing knowledge-guided methods and achieves superior performance in three challenging benchmarks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
visual-language prompt tuning;few-shot learning;zero-shot learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2f2e32cf76eef9890ca741303b51a5a326ef9524.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Divergence-enhanced Knowledge-guided Context Optimization for Visual-Language Prompt Tuning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission