Prompt Diffusion Robustifies Any-Modality Prompt Learning
Prompt learning;Diffusion model;Vision-language models
Yes
No ethics review needed.
1.  Is the author's approach a two-stage process, starting with a prompt study followed by a prompt proliferation.
2. Diffusion models incorporate randomness in the generation process, which may lead to uncontrollable fluctuations in the generated prompts and thus affect the robustness of the model. How to cope with the randomness of the generated prompts and avoid the instability of prediction caused by it?
3. The authors' approach seems to be applicable only to VPT-shallow prompt types, and whether the authors' approach can be migrated to the VPT-deep prompt learning paradigm.
1. The method in this paper generates customized prompts for each sample by gradually optimizing the prompts through diffusion, which enhances the accuracy of prediction and generalization across downstream tasks.
2. The diffusion prompting method in this paper is a plug-and-play module that can be seamlessly integrated into existing textual, visual, or multimodal prompt learning methods.
3. The method in this paper improves the prompt learning process by efficiently extracting unique domain details from test images without mixing them with class labels.
This paper proposes a method called Prompt Diffusion, which employs a diffusion model to progressively refine prompts, enabling customized prompts for each sample. By introducing a technique for creating tailored prompts for individual test samples, this method addresses the limitations of fixed prompts, enhancing the model's robustness to distribution shifts. Empirical results on extensive datasets validate the effectiveness of this approach, demonstrating its robustness in generalization tasks.
1.  The authors' method requires stepwise optimization of the prompts and may require several iterations to obtain optimal results, in addition, the introduction of a diffusion model increases the complexity of the system, and therefore whether the training time is likely to be relatively long.
2.  Whether the authors' approach is a two-stage process, where prompt learning is performed first, followed by diffusion of the prompts, and the final model performance relies on the goodness of the previously learned prompts. In addition, the diffusion process relies on random noise vectors to generate the prompts and therefore may be sensitive to noise, which may affect the stability of the final performance.
Yes
No ethics review needed.
1. I am curious about the setting of the two loss weights β in Equation (8). Can further experimental analysis be provided?
1. Experiments have shown that the proposed method outperforms baseline methods.​

2. The overall idea is intuitive and straightforward, addressing the limitations of fixed prompts by leveraging diffusion models to generate over-fitted prompts per sample, which enhances model robustness against distribution shifts.
In this paper, the authors introduce prompt diffusion, which utilizes a diffusion model to refine prompts for each input image, thereby enhancing the model's ability to generalize across different distributions. The proposed prompt diffusion is a straightforward plug-and-play module that can be seamlessly integrated into existing prompt learning frameworks.
1. Considering that the proposed method is conducted on per sample. during training, does it introduce a significantly larger computational load compared to conventional prompt learning methods? Can a comparative analysis be provided to address this concern?

2. While the proposed method is plug-and-play and the pipeline figure demonstrations are based on CoCoOp, it would be beneficial to include sections addressing visual prompt tuning and multi-modal prompt tuning.  Additionally, the method emphasizes the meta-net π within CoCoOp, but it is unclear how it handles other prompt learning methods that do not involve π, such as VPT and MaPLe.

3. The length of prompts in prompt learning methods can affect the final performance.  Does the proposed method also encounter similar situations?  It is encouraged for the authors to supplement relevant ablation studies to address this concern.

4. There are also some works in the field of prompt learning that address the limitations of fixed prompts by generating instance-level prompts (e.g. [1]).  It is recommended that the authors supplement the related work to make the paper more comprehensive.

[1] Xinyang Liu ,et al. Patch-Token Aligned Bayesian Prompt Learning for Vision-Language Models. UAI 2024
Yes
No ethics review needed.
1. What are the key motivations behind using diffusion models for prompt learning, and how does it address the limitations of fixed prompts?
2. How does Prompt Diffusion leverage the diffusion model to gradually transition from a random to a sample-specific prompt?
3. In what ways does Prompt Diffusion enhance generalization capabilities across base-to-new, cross-dataset, and domain generalization tasks?
4. How does Prompt Diffusion ensure compatibility with existing prompt learning models across textual, visual, and multimodal prompts?
1.	Introduces an innovative, modality-agnostic diffusion process that significantly enhances robustness in prompt-based learning.
2.	Demonstrates consistent empirical improvements across various prompt learning tasks, supporting the efficacy of diffusion models.
3.	Efficient design reduces inference time, making it suitable for diverse real-world applications.
This paper introduces a novel framework, Prompt Diffusion, which aims to improve the generalizability and robustness of prompt-based learning across various modalities (e.g., visual, textual, multimodal). In prompt-based learning, especially for foundation models in zero-shot and few-shot learning, fixed prompts often suffer from distributional shifts, impacting performance on unseen data. Prompt Diffusion leverages a diffusion model to refine prompts gradually, transforming them from a generic to a sample-specific prompt. This process enhances the robustness and adaptability of prompts across datasets with distinct distributions, providing a plug-and-play solution compatible with existing prompt-learning methods.
1.	The paper does not fully articulate the specific limitations of the SOTA prompt mehthods in adapting to distributional shifts in data, which creates ambiguity around the critical nature of these issues within broader prompt-learning applications. To make this critique more actionable, the authors could quantify the performance degradation caused by these shifts in existing methods to better contextualize the importance of their contribution.  Specific examples are not enough to illustrate the problem.
2.	Although the diffusion model is proposed to generate sample-specific, customized prompts, the paper does not clearly explain why diffusion was chosen over other, potentially simpler methods. This raises questions about the model's unique contributions and practical effectiveness. For instance, if simpler statistical methods like ProDA[1] are available, what advantages does the complex diffusion model offer? Moreover, there are already several statistical approaches for prompt learning, such as Bayesian Prompt Learning[2], which the authors could consider referencing.
3.	The approach has limited empirical exploration outside the image-text domain, raising questions about its generalizability to other modalities. To strengthen this point, the authors could discuss the potential challenges and adaptations needed to apply their method to other modalities, such as audio or video. 
4.	The high resource demands of diffusion models, including substantial GPU and training time requirements, make them impractical for parameter-efficient methods such as prompt learning. The complexity and cost of implementing diffusion models in this context undermine their accessibility and practicality. 

[1] Lu, Yuning, et al. "Prompt distribution learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Derakhshani, Mohammad Mahdi, et al. "Bayesian prompt learning for image-language model generalization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
This paper introduces prompt diffusion, which uses a diffusion model to gradually refine prompts to obtain a customized prompt for each sample.
@inproceedings{
anonymous2024prompt,
title={Prompt Diffusion Robustifies Any-Modality Prompt Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=2x1U8a3s7G},
note={under review}
}
Foundation models enable prompt-based classifiers for zero-shot and few-shot learning. Nonetheless, the conventional method of employing fixed prompts suffers from distributional shifts that negatively impact generalizability to unseen samples. This paper introduces prompt diffusion, which uses a diffusion model to gradually refine prompts to obtain a customized prompt for each sample. 
Specifically, we first optimize a collection of prompts to obtain over-fitted prompts per sample. Then, we propose a prompt diffusion model within the prompt space, enabling the training of a generative transition process from a random prompt to its overfitted prompt. As we cannot access the label of a test image during inference, our model gradually generates customized prompts solely from random prompts using our trained, prompt diffusion. Our prompt diffusion is generic, ﬂexible, and modality-agnostic, making it a simple plug-and-play module seamlessly embedded into existing prompt learning methods for textual, visual, or multi-modal prompt learning.
Our diffusion model uses a fast ODE-based sampling strategy to optimize test sample prompts in just five steps, offering a good trade-off between performance improvement and computational efficiency.
For all prompt learning methods tested, adding prompt diffusion yields more robust results for base-to-new generalization, cross-dataset generalization, and domain generalization in classification tasks tested over 15 diverse datasets.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Prompt learning Diffusion model Vision-language models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/056756f56239effc7b9d64d580cd8f6620882956.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Prompt Diffusion Robustifies Any-Modality Prompt Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission