Diversity-Enhanced and Classification-Aware Prompt Learning for Few-Shot Learning via Stable Diffusion
meta-learning;synthetic dataset generation;diffusion model
Yes
No ethics review needed.
Could you provide a few examples of prompts that were either consistently selected or commonly rejected across different datasets? Additionally, would it be possible to include visualizations that demonstrate the prompt selection process over the course of your optimization? Such examples and visualizations would greatly aid in understanding the behavior of your DeCap method and the rationale of the prompts it selects for synthetic image generation？

A critical aspect of DeCap's approach is the ability to generate diverse and classification-aware synthetic images, which hinges on the richness and detail of the prompt pool. My concern is whether the current prompt pool contains a sufficient level of fine-grained detail to effectively capture the diversity required for various few-shot learning tasks. Could you elaborate on how you ensure that the prompts in your pool are not only comprehensive but also detailed enough to represent the nuances of different classification tasks? Additionally, how do you assess whether the prompts selected across different optimization iterations and datasets are indeed the most contributive ones in terms of fine-grained detail?

Could you consider expanding the evaluation framework to include a range of "n-shot n-way" scenarios? The paper would benefit from a more holistic evaluation that includes the F1 score, precision, and recall in your analysis. Specifically, how would these metrics help in understanding the model's generalization performance in terms of false positives and false negatives, which are critical considerations in few-shot learning?
The innovation of this work lies in the adaptive learning of prompts for synthetic data generation, which is tailored to the specific requirements of few-shot learning tasks. 
This work use a genetic algorithm for the outer-loop optimization to search for the optimal prompt set, which is a discrete search problem. The inner-loop optimization involves training a classification model on synthetic data generated by the selected prompts.
The paper demonstrates that DeCap can improve the performance of existing zero/few-shot learning methods in different network architectures
The paper presents DeCap, a novel method for enhancing few-shot learning through the generation of diverse and classification-aware synthetic images using text-to-image generative models.
The proposed automated prompt learning method reduces the manual effort required to craft effective prompts and its meta-learning strategy can align synthetic data generation with downstream classification tasks to improve the performance in various network architectures.
A significant concern with DeCap's approach is the adequacy of the prompt pool in capturing the full spectrum of variations necessary for diverse few-shot learning tasks. The paper does not provide a clear methodology for ensuring that the prompt pool is comprehensive enough, or that the most contributive prompts are consistently selected across different optimization iterations and datasets. This lack of clarity raises questions about the reliability and robustness of the prompt selection process.

The paper primarily relies on accuracy as the performance metric for evaluating the few-shot learning models. However, in the context of few-shot learning, a more nuanced evaluation that includes "n-shot n-way" scenarios and a suite of metrics such as precision, recall, and F1 score is essential. The current approach may not fully capture the model's performance across different classification tasks, especially when the distribution of samples per class varies significantly. This limitation restricts the comprehensive assessment of the model's robustness and generalization capabilities in few-shot settings.
Yes
No ethics review needed.
Comments in weaknesses must be resolved.
- The method is easy to follow.
- It proposes a total solution for few-shot classification problem.
The authors propose a prompt learning method to boost classification performance. To overcome the lack of diversity of hand-crafted prompts, they also leverage prompts from language models. Once a prompt pool has been created, genetic algorithm finds out the best working prompt for classification performance.
- At the first glance of this paper, the though 'commercially or open-sourced LLM or Multi-modal LLM will show much better performance with even not using genetic algorithm' came up in my mind. That means many readers would think just the same I did and the authors are responsible for prove them wrong. I have tried myself requesting various prompts to a commercial LLM and the results seemed much better than the examples in this paper.
- The task is limited to classification problem and datasets used for evaluation are insufficient to make this paper solid. It is hard to tell if this paper worth dropping another paper for this conference.
Yes
No ethics review needed.
My questions regarding efficiency and generalization capability are shown in the weakness.
1. This paper presents an innovative strategy that involves constructing more diverse prompts and introduces a prompt-learning technique tailored for downstream classification tasks to achieve optimal prompts. The approach is novel, and its effectiveness is supported by experimental results.
2. The paper systematically compares and summarizes various methods of data curation through prompts in T2I models. This overview is valuable for readers seeking to understand progress in this area.
The paper proposes a Diversity-Enhanced and Class-Aware Prompt (DeCap) learning strategy to discover appropriate text prompts for downstream few-shot classification tasks, which addresses the issue of the lack of diversity in existing prompt-based methods. The proposed method includes a diversity-enhanced prompt pool construction and a class-aware prompt learning strategy. Empirical experiments are included to demonstrate the effectiveness of this approach.
1. Although this work proposes an interesting gradient-free method for discovering effective prompts for synthetic data, the main limitation is that “**attempting to approximate the few-shot data domain by altering prompts is inefficient and lacks generalization capability**”.

- **Regarding efficiency**: The meta-learning paradigm requires substantial sampling, and the pipeline includes too many uncontrolled components. For instance, generating images from text involves a high degree of freedom, while even slight variations in the images can significantly alter the classifier’s decision boundary. Thus a discussion of the running time should be included, along with an analysis of the strategy’s robustness with respect to T2I hyperparameters beyond prompts, such as generation using fixed seed to a random seed.
- **Regarding generalization**: The genetic algorithm essentially searches for effective prompts to approximate the domain of the few-shot dataset (assuming Equation 2 uses accuracy on the few-shot data as the fitness function for prompts rather than accuracy on the test set). Consequently, this approach may limit performance on out-of-domain test scenarios. Results on ImageNet domain generalization datasets, such as ImageNet-R, would be beneficial to verify the generalization capability. 

2. Some recent data augmentation studies are not included [1,2,3]. Leveraging the inherent knowledge in few-shot images directly to curate data, bypassing text prompts, may offer a more efficient solution. Adding discussions and comparisons between prompt-based and image-based (e.g., image editing) data curation methods would provide better insight, such Real-Guidance [4] and Da-fusion [1].


3. The impact of varying shot numbers on the prompt-mining strategy is not addressed. Section 2.2 indicates that all experiments used a 10-shot setting. I suspect that the strategy may be ineffective in extremely low-shot cases (e.g., 1-shot). As noted in Weakness #1.2, this could result in synthetic data failing to adequately cover the evaluation dataset. Can you provide the more results across scenarios with varying shot numbers?

[1] DA-fusion: Effective Data Augmentation With Diffusion Models. Brandon Trabucco, et-al.

[2] Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model. Zhicai Wang, et-al.

[3] Diffusemix: Label-preserving data augmentation with diffusion models. Khawar Islam, et-al.

[4] Is synthetic data from generative models ready for image recognition? Ruifei He, et-al
Yes
No ethics review needed.
1) What is your justification for dataset selection? Does it align with previous work?

2) What is the cost of DeCap? I'm not finding the breakdown, but please point me to it if it is already there.
- Well written

- Novel methodology

- Good show of qualitative results
Authors tackle the problem of synthetic data generation for downstream training. They look at this from the perspective of increasing prompt diversity, to increase image diversity. This do this by generating proper prompts through meta learning of the prompt pool. They evaluate this against other prompt-learning methods and on top of SOTA methods.
Weaknesses are given in order of how I perceive their importance (issues that weigh heavier in my review decisions are first):

1) I find the main experimental results comparing to SOTA prompt methods in Table 1 lacking (specifically in terms of datasets), for the following reasons:

1a) The number of datasets evaluated is lacking very far behind other SOTA methods. E.g. Sus-X (cited in this paper) trains on 19 datasets, only three of which are used here (and two subsets). Furthermore, the datasets used in DeCap are quite similar in terms of subject, granularity, modality, etc. This is concerning because the problem statement of generating training images is highly class / dataset dependent. To be rigorously shown, this needs to be evaluated on a diverse variety of datasets. I believe that to support the claims in this paper, this method would need to be evaluated on a minimum of 3 / 4 of the following categories: a FG dataset (e.g. Stanford Cars of FGVC Aircraft), two more 'types' of objects (e.g. Food101, FGVC Aircraft, assuming animals has already been covered by Pets), a different modality (e.g. EuroSAT, ImageNet-Sketch), a scene-based dataset (e.g. SUN397). I would like to reiterate that I believe this is the minimum, and more ideally a few more strategically chosen datasets should be evaluated.

1b) CIFAR and STL are 32x32 and 96x96 resolutions, respectively. SDXL-Turbo generates images at 512x512. Hence, they are too far out-of-distribution for how the diffusion model generates to be strong evidence in themselves. I don't find it problematic in itself to include them, but these datasets being weak further exacerbates the issue described in 1a.

2) Intuitively, DeCap looks expensive. I'm not convinced that the performance gains are enough to justify the cost (unless you correct me on the cost). This would be stronger if the GPU hours were compared between DeCap and other SOTA in table 1 (LE, CiP) in addition to how much it adds for other SOTA methods (FakeIt, SuS-X, CaFo).

3) When combining DeCap with SOTA, significant gains are only seen on FakeIt. Already SuS-X and CaFo (more recent works) show gains of only 1% or less on average. There are two outcomes that I am concerned these results imply at least one of the following:

3a) FakeIt is trained from scratch, while the other two fine-tune SOTA models. It could be that in the context of fine-tuning pre-trained models, DeCap does not affect as much.

3b) FakeIt is older and both SuS-X and CaFo are newer. It could be that the challenges DeCap combats have been already well-addressed by newer works. 

4) The work misses ablations on the design choices. E.g. why not optimize the prompt without meta learning, choice of loss function, choice of meta learning algorithm, etc.

5) The ablations on number of prompts, etc. are done on STL-10 (see 1b)
Using meta-learning to learn classification-aware prompts for synthetic dataset generation
@inproceedings{
anonymous2024diversityenhanced,
title={Diversity-Enhanced and Classification-Aware Prompt Learning for Few-Shot Learning via Stable Diffusion},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=PulKaNibeQ},
note={under review}
}
Recent text-to-image generative models have exhibited an impressive ability to generate fairly realistic images from some text prompts. In this work, we explore to leverage off-the-shelf text-to-image generative models to train non-specific downstream few-shot classification model architectures using synthetic dataset to classify real images. Current approaches use hand-crafted or model-generated text prompts of text-to-image generative models to generated desired synthetic images, however, they have limited capability of generating diversity images. 
Especially, their synthetic datasets has relatively limited relevance to the downstream classification tasks. This makes them 
fairly hard to guarantee training models from synthetic images are efficient in practice. To address this issue, we propose a method capable of adaptively learning proper text prompts for the off-the-shelf diffusion
model to generate diverse and classification-aware synthetic images. Our approach shows notable improvements in various
classification datasets, with results comparable to existing prompt designing methods. 
We find that replacing data generation strategy of existing zero/few-shot methods with proposed method could consistly improves downstream classification performance across different network architectures, demostrating its model-agnostic characteristic for few-shot learning. This makes it possible to train an efficient downstream few-shot learning models from synthetic images generated by proposed method for real problems.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
meta-learning synthetic dataset generation diffusion model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7f7dffc11ef2ef21392742e4b8a42042a036dadc.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Diversity-Enhanced and Classification-Aware Prompt Learning for Few-Shot Learning via Stable Diffusion
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission