Attribute-based Visual Reprogramming for Image Classification with CLIP
Visual Reprogramming;Model Reprogramming;Vision-Language Models;Image Classification
Yes
No ethics review needed.
1. How significantly would limited or unavailable access to these large language models impact the method's effectiveness?

2. Is it possible to use smaller or open-source language models to generate attribute descriptions to reduce dependence on large models?

3. Does the paper provide mechanisms to assess and ensure the quality and relevance of attribute descriptions generated by large language models?

4. If the generated top attributes do not match the target categories, how does it impact model performance? 

5. Has consideration been given to validating the method's effectiveness on larger or more challenging datasets to ensure its generalizability?

6. Although the AttrVR method leverages learnable noise, how does its performance improve in more complex image scenarios?
This paper addresses the limitations of Visual Reprogramming (VR) in vision-language models like CLIP by introducing Attribute-based Visual Reprogramming (AttrVR). Unlike traditional VR methods that rely on fixed text templates and category labels, AttrVR leverages Descriptive Attributes (DesAttrs) and Distinctive Attributes (DistAttrs) generated by large language models (e.g., GPT-3.5) to capture both common and unique features of each category. During training, AttrVR dynamically selects relevant attribute descriptions for each image, optimizing input noise patterns to better capture detailed features and reduce category confusion. This attribute-based approach significantly improves classification performance across 12 downstream tasks with ViT and ResNet backbones, even with limited training samples, demonstrating robustness and effectiveness. Additionally, theoretical analysis and visualizations show that AttrVR reduces intra-class variance and enhances inter-class separability, thereby enriching the model's embedding space and offering new insights for advancing vision-language model applications. Overall, AttrVR effectively combines the strengths of visual and language models, addressing the shortcomings of label-based VR methods and fully utilizing CLIP's capability to understand rich textual descriptions, thereby advancing the application of vision-language models in downstream tasks.
This paper introduces a new method called Attribute-based Visual Reprogramming (AttrVR) for CLIP-based image classification tasks. Unlike traditional label-based visual reprogramming methods, AttrVR fully leverages CLIP's rich attributes and diverse textual representation capabilities. The method introduces Descriptive Attributes (DesAttrs) and Distinctive Attributes (DistAttrs), which represent the common and unique features of different categories, respectively. Additionally, AttrVR achieves more dynamic and sample-specific optimization by iteratively optimizing the $k$ nearest DesAttrs and DistAttrs for each image sample. Theoretical analysis shows that AttrVR can reduce intra-class variance and increase inter-class distance, thereby improving classification performance. Experimental results demonstrate that, whether using ViT-based or ResNet-based CLIP, AttrVR outperforms other VR methods across 12 downstream tasks. Visualization results further confirm the effectiveness of AttrVR in enhancing image-attribute alignment. In summary, AttrVR not only improves CLIP's performance in downstream image classification tasks but also provides a more effective integration scheme for transitioning visual reprogramming techniques from unimodal visual models to multimodal visual-language models.
1. **Dependence on Large Language Models:** The AttrVR method relies on large language models (e.g., GPT-3.5) to generate descriptive and distinctive attributes. This dependence may increase computational costs and limit the method's applicability in resource-constrained environments.

2. **Reliability of Attribute Generation:** The attribute descriptions generated by large language models may have issues with accuracy and relevance. If the generated attributes do not match the target categories, they could negatively impact the model's performance.

3. **Computational Complexity and Efficiency:** AttrVR queries 𝑘 nearest neighbor attributes for each image sample in every training cycle, increasing computational overhead. This may impose pressure on training time and resource requirements for large datasets.

4. **Scope of Experimental Validation:** The paper primarily conducts experiments on 12 downstream tasks. It remains unclear whether the method's effectiveness can be validated on larger or more challenging datasets.

5. **Impact of Hyperparameter Selection:** Hyperparameters in AttrVR (e.g., the number 𝑘 of nearest neighbor attributes) significantly affect model performance. While the paper selects the optimal 𝑘=3, it does not demonstrate whether this choice is equally suitable in more general scenarios across different datasets, failing to prove the method's reliability in various practical environments.
Yes
No ethics review needed.
- According to the equation for Class Separability (CS), CS would be maximized when the intra-class variance is maximized, and the inter-class distance is minimized. Can the authors clarify if this is an error in the equation (requiring a negative sign) or if there's a reason for defining CS this way that isn't apparent to me?
- AttrZS in Table 1 is unclear? How are the attributes used for zero-shot classification? The authors can add a brief explanation of this for clarity.
- Tables 5 and 6 incorrectly mention the backbone as ViT-RN. I recommend the authors carefully review all table headers and labels for consistency and accuracy throughout the paper.
- The paper presents a neat incorporation of the vast knowledge of LLMs to improve downstream classification of vision language models. The use of descriptive and distinctive attributes of classes to leverage the language capability in vision-language joint pre-training is well validated through experimental results and theoretical backing.
- The proposed method achieves excellent improvements above existing visual reprogramming methods across a diverse set of visual classification datasets and different CLIP backbones.
This paper proposes an adaptation technique to improve pre-trained vision-language models for downstream classification tasks. The method uses visual reprogramming, a technique that alters the space of the input to the model – that is a learnt padding. This is done by leveraging the vast knowledge of LLMs to generate descriptive details, thus allowing to harness the potential of the joint embedding training of vision-language models, unlike prior works that only employ the class label used in the cross entropy objective. Distinctive attributes and descriptive attributes of classes together with a few clever training time tricks improves the downstream classification accuracy of VLMs.
- Is there a way to ascertain that the distinctive captions generated are actually distinct in the embedding space? While lines 78-79 claims this is the case in Figure 1, I disagree. The distinctive attributes seem to not necessarily be farther from the cluster center compared to descriptive attributes. The authors could compare of cosine distance of descriptive attributes and distinctive attributes with the class labels. Ideally, there should be a higher distance from other classes for distinctive attributes than descriptive attributes.
- It would be interesting to see an ablation on the use of kNN for the attributes. How does it compare to randomly sampling k attributes.  
- Line 167 wrong citation. I believe the intended citation is [1]. A kind suggestion for authors to double-check all their citations for accuracy

[1] Cai, Chengyi, et al. "Sample-specific Masks for Visual Reprogramming-based Prompting." arXiv preprint arXiv:2406.03150 (2024).
Yes
No ethics review needed.
See Weaknesses
1. The paper is well-written, which provides theoretical insights showing how AttrVR reduces intra-class variance and increases inter-class separation, enhancing the model’s discriminative power.
2. The motivation for this submission is easy to understand, which leverages descriptive and distinctive attributes instead of traditional label prompts, maximizing CLIP’s multimodal capabilities to improve classification accuracy.
3. AttrVR is tested across multiple datasets, with results indicating superior performance over existing methods in various visual classification tasks, especially in fine-grained categories.
The paper introduces Attribute-based Visual Reprogramming (AttrVR), a method for enhancing image classification by leveraging descriptive (DesAttrs) and distinctive attributes (DistAttrs) within vision-language models like CLIP. AttrVR replaces traditional label-based reprogramming with these attributes, refining classification through sample-specific, attribute-guided adjustments that improve intra-class consistency and inter-class separability. The approach demonstrates superior performance across multiple benchmark datasets, offering a more dynamic and attribute-focused alternative for repurposing pre-trained models in diverse downstream tasks.
In general, the motivation for this submission is easy to understand and insight is interesting. However, there are still several weaknesses, as follows:  
1. While the paper mentions using GPT-3.5 for generating descriptive and distinctive attributes (DesAttrs and DistAttrs), it lacks detailed reasoning on why GPT-3.5 was specifically chosen over other potential models. Further, the method relies heavily on the accuracy and quality of DesAttrs and DistAttrs generated by a language model. However, there is limited discussion on how inaccuracies or noise in these attributes might affect the model's performance.  
2. The effectiveness of the model appears sensitive to hyperparameters like λ (balance between DesAttrs and DistAttrs) and k (the number of nearest attributes). While the authors conduct a hyperparameter analysis, this could raise concerns regarding the robustness of the model in practical settings where optimal hyperparameters may vary widely across datasets.  
3. While standard contrastive learning ablation is provided, the study lacks a comparison between hard negatives and simple negatives. Additional experiments could assess performance when hard negatives are removed, highlighting whether they significantly enhance alignment and fine-grained recognition capability.  
4. Although the k-NN aggregation outperforms other methods like max, avg, and mean aggregation in attribute selection, the iterative querying process for the k-nearest attributes per sample per epoch raises concerns about computational efficiency and scalability, especially for large datasets. In addition, the sample-specific optimization approach using k-NN may cause the model to overfit to specific samples instead of capturing generalizable features.
@inproceedings{
anonymous2024attributebased,
title={Attribute-based Visual Reprogramming for Image Classification with {CLIP}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=j964C6y92q},
note={under review}
}
*Visual reprogramming* (VR) reuses pre-trained vision models for downstream image classification tasks by adding trainable noise patterns to inputs. When applied to vision-language models (e.g., CLIP), existing VR approaches follow the same pipeline used in vision models (e.g., ResNet, ViT), where ground-truth class labels are inserted into fixed text templates to guide the optimization of VR patterns. This label-based approach, however, overlooks the rich information and diverse attribute-guided textual representations that CLIP can exploit, which may lead to the misclassification of samples. In this paper, we propose ***Attr**ibute-based **V**isual **R**eprogramming* (AttrVR) for CLIP, utilizing ***des**criptive **attr**ibutes* (DesAttrs) and ***dist**inctive **attr**ibutes* (DistAttrs), which respectively represent common and unique feature descriptions for different classes. Besides, as images of the same class may reflect different attributes after VR, AttrVR iteratively refines patterns using the $k$-nearest DesAttrs and DistAttrs for each image sample, enabling more dynamic and sample-specific optimization. Theoretically, AttrVR is shown to reduce intra-class variance and increase inter-class separation. Empirically, it achieves superior performance in 12 downstream tasks for both ViT-based and ResNet-based CLIP. The success of AttrVR facilitates more effective integration of VR from unimodal vision models into vision-language models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Visual Reprogramming Model Reprogramming Vision-Language Models Image Classification
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4d6d1e5e1a624949a67a7a3be204d889fe3d09b7.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Attribute-based Visual Reprogramming for Image Classification with CLIP
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission