Unclipping CLIP's Wings: Avoiding Robustness Pitfalls in Multimodal Image Classification
robustness;CLIP;spurious correlations;contrastive learning;multimodality
Yes
No ethics review needed.
1. In the 4th row of Table 1, OpenCLIP shows high average performance and spurious prediction on CelebA, yet the Worst Group Accuracy is very low. If this is due to dataset bias, how can the high performance in spurious prediction be explained?
This paper demonstrates, through performance comparisons, feature visualizations, and dataset statistics, that the choice of prompts has a significant impact on zero-shot performance. The authors also propose a BLIP-CLIP architecture that enhances CLIP’s zero-shot capability without requiring fine-tuning.
This paper demonstrates through experiments that factors such as dataset bias can introduce spurious correlations in pre-trained models. To mitigate these spurious correlations, the paper proposes using captions generated by BLIP as class prompts. Experimental results show that using BLIP-generated class prompts can enhance CLIP’s robustness to spurious correlations without requiring fine-tuning.
1. First, the author should adjust the layout of the tables in the paper. Some tables exceed the page width. (Tables 2, 3, 4, and 5).

2. This paper demonstrates in multiple ways that CLIP is sensitive to prompts. Previous work, such as MaPLE [1], had already presented similar insights.


3. Although the method in this paper requires no training, it involves comparing the similarity between each test sample and the captions generated for all training samples during the inference stage. On a large-scale dataset, wouldn’t this approach introduce a substantial additional time cost? It would be helpful if the authors could provide a comparison of the time cost between this method and direct inference with CLIP. Have you considered optimizing the computational burden on large datasets?

4. The experiments in this paper seem to focus more on the classification of spurious correlations. For the experiments on ImageNet, only 13 different cat categories were selected from the 1,000 classes to classify Pantherinae and Felinae. Since the method proposed in this paper does not require retraining, why didn’t conduct a performance evaluation directly on the entire ImageNet dataset? FD-Align [2] also pointed out that fine-tuning directly on ImageNet can impact model generalization due to spurious correlations, which is consistent with the distribution shift caused by OOD text as proposed in this paper.


[1] MaPLe: Multi-modal Prompt Learning

[2] FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning
Yes
No ethics review needed.
As I understand Section 5, all samples from the downstream training set are used for the knn algorithm. How strong is the effect on the results if a smaller subset is used instead?

For the experiments on ImageNet-1K, the training set is used for evaluation. Was the training set further split into train and test data or were the evaluated images also used to create the text-embeddings of the BLIP captions?
The sensitivity of CLIP zero-shot classification to the choice of class prompts is a known problem. Improving the performance by choosing more suitable class prompts is computationally very efficient and thus an interesting alternative to methods based on fine-tuning. The lacking coverage of used class prompts in the original training data of the CLIP model seems to be a reasonable explanation for bad performance on downstream classification tasks and is to some extent supported by the experimental evidence. Further, the proposed method is computationally efficient and does not require spurious feature annotations.
The paper investigates the robustness of CLIP zero-shot classification to spurious correlations and identifies out-of-distribution class prompts as possible cause. It proposes a mitigation by generating proxy class prompts using an image-to-text model on the downstream training data and performing k-nearest neighbours on the resulting embeddings.
- The proposed method improves over zero-shot classification but performs significantly worse compared to other methods using training data of the downstream task.
- The initial experiments show that the CLIP models perform much worse on Waterbirds compared to CelebA leading to the conclusion that lacking coverage of the prompts “waterbird” and “landbird” in the pertaining data are the cause for this discrepancy. However, for Waterbirds, the spurious labels “water” and “land” are contained in these class prompts which might cause a larger CLIP similarity between the corresponding text embeddings and the image embeddings containing “water”/“land” backgrounds. This is not the case for CelebA (class attributes: blond/not blond, spurious attributes: male/female). This simple explanation would cause a similar discrepancy but is not considered in the paper. Thus, the experiments on Waterbirds do not seem suitable to properly support the claims regarding lacking coverage.

Minor: Table 2, 3, 4, and 5 are too wide and go over the margin. The plots in Figure 1 are too small, axis labels and legend are not readable. Section 5 mentions the use of Llama-2 in Fig. 2 which contradicts its caption and description in section 4.
Yes
No ethics review needed.
How many captions are used for each image? Were there other models than BLIP testesd for captioning?
* S1. Investigating the biases of CLIP models is a good direction.

* S2. Investigating the frequency of class labels in LAION dataset and looking at the marginal log-probabilities given by MetaCLIP models is interesting and suggests that the class names were not frequent in the training datasets.
The paper investigates the performance of CLIP models on datasets containing spurious correlations. First, the paper shows that multiple CLIP models have poor zero-shot performance when predicting the target class or the spurious attribute in the considered datasets (Waterbirds, CelebA). A cause of this is that the class/attribute labels are unusual, out-of-distribution of the training captions. The proposed solution is to not rely on the target name, but instead use an image captioning model (BLIP) to get a caption for each image and then embed it with the CLIP text module. A testing image will be encoded either by: a) the CLIP image encoder or b) captioning the image with BLIP and then using the CLIP text encoder. This embedding will be used in a k-NN fashion together with the previous support set.
* W1. The message of the paper is not that novel. The community knows that large models are very sensitive to prompts, it is not new or surprising that CLIP performance varies for class templates. 

* W2. Waterbirds and CelebA are small datasets and having most of the analysis done on them is a big limitation. Fundamental research questions could still be studied on these datasets, but investigating the robustness prompts / templates and proposing complex solutions involving multiple large models to solve these datasets is hard to justify. It’s hard to say if these investigations can lead to insights and benefits for more realistic settings.

* W3. There are simple baselines missing. a) Linear evaluation on CLIP image features b) k-nn directly on CLIP image features c)  same for BLIP image features. These are necessary to see if the captioning is important for robustness or if simply using the good features of CLIP / BLIP is enough.

* W4. It is common to use image captions and LLMs to solve different tasks (see Blip2 or LLaVA). Using image captions from large models like BLIP to help in CLIP classification via k-NN seems like a step behind such approaches, without giving any additional insights.

* W5. Papers like [A] also investigate the non-robustness of CLIP class templates and suggest instead to use detailed descriptions generated by LLM as templates. It will be relevant to discuss the similarities to the proposed solution.

[A] Menon, Sachit, and Carl Vondrick. "Visual classification via description from large language models."
Yes
No ethics review needed.
Presentation tips:

- "WGA can be arbitrarily improved": In the abstract, this sentence should be rephrased. The current sentence means that it could be improved up to 100%, which is not the case I believe.

- Tables 3/4/5 do not fit the width of the page. The tables might also be nicer with the booktabs package (with \toprule, \midrule, and \bottomrule) and possibly fewer vertical lines (?).
- Clear writing, thorough presentation of the problem, methods, and results. The paper covers particularly well the context of the problem (spurious correlations, shortcut learning) and the current state of knowledge about how CLIP models behave and are currently being used.

- Scientific evaluation of a well known problem with CLIP models 

- Simple mitigation method that appears effective, and also seem practically relevant to many uses since it allows using CLIP as a classifier without manually creating text prompts/class descriptions.

- Evaluation of several CLIP variants (OpenCLIP, MetaCLIP)

- Preliminary demonstration of usefulness on data without spurious correlations (IN-1k)
The paper studies the robustness of CLIP models on datasets with spurious correlations (Waterbirds, CelebA).

The authors find that the zero-shot classification performance is highly dependent on the choice of class prompts

They then present a method to improve this, using class prompts generated by an image-to-text model. They show that this works better because the prompts are "ID" w.r.t. CLIP's pretraining data.
I do not see major flaws in this paper. It has interesting findings that (to my knowledge) are novel, and a simple method that seems practically useful.

The findings are not earth-shattering but I think they will be of interest to the community working on spurious correlations and on the robustness of CLIP models.
@inproceedings{
anonymous2024unclipping,
title={Unclipping {CLIP}'s Wings: Avoiding Robustness Pitfalls in Multimodal Image Classification},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=DPp5GSohht},
note={under review}
}
Despite being pretrained on large-scale data, multimodal models such as CLIP can still learn spurious correlations. However, CLIP does not seem to learn the same spurious correlations as standard vision models, performing worse on some benchmark datasets (Waterbirds) yet better on others (CelebA). We investigate this discrepancy and find that CLIP's robustness on these datasets is highly sensitive to the choice of class prompts. Worst-group accuracy can be arbitrarily improved or worsened by making minute, single-word changes to prompts. We further provide evidence that the root cause of this phenomenon is \textit{coverage} --- using class prompts that are out-of-distribution with respect to pretraining can worsen spurious correlations. Motivated by these findings, we propose using class prompts that are generated from a public image-to-text model, such as BLIP. We show that performing $k$-nearest neighbors on these prompt embeddings improve downstream robustness without needing to fine-tune CLIP.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
robustness CLIP spurious correlations contrastive learning multimodality
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cf0f305b8cb5de089ae4e47ab0886cb1791e2e00.pdf
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Unclipping CLIP's Wings: Avoiding Robustness Pitfalls in Multimodal Image Classification
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission