Semi-Supervised CLIP Training by Enforcing Semantic and Trapezoidal Consistency
Semi-supervised learning;Vision-language pre-training
Yes
None
No ethics review needed.
-- In SEMANTIC CONCEPTS MINING,  the linear classifier is initialized from the concept features of clip text encoder. Do the parameters of this module update during pre-training?

-- It seems this linear classifier is a close-set classifier, why not use the open-set method?

-- Does each concept have its own prompts-driven template [V ]1 [V ]2 [V ]3? Is the prompts-driven template [V ]1 [V ]2 [V ]3 shared for all concepts?

-- It is unclear whether the model is trained from scratch or initialized from pre-trained CLIP.

-- The sizes of datasets in the experiments are unclear.

-- Why do the proposed models suppress the fine-tuned models (CLIP (fine-tuned))? And what is the upper bound for performance?

-- Why is the lower base not used in Figure 1(b)?
-- SEMICLIP mines candidate semantic concepts from labeled data and learns to associate images with concepts, which may be a new way of using unlabeled images in vision-language pre-training.

-- This paper proposes the trapezoidal consistency to enhance the multi-modal alignment by exploiting the geometric structure of trapezoids in the representation space.

-- This paper uses prompts-driven templates and predicts concepts to construct surrogate captions for unlabeled images.
-- This paper proposes a new semi-supervised CLIP training method SEMICLIP that adapts CLIP to target tasks using only a small amount of image-text pairs.

-- This paper designs concept-level consistency and caption-level trapezoidal consistency for learning from unlabeled images to enhance visual representations and improve cross-modal alignment, respectively.

-- Extensive experiments demonstrate that the proposed method achieves state-of-the-art results in both zero-shot classification and image-text retrieval tasks.
-- The proposed method consists of two stages. The finetune stage heavily relies on the concept ming from the pretraining stage. The contribution of each stage is hard to evaluate.

-- The experiments are only conducted on task-specific datasets. What is their performance on common zero-shot classification and multimodal retrieval?
Yes
No ethics review needed.
Please refer to the weaknesses section to see the exact questions.
- The paper is written in a clear manner, making the concepts and methodologies easy to understand. This clarity enhances the overall accessibility of the work, allowing readers to grasp the key ideas without difficulty.
- Extensive experiments to evaluate the effectiveness of the proposed method. The experiments are conducted on 8 classification benchmarks and 6 retrieval benchmarks.
The paper presents SEMICLIP, a semi-supervised training method that enhances CLIP’s performance with limited number of image-text paired data. It utilizes a small number of labeled pairs along with a large set of unlabeled images by employing semantic concept mining to create pseudo-labels for the unlabeled data. The method introduces trapezoidal consistency regularization to maintain geometric relationships between image-text pairs, optimizing the model's alignment. Experimental results show that SEMICLIP bring improvements on zero-shot classification and image-text retrieval performance on various domains datasets.
- The proposed Semantic Concepts Mining method is not novel. Previous work [1] has already used CLIP as a concept labeler to construct pseudo labels for contrastive learning and image captioning. While [1] conducted experiments under the pretraining setting, this paper focuses on small domain datasets.
- Caption-level trapezoidal consistency builds incrementally on CyCLIP [2] in a semi-supervised setting. CyCLIP introduced cross-modal and in-modal consistency. While the paper does provide some comparisons between CyCLIP and SEMICLIP, it primarily emphasizes SEMICLIP focuses on unlabeled images and surrogate captions. However, this change may not be particularly novel.
- The experiments are primarily conducted on specific domain datasets. What about the performance on general benchmarks, such as the retrieval benchmarks Flicker30k and COCO, as well as classification on ImageNet?

[1] Huang, Runhui, et al. "Nlip: Noise-robust language-image pre-training." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 1. 2023.

[2] Goel, Shashank, et al. "Cyclip: Cyclic contrastive language-image pretraining." Advances in Neural Information Processing Systems 35 (2022): 6704-6719.
Yes
No ethics review needed.
1. Since the paper mainly focus on the CLIP downstream adaptation, I suggest authors change the title from "SEMI-SUPERVISED CLIP TRAINING" to "SEMI-SUPERVISED CLIP ADAPTING".

2. More explanation towards trapezoidal distance hypothesis. Why it is necessary to restrict the trapezoid to have equal-length legs and diagonals?
+ The paper proposes a new method to leverage unlabeled images alongside limited labeled data, enhancing CLIP's cross-modal alignment.

+ Authors design a caption-level trapezoidal consistency loss to appropriately aggregate mined concepts, which is new to me.
This paper presents a new semi-supervised training method for vision-language pre-training models like CLIP, called SemiCLIP. The method is designed to improve CLIP's adaptability to downstream tasks when there's limited image-text paired data. SemiCLIP uses a small amount of image-text pairs and a large volume of images without text descriptions to enhance cross-modal alignment. It introduces semantic concept mining to improve visual representations by matching images with relevant concepts from labeled data. The method also creates learnable surrogate captions for unlabeled images and optimizes a trapezoidal consistency to regulate the geometric structure of image-text pairs. The experiments show that SemiCLIP significantly improves CLIP's adaptability, increasing zero-shot classification accuracy by 1.72% - 6.58% and image-text retrieval performance by 2.32% - 3.23% on standard benchmarks.
- The method used to mine semantice concepts had been widely used in semi-supervised works thus is not quite novel to me.
- Why does the distance between I_i and Tˆ_j should be consistent with the distance between I_j and T_i? The reason for the consistency for diagonals constrains is not clear to me.
Yes
No ethics review needed.
For Table 1~4, we recommend  the authors to show the upper bound (supervised learning on all data) to reveal the gap between semi-supervised learning and fully-supervised learning.
1. The correlation between images and concepts is intuitive, which can be effectively used to optimize CLIP in semi-supervised learning scenarios. 

2. The experimental results are promising. 

3. The writing is polished and easy to understand.
In this manuscript, the authors focus on efficiently mitigating the domain gap in semi-supervised learning of CLIP on downstream tasks. Specifically, the authors first leverage labeled images to extract the concept list (used as pseudo labels) of the target dataset, and employ CLIP loss and classification loss on pseudo labels for supervised learning. After initialization on labeled set, the authors propose concept-level semantic consistency and caption-level trapezoidal consistency to optimize CLIP on the whole dataset. Experimental results show that the proposed method can effectively
1. My main concern is the analysis of prompting strategy of \hat{T}_{j}. The position of prompts, the initialization of prompts, or even using fixed prompts rather than learnable prompts can be analyzed. The authors can conduct empirical analysis to investigate which prompting strategy is better. For example, comparing performance with prompts at different positions (e.g. beginning, middle (similar to current configuration), and end of all concepts), different initialization (zero init or normal init for prompt), and fixed vs learnable prompts. All these factors may affect the quality of pseudo captions for unlabeled images. 

2. The quality of pseudo labels (i.e., mined semantic concepts) may be critical for semi-supervised learning. The authors could conduct some analysis (e.g., mining concepts with different granularities) to investigate the effect from quality of pseudo labels. For example, "your current configuration" vs "mined concepts but unified into some common objects or attributes" vs "oracle fine-grained concepts (e.g., mining concepts from the whole original dataset)".

3. The authors can also conduct ablation study regarding different percentages of labeled data (e.g. 1%, 5%, 10%, 25%, 50% labeled data) to evaluate the robustness of proposed training framework. Intuitively, a good semi-supervised learning framework may still robust against low percentage labeled data. Nevertheless, in low percentage labeled data settings, the lack of ground-truth caption may restrict the quality of mined concepts.
Yes
No ethics review needed.
- Effect of linear classifier: Semantic concept classifier could be performed by CLIP text encoder, akin to zero-shot classification. Is there a specific reason for using linear classifier head instead?
- The paper is well written and easy to follow.
- Experiments are comprehensive.
- Results show consistent improvements over baselines across diverse domains.
The paper proposes a semi-supervised CLIP training method, termed SemiCLIP, which leverages a large amount of unlabeled images when only limited image-text pairs are available. SemiCLIP consists of two training stages: supervised pre-training and semi-supervised fine-tuning. In the supervised pre-training stage, the paper introduces a concept classifier along with the standard contrastive loss to enhance image-text alignment using labeled data. The semi-supervised fine-tuning stage includes two key components: concept-level semantic consistency, which ensures that the model maintains consistency in understanding visual concepts, and caption-level trapezoidal consistency, designed to improve cross-modal alignment by refining the geometric structure between image-text pairs. Experimental results show promising improvements over existing baselines across diverse domains.
The motivation behind caption-level trapezoidal consistency (Section 3.3.2) is not sufficiently explained. 

- The surrogate caption is generated by concatenation of templated keywords. Even though the templates are learnable, the distribution of generated captions is likely to differ significantly from the labeled captions. It would be helpful if the underlying motivation for this choice of surrogate caption generation process is explained in detail. (One simple alternative might be generating surrogate captions based on keywords using a large language model.)
- The main objective of trapezoidal consistency loss is to effectively utilize less-than-ideal surrogate captions due to their inexact nature. However, it is unclear why diagonal and legs regularization is beneficial, whereas the direct usage of contrastive loss is not, given the same inexact caption. It would be useful to explain why trapezoidal consistency is better suited to handle noisy captions. (A simpler alternative could involve using a soft-label in the contrastive loss, specifically for surrogate captions.)

I look forward to authors’ clarification and am willing to increase the score based on their clarification.
We propose a method to leverage limit image-text pairs and abundant unlabeled images to improve the vision-language pre-training
@inproceedings{
anonymous2024semisupervised,
title={Semi-Supervised {CLIP} Training by Enforcing Semantic and Trapezoidal Consistency},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=97D725GJtQ},
note={under review}
}
Vision-language pre-training models, such as CLIP, have demonstrated strong capability in rapidly adapting to downstream tasks through fine-tuning, and have been widely applied across various tasks. However, when the downstream tasks are constrained by limited image-text paired data, CLIP struggles to effectively address the domain gap between the pre-training and the target tasks. To address this limitation, we propose a novel semi-supervised CLIP training method coined SemiCLIP that leverages a small amount of image-text pairs alongside a large volume of images without text descriptions to enhance CLIP’s cross-modal alignment. To effectively utilize unlabeled images, we introduce semantic concept mining to improve task-specific visual representations by matching images with relevant concepts mined from labeled data. Leveraging matched semantic concepts, we construct learnable surrogate captions for unlabeled images and optimize a trapezoidal consistency to regulate the geometric structure of image-text pairs in the representation space. Experimental results demonstrate that our approach significantly improves the adaptability of CLIP in target tasks with limited labeled data, achieving gains ranging from 1.72\% -- 6.58\% for zero-shot classification accuracy and 2.32\% -- 3.23\% for image-text retrieval performance on standard benchmarks. The source code is provided in the supplementary material.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Semi-supervised learning Vision-language pre-training
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/eb5c29d12f55516350028d73504f5b1801050d61.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/7cbb74720e55b771af1070ad77b6700e9cfced9d.zip
Semi-Supervised CLIP Training by Enforcing Semantic and Trapezoidal Consistency
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission