Refining CLIP's Spatial Awareness: A Visual-Centric Perspective
Self-distillation; CLIP; Open-vocabulary dense prediction
Yes
No ethics review needed.
Please see weaknesses.
Motivated by the observation that RLA methods suffer from notable loss in spatial awareness for CLIP ViTs, SCD is specifically designed to capture the spatial structure in a region. The widespread experiments show the superior performance of the proposed method across multiple open-vocabulary dense prediction benchmarks.
This paper aims to improve upon the Region-Language Alignment (RLA) approaches for Contrastive Language-Image Pre-training (CLIP) models. In order to not only promote the linguistic alignment but also preserve the spatial awareness, Spatial Correlation Distillation (SCD) is proposed to plug into the existing methods such as RegionCLIP and CLIPSelf. Refiner is also introduced to enhance the regional representation of teacher model. The experiments on the open-vocabulary dense prediction tasks demonstrate the effectiveness of the proposed method.
The motivation to design Refiner has been implied in CLIPSelf. When K = N, the proposed Refiner is almost the same as CLIPSelf, which reduces the technical novelty. Also, the ablation study of K is absent in this work.

The application of the proposed method is restricted. Spatial Correlation Distillation (SCD) is auxiliary and is to preserve the spatial awareness when another optimization is applied (e.g. RLA). Therefore, it seems that SCD cannot be applied independently since in this case the weights of teacher model and student model are always equal. Besides, R-SC-V is only learned from the teacher model that is optimized by Refiner (similar to CLIPSelf), so it cannot be further applied to CLIPSelf based approach.

In order to showcase the generalizability and scalability of the proposed method, the experiments with data scaling up are expected to be provided, which is missing in the current version.
Yes
No ethics review needed.
1. In the SC-RLA, how to ensure the correspondence between the RoI regions from teacher and student models? The student model would be trained, so the number, order and attribute of the RoI regions would become different with the frozen image encoder.

2. In the line 242 -258, would the contents of embeded sampled images {X_i} affect the SPATIAL AWARENESS? Could you provide some quantitative analysis like the cosine similarity between the {X_i} and X_t ?

3. Since the motivation and module is to improve spatial awareness (which can also be seen from the visualization), are there more segmentation related visualizations? Qualitative results using segmentation would be more convincing (e.g. finer edges)
1. Experiments. The experiments are sufficent and solied to prove the improvement of framework.

2. Significance. The framework is a convience, plug and play, and effecitive pipeline to improve the existing methods of the OV tasks.

3. Motivation. The motivation is clear. The Fig.1 and intro explain the loss of dense prediction ability of RLA process.
This paper proposes a framework called Spatial Correlation Distillation(SCD) to refine the spatial awareness of CLIP. To recall the visual perception of vanilla CLIP loss in Region-Language Alignment (RLA) training, it proposes a Spatial Correlation Distillation training process and a light-weight refiner to distill the visual relationship from the vanilla CLIP. SCD can improve the existing OV methods and achieve the new sota on the ov tasks.
1. Novelty. The frame work is composed with SCD and a refiner and the novelty of the SCD is limited. The SCD is a common distillation module to reserve the similarity between the RoI features of student and teacher models. 

2. Lack of analysis. This paper does not provide a deeper analysis of the experimental results. For example, why R-SC can effectively improve the classification accuracy in Tab.1? In terms of motivation and structure, this module improves spatial perception and does not have much gain in classified task with the annotated masks..

3. Typo. What is the Fig.2.o in the line 193?
Yes
No ethics review needed.
1. It might be better to use "language" or "text" instead of "linguistic"

2. How come methods specifically designed for CLIP dense prediction like CLIPSelf and RegionText work even worse than vanilla CLIP?
1. This pager reveals the potential problems of previous works, that "the RLA process projects dense visual embeddings into a text-oriented domain, making them incompatible with visual-centric objectives". To tackle this, the paper proposes to conduct Spatial-Correlation-guided Region-Language Alignment with Refiner to preserve spatial awareness. This design is novel and reasonable.

2. The performance improvement is significant compared with baseline methods.

3. The experiments and analysis are comprehensive.
This paper aims to enhance CLIP's spatial awareness. That is to say, increases the quality of dense features extracted by CLIP. It proposes the Spatial Correlation Distillation (SCD) framework, which preserves CLIP's inherent spatial structure and mitigates degradation for spatial awareness by Region-Languaeg Alignment. It also introduces a lightweight Refiner that extracts refined correlations directly from CLIP before feeding them into SCD, based on an intriguing finding that CLIP naturally captures high-quality dense features.
1. The paper writing is not so rigorous. Authors should give clear definitions of each term they are discussing. Like what is the definition of "spatial awareness", what it means by a better dense feature, what is "visual-centric", what is "intra-feature structural relationships", etc.

As an example, spatial awareness here is (probably) defined as performance for tasks like localization and recognition, which I think, is equivalent to increasing the quality of dense features extracted by CLIP, which means different parts in the image with different semantics should be extracted with the features that are distinguishable from each other.
We refine CLIP’s region-language alignment by enhancing its spatial awareness, improving performance from both visual-centric and vision-language perspectives.
@inproceedings{
anonymous2024refining,
title={Refining {CLIP}'s Spatial Awareness: A Visual-Centric Perspective},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=38No4B8sx6},
note={under review}
}
Contrastive Language-Image Pre-training (CLIP) excels in global alignment with language but exhibits limited sensitivity to spatial information, leading to strong performance in zero-shot classification tasks but underperformance in tasks requiring precise spatial understanding. Recent approaches have introduced Region-Language Alignment (RLA) to enhance CLIP's performance in dense multimodal tasks by aligning regional visual representations with corresponding text inputs. However, we find that CLIP ViTs fine-tuned with RLA suffer from notable loss in spatial awareness, which is crucial for dense prediction tasks. To address this, we propose the Spatial Correlation Distillation (SCD) framework, which preserves CLIP's inherent spatial structure and mitigates above degradation. To further enhance spatial correlations, we introduce a lightweight Refiner that extracts refined correlations directly from CLIP before feeding them into SCD, based on an intriguring finding that CLIP naturally capture high-quality dense features. Together, these components form a robust distillation framework that enables CLIP ViTs to integrate both visual-language and visual-centric improvements, achieving state-of-the-art results across various open-vocabulary dense prediction benchmarks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Self-distillation; CLIP; Open-vocabulary dense prediction
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a5a579b871219820f18d259dcedc27c48b62edeb.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Refining CLIP's Spatial Awareness: A Visual-Centric Perspective
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission