OC-CLIP : Object-centric binding in Contrastive Language-Image Pretraining
object-centric representations;object binding;CLIP;contrastive learning;compositional image-to-text retrieval
Yes
No ethics review needed.
see weaknesses
(1) The introduction of the binding module is well-motivated and technically sound.
(2) The structured similarity score, composed of an object scoring function and a relationship scoring function, seems a reasonable method for modeling the scene graph and the visual slots relationships.
The paper introduces OC-CLIP, a novel approach aimed at enhancing the compositional understanding capabilities of CLIP-like models. 
The key innovation is the integration of a binding module that connects a scene graph derived from text descriptions with a slot-structured image representation. 
The proposed structured similarity score between the two modalities seems to capture interactions between objects and their contextual relationships more effectively.
(1) The binding module and structured similarity score introduce additional complexity to the CLIP model. The authors should provide an analysis of the trade-offs of computational overhead and performance gains.

(2) The authors evaluate OC-CLIP on several standard benchmarks of compositional understanding and spatial relationship understanding. However, as a fundamental multi-modal representation model, additional downstream experiments on tasks such as zero-shot image classification, video action recognition, geo-localization, text-to-image retrieval etc. should strengthen the paper.
Yes
No ethics review needed.
-Can some MLLMs (e.g., internVL, llava) be regarded as text parser? Can MLLMs (use text and image) parse better relation and objects than LLM from text

-Tagalign[2] also use a LLM-based as text parser. Suggest comparing with this paper.

[2] Tagalign: Improving vision-language alignment with multi-tag classification.
-The proposed method is simple and intuitive.

-The experimental performance on several datasets demonstrate large improvements in terms of accuracy.

-Interesting idea to generate object/relation tags by utilizing LLM to help scene graph undetstanding
This paper roposes a approach that diverges from commonly used strategies relying on the design of hard-negative augmentations. This work instead focuses on integrating sufficient inductive biases into pre-trained CLIP-like models to improve their compositional understanding without using additional data annotations.
-Using LLM for parsing and training with object + relation classification loss, are a bit too straightforward to be claimed as a contribution

-One possible way to make LLM contributions more reasonable is to compare multiple different LLM/cues and analyze the results/performance.

-Extract the computational cost of labels with LLM. LLM is known for its large scale and requires a significant amount of computing resources. I would like to know how much computation is required to process the visual genomic data used in the paper.
Yes
No ethics review needed.
please see weaknesses, lots of questions there
* compositional reasoning is still an interesting problem, yet as the field has mostly shifted to decoder LMMs it is unclear how long the encoder-only models can still be compared just to themselves. Decoder LMMs have much higher compositional reasoning performance.
* designing and training explicit alignment modules for objects and relations is interesting
* performance improvements reported
The paper proposes augmenting CLIP contrastive finetuning with a "binding module" supervised by objects and relations data extracted from high-quality human-annotated data via an LLM. The proposed architecture of the binding module explicitly matches objects and relations to learned attention slots applied to visual encoder output. Noise register tokens are also used in the matching. Experiments were conducted on either synthetic data (train and test generated by the authors) or training on high-quality real VG / COCO / GQA annotated data. Testing of the real data-trained models is primarily done on ARO, SugarCrepe, and GQA (which are derived from the same datasets used for training). Additionally, WinoGround performance improvements are reported. Seems some important baselines are missing.
1. synthetic data experiments seem limited and less convincing. 
2. in the real data experiments, the model is finetuned on high-quality human annotated data from the same datasets out of which the evaluation benchmarks (ARO, SugarCrepe, COCO) are constructed. It seems that the method performs supervised training as opposed to hard-negative based techniques mentioned which train on unconstrained internet data (LAION, CC, etc)
3. seems that important baselines are missing as well as additional compositional reasoning benchmarks to compare to those baselines.
e.g. https://paperswithcode.com/sota/visual-reasoning-on-winoground - for many higher reported results on WinoGround; more hard negative mining methods (DAC = https://arxiv.org/pdf/2305.19595 being one example); etc
4. no zero-shot evaluation of the fine-tuned model is done - the model is fine-tuned for 100 epochs on supervised data explicitly focusing objects and relations there - one might expect significant degradation on zero-shot benchmarks like ELEVATER used in DAC above. It is important to show the proposed technique does not solve some compositional reasoning issues on expense of the general applicability of the model (zero-shot performance)
5. Ablation of the proposed components and losses is completely missing, but should be a significant part of the paper informing the reader on the relative contribution of components, insights related to them, and so on.
6. More minor:
a. the "rel" loss is very similar to the hard negative losses employed by others, a bit counter the (a bit unjustified) reasoning of the authors that hard negative based approaches are detrimental (if anything they require much less supervision than the proposed approach) - also rel loss is not ablated (well nothing is ablated except the synthetic dataset)
b. reliance on LLM parsing is not ablated, what if it introduces noise how to deal with it? Parsing real captions (eg if one would try to apply this on unconstrained data like LAION or CC) seems a much harder task compared to negative augmentation used in other works (contrary to the reasoning of the authors in the intro)
c. in eq. 3 - what prevents the collapse of f_s and f_o to just extracting the "r" part from the concat? not ablated / analyzed...
Yes
No ethics review needed.
Experimental Comparisons:
1. Why not include comparisons with mainstream datasets like CoCo, Flickr (ARO dataset), VL-Checklist, and sDCI?

2. Why are mainstream methods such as DAC, sDCI, and SVLC not included in your experiments?

Ablation Studies:
3. Provide an analysis of the ablation of the loss function.

4. Include an ablation study on the selection of object and relationship extraction models.

Scene Graph Accuracy:
5. How do you address the issue of misalignment in compositional reasoning, as seen in models like Llama3?

6. What methods do you propose to ensure the accuracy of the scene graph?

CLIP Fine-Tuning:
7. Analyze whether CLIP retains its original image understanding capabilities after fine-tuning with the scene graph.

Loss Function Hyperparameters:
8. Provide experimental or theoretical analysis to justify their inclusion and effectiveness.
The idea presented in this paper of using scene graphs to help VLMs improve their compositional reasoning capabilities is novel and interesting, and it appears to be an effective approach.

This paper proposes a new structured similarity score based on contrastive loss, consisting of an object scoring function and a relationship scoring function, which seems capable of effectively evaluating the relationships between nodes in the scene graph.

The organization of this paper is clear, the problem definition is explicit.

The proposed method has achieved effective improvements on certain datasets.
Most existing vision and language models (VLMs) face challenges when dealing with compositional reasoning in images and texts. To enhance the compositional reasoning capabilities of VLMs, previous research primarily refined these models through the construction of specialized datasets, such as introducing hard negative examples via textual modifications. This paper presents an innovative approach aimed at overcoming the limitations of existing VLMs when handling complex compositional scenes involving multiple objects and their spatial relationships. The work introduces a novel framework focused on integrating sufficient inductive biases into pre-trained CLIP-like models, providing additional guidance through the use of scene graphs rather than relying on traditional hard negative augmentation strategies. Experimental results indicate that such a design optimizes the performance of CLIP models across multiple benchmark datasets.
This paper lacks convincing evidence in several key areas, specifically:

The experimental comparisons are clearly insufficient. Mainstream datasets in this field, such as the CoCo and Flickr datasets from the ARO dataset, the VL-Checklist dataset, and the sDCI dataset, have not been included in the comparative experiments. Additionally, mainstream methods such as DAC, sDCI, and SVLC have not been compared, making the experimental results significantly unconvincing.

VL-Checklist: "Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations." arXiv preprint arXiv:2207.00221 (2022).
sDCI: "A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
DAC: "Dense and aligned captions (dac) promote compositional reasoning in vl models." Advances in Neural Information Processing Systems 36 (2024).
SVLC: "Teaching structured vision & language concepts to vision & language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Ablation studies are missing. There is no analysis of the ablation of the loss function or the selection of object and relationship extraction models.

The accuracy of the scene graph heavily relies on the relationship extraction model. However, models like Llama3 also face issues with misalignment in compositional reasoning. This paper does not provide any method to ensure the accuracy of the scene graph.

There is no analysis of whether CLIP retains its original image understanding capabilities after fine-tuning with the scene graph.

The proposed loss function in the paper includes multiple hyperparameters, but there is no experimental or theoretical analysis provided.
We analyze and propose an object-centric solution to the binding problem in CLIP
@inproceedings{
anonymous2024occlip,
title={{OC}-{CLIP} : Object-centric binding in Contrastive Language-Image Pretraining},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=a84AD957m9},
note={under review}
}
Recent advancements in vision-language models (VLMs) have been driven by contrastive models like CLIP which learn to associate visual information with their corresponding text descriptions. However, these models have limitations in understanding complex compositional scenes involving multiple objects and their spatial relationships. To address these challenges, we propose a novel approach that diverges from traditional data-centric methods of enhancing model performance with hard negatives examples. Our work instead focuses on integrating sufficient inductive biases into pre-trained CLIP-like models to improve their compositional understanding without using additional data annotations. We introduce a binding module that connects a scene graph of the text with an induced graph-like representation of the image, facilitating a structured similarity assessment. We also leverage relationships as text-conditioned visual constraints, thereby capturing the intricate interactions between objects and their contextual relationships more effectively. Our resulting model (OC-CLIP) not only enhances the performance of CLIP in multi-object compositional understanding but also paves the way for more accurate and efficient image-text matching in complex scenes.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
object-centric representations object binding CLIP contrastive learning compositional image-to-text retrieval
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5694930661f489c7cafaf99338edb763bfaee4f9.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
OC-CLIP : Object-centric binding in Contrastive Language-Image Pretraining
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission