A Simple Framework for Open-Vocabulary Zero-Shot Segmentation
Vision-language models;Zero-shot segmentation
Yes
No ethics review needed.
What is the motivation of the proposed method?
What is the reason of the good performance? I think it is because the large training dataset.
The experimental performance is good.
This work proposes a simple framework for open-vocabulary Zero-Shot Segmentation with two key principles of leveraging frozen
vision-only models and exploiting the discrete nature of text and linguistic knowledge.
1. The novelty and contribution is limited. First, I prefer simple method. I believe that the simple method is more valuable for applications and research. However, simple method requires more deep analysis and insights. Unfortunately, this work only presents a simple method without any insights. The method is designed without motivation or explanation. This paper looks like a experiment report, rather than a research paper. For a good research paper, the authors need to tell new insights, rather than just propose a model and conduct some experiments.
2. The comparison is unfair. The proposed method is trained on COCO Captions (Lin et al., 2014; Chen et al., 2015) and LAION-400M (Schuhmann et al., 2021) and the image encoder is pretrained on LVD-142M dataset, as shown in L764-L774. However, other methods are not trained on these large-scale dataset. Even trained on these large datasets, the performance improvement is limited and the proposed method even performs worse than previous methods on Pascal VOC dataset. For example, the proposed method SimZSS trained on LAION-400M achieves 48.6 on Pascal VOC, which is much lower than CLIP-DINOiser (Wysoczanska et al., 2023), CLIP-DIY (Wysoczanska et al., 2024), OVSegmentor (Xu et al., 2023a), OVDiff (Karazija et al., 2023), CLIPpy (Ranasinghe et al., 2023) and TCL (Cha et al., 2023).

Overall, this work propose a simple method without motivation and experimental analysis for the working mechanism. The good performance is mainly derived from its large training dataset.
Yes
No ethics review needed.
1. Since the visual encoder is frozen, the quality of the segmentation masks heavily depends on the encoder's ability to recognize spatial positions. Would the performance improve if the visual encoder were fully or partially fine-tuned?
2. In an extreme scenario, how significantly would the test performance be affected if the class names in the training set and test set are entirely different?
1. The paper is easy to understand and follow.
2. The method is straightforward, easy to implement, and can be readily adapted to various backbones and training datasets.
3. The method can be trained without the supervision of semantic masks, reducing the burden of annotations.
4. The motivation for proposing a concept-level objective is clear and more suitable compared to a contrastive objective in scenarios where concepts encode individual objects that are likely to occur multiple times within a batch.
The paper introduces SimZSS, a straightforward yet effective framework for zero-shot semantic segmentation. This framework aims to align vision encoders, which exhibit spatial awareness, with textual descriptions. SimZSS leverages text concept representations to extract corresponding visual representations based on similarity metrics. It ensures cross-modality consistency by employing different objectives at the sample and concept levels. Notably, the method requires minimal hyperparameters and does not rely on the supervision of semantic masks. Additionally, SimZSS can be easily adapted to various backbones and datasets. Overall, the method achieves state-of-the-art results across standard zero-shot segmentation benchmarks.
1. It is unclear how the final segmentation masks are generated during inference. Is there a similarity threshold used to determine the class names to which visual tokens belong? If so, how does the performance vary with different threshold settings?
2. There is a lack of analysis explaining why SimZSS outperforms other zero-shot semantic segmentation methods.
Yes
No ethics review needed.
- Given that the predefined concept bank seems to impact performance, how does the model perform on datasets with a much broader set of concepts, such as OpenImagesV7, which covers over 5000 classes? Has the model been evaluated on classes outside of the concept bank, and if so, what are the results?

- Have you evaluated the recognition capabilities of the trained model on other classification datasets beyond ImageNet? Including evaluations on a wider range of datasets could provide deeper insights into how the method affects recognition performance.

-  How does your model compare with training-free approaches?

- Could you discuss the potential trade-off between improved localization and any degradation in recognition performance? Understanding this trade-off would help assess the overall impact of your method on different tasks.
- The paper is well-written and easy to follow. The authors provide clear explanations of their methodology. I would like to highlight the quality of the figures and tables in a visually pleasing way that enhances the understanding of the content.

- While the problem of localization in vision-language models is not novel, the proposed approach offers a novel perspective. By freezing the vision backbone and only training the text encoder, the authors leverage pretrained self-supervised models effectively, resulting in efficient training.

- The method demonstrates state-of-the-art performance on multiple zero-shot segmentation benchmarks.
The paper introduces SimZSS, a simple framework that enables open-vocabulary zero-shot segmentation by leveraging pretrained vision-only models within a vision-language contrastive learning paradigm. Addressing the challenge of poor localization in dense tasks, SimZSS decouples visual representation learning from cross-modality alignment by utilizing frozen, spatially aware vision models like DINOv2 and focusing on training only the text encoder. The framework identifies local concepts within captions using linguistic cues and aligns them with corresponding visual concepts in images through similarity-based pooling.
1. It is surprising that the main paper lacks essential details about the training data and the pretrained models used. Given that the paper is only 9 pages (below the 10-page limit), including this information in the main text is necessary.

2. 
(a) One major limitation is the use of a predefined concept bamk. The authors claim that it does not impact the breadth of the concept the model can localize(sec. 4.4). However, in Table 4, removing PascalVOC classes from the concept bank decreases the performance. Therefore, it seems that the breath of the concept bank is a crucial component. Also, the authors could evaluate on datasets with much diverse set of concepts e.g. OpenImagesV7 (which covers more than 5000 classes), this way the breadth of the vocabulary learned by the model can effectively be assessed. 

(b) Moreover, to identify the recognition capabilities of the trained model, I would expect the authors to evaluate the performance of more classification datasets than just ImageNet. More specifically, it is common practice I think to evaluate on the set of 38 classification datasets (https://github.com/LAION-AI/CLIP_benchmark). Also, table 3 should include the performance of at least the OpenCLIP-ViT/16 trained on LAION-400M as the training dataset is the same, whose zero-shot accuracy on ImageNet is 67.05 vs 64.1 for your ViT-B/14 which indicates that your method hurts the overall recognition capability of the final model. This tradeoff between localization and recognition should be discussed in the paper.

(c)  Since the concept bank is derived from the union of class names from the segmentation datasets used for evaluation, the framework may be biased towards these classes. This design choice limits the model's ability to generalize to a truly open vocabulary, as it may not effectively localize concepts outside the predefined bank.
 
3. The paper misses important related work, particularly training-free approaches that tackle poor localization in VLMs from a different angle. For instance, methods like [a][b][c][d] achieve competitive performance without requiring additional training and can detect concepts not covered by a predefined concept bank. Including comparisons with such methods would provide a more comprehensive evaluation.

References:
[a] Lan, M., Chen, C., Ke, Y., Wang, X., Feng, L. and Zhang, W., 2024. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. arXiv preprint arXiv:2408.04883.

[b] Li, Y., Wang, H., Duan, Y. and Li, X., 2023. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.05653.

[c] Bousselham, W., Petersen, F., Ferrari, V. and Kuehne, H., 2024. Grounding everything: Emerging localization properties in vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3828-3837).

[d] Zhou, C., Loy, C.C. and Dai, B., 2022, October. Extract free dense labels from clip. In European Conference on Computer Vision (pp. 696-712). Cham: Springer Nature Switzerland.

[e] Benenson, R. and Ferrari, V., 2022. From colouring-in to pointillism: revisiting semantic segmentation supervision. arXiv preprint arXiv:2210.14142.
Yes
No ethics review needed.
- What are the current limitations of the concept bank used? Would a larger bank lead to better generalization? What are the complexities in scaling the concept bank to, e.g., WordNet or other taxonomies?
- As mining concepts from the captions provide promising results, I wonder how well the approach would work on more noisy captions, e.g., from web-scale datasets. How sensitive is SimZSS to such noise? Are there any techniques that could be employed to improve the robustness of the generation of the concept bank?
- The authors propose a simple yet effective approach for the task that exploits pre-trained vision models.
- The method is both data- and compute-efficient, as it does not require long training periods and needs captioned images for training.
- The method achieves state-of-the-art results on seven benchmark datasets.
- The method takes inspiration from LiT and extends it for the segmentation task while maintaining its classification capabilities.
The work proposed SimZSS, a simple and efficient framework for open-vocabulary zero-shot semantic segmentation. The work builds on top of a frozen pre-trained vision-only model and aligns a text encoder to achieve cross-modality concept-level alignment. Despite its simplicity, SimZSS achieves state-of-the-art results on seven out of eight benchmarks.
- The work relies on a bank of pre-defined concepts to pre-process the captions and extract concepts to segment, which may represent a limitation in some situations. For example, the concept bank may not cover the diversity of potential objects in the image.
- While performance is competitive when using ViT-B, scaling the model backbone to a ViT-L model does not improve performance.
@inproceedings{
anonymous2024a,
title={A Simple Framework for Open-Vocabulary Zero-Shot Segmentation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=QzPKSUUcud},
note={under review}
}
Zero-shot classification capabilities naturally arise in models trained within a vision-language contrastive framework. Despite their classification prowess, these models struggle in dense tasks like zero-shot open-vocabulary segmentation. This deficiency is often attributed to the absence of localization cues in captions and the intertwined nature of the learning process, which encompasses both image/text representation learning and cross-modality alignment. To tackle these issues, we propose SimZSS, a $\textbf{Sim}$ple framework for open-vocabulary $\textbf{Z}$ero-$\textbf{S}$hot $\textbf{S}$egmentation. The method is founded on two key principles: $\textit{i)}$ leveraging frozen vision-only models that exhibit spatial awareness while exclusively aligning the text encoder and $\textit{ii)}$ exploiting the discrete nature of text and linguistic knowledge to pinpoint local concepts within captions. By capitalizing on the quality of the visual representations, our method requires only image-caption pair datasets and adapts to both small curated and large-scale noisy datasets. When trained on COCO Captions across 8 GPUs, SimZSS achieves state-of-the-art results on 7 out of 8 benchmark datasets in less than 15 minutes. The code and pretrained models will be publicly available upon acceptance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Vision-language models Zero-shot segmentation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d0e619c3c8a5a9f04d5145b635ecd81000922315.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
A Simple Framework for Open-Vocabulary Zero-Shot Segmentation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission