A Clustering Baseline for Object-Centric Representations
object-centric representations;self-supervised learning
Yes
No ethics review needed.
I recommend the authors delving deeper into the studied problem to reveal some insightful observations.
1. The presentation is good. The story is very clear to understand.
2. The method is simple, and simplicity means easy to be applied to downstream fields.
This paper presents a simple method that clusters feature from the representation from existing backbones, i.e., K-means clustering based on the extracted feature. The K is a fixed value. Some experiments are conducted to claim that this simple clustering method performs better than many other methods requiring additional training.
1. Claiming a kind of generally learned representation is better needs numerous diverse experiments to fully evaluate the method from different perspectives.  The experiments, which include only single-label classification and unsupervised segmentation, are insufficient. I think this work just selects a few experimental settings supporting the desired conclusion.
2. Clustering based on the representation produced by existing backbones is a commonly used baseline in fields like few-shot and unspervised learning. There is no new information in this paper.
Yes
No ethics review needed.
Please refer to the weaknesses.
- The motivation is clear, and the method is straightforward to understand.
- Extensive experiments and thorough analysis are conducted to explore the characteristics of embeddings obtained by applying K-Means on DINO V2 features, along with comparisons to those captured by learning-based methods.
This paper reveals that directly applying the classic K-Means clustering algorithm to DINOv2 features can yield meaningful object-centric representations. To enhance flexibility, the authors propose a multi-round K-Means clustering pipeline that utilizes varying K values and hierarchical resolutions. Experiments demonstrate the characteristics of the object-centric embeddings captured through K-Means clustering and compare these results to those obtained from learning-based methods.
- The main weakness of this work is the lack of novelty. DINOv2 features are already well-established for their ability to be clustered to represent objects, as evidenced not only by the original figures in DINO but also by numerous studies (both learning-based and non-parametric) in the field of unsupervised segmentation that have explored this capability. Consequently, it is expected that a simple K-Means clustering algorithm can be applied directly to DINOv2 features to yield meaningful object-centric representations.
- K-Means clustering also offers limited flexibility, as the choice of the number of clusters K and the design of the hierarchical structure to achieve optimal performance are hyperparameters that vary per dataset. Furthermore, whether the origin information contained in the resulting object embeddings is closely tied to the specific downstream tasks being addressed, making it difficult to assert that maintaining full semantic information is a definitive advantage.
Yes
No ethics review needed.
1) Is it possible to evaluate more video/image-based segmentation tasks to show the quality of the semantics of the learned features?

2) What layer of self-attention to extract the patch tokens? Using keys, queries, or values?

3) Time and efficacy analysis: comparison with the baselines?

4) Ablation to the number of running clustering K?
In comparison with learning-based methods, the proposed method offers a set of advantages. 1) It doesn't require any training or fine-tuning to extract object features from the off-the-shelf models. 2) It also renders object features with different levels of granularity on multiple objects/parts with multiple running of the k-means algorithm. 3) The centroids of the extracted features hold semantics that can be applied to downstream tasks.
This paper presents a simple and effective clustering-based method for obtaining object-centric feature representations from state-of-the-art ViT models trained with self-supervision. While existing object-centric learning methods are designed with specialized architectures or training techniques, they are limited to specific granularity and small flexibility. The proposed method takes patch features from state-of-the-art ViT models and applies multiple steps of the K-means algorithm to obtain a small set of cluster centroids that represent objects at different levels of granularity. The proposed method shows efficacy at downstream tasks such as image and video classification, with much lower computational cost compared with the baselines.
1) There is a lack of interpretability and expandability of the "object-centric" features extracted from ViT models. Current ViT models DINO and DINOv2 take image patches as input and are optimized with self-supervised losses. The self-attention layers extract the features at the image patch level, not the object level.  Simply applying K-means clustering on patch features does not guarantee that these features stand for the objects in the images. Wondering about the rationale behind why these features are defined as "object-centric features"?

2) The whole framework of the proposed method relies on the "well-trained features" from ViT models. With some epochs of clustering, these features are used for downstream tasks. Despite its simplicity, I wonder if further steps involving fine-tuning could be taken to update the ViT backbone. This is meaningful as it can help refine the self-supervised ViT features to bring out more semantic information.

3) What is the motivation for choosing video action recognition as one of the experiment setups? To evaluate the semantics quality of object-centric features, it may be a good idea to evaluate on more segmentation tasks on video or image datasets.
Yes
No ethics review needed.
Please see weaknesses.
* Learning object-centric representations in an unsupervised manner is both crucial and challenging, making the task addressed in this paper highly significant.
* Evaluating object-centric representations beyond object segmentation, on downstream tasks like video action recognition, is an important demonstration of their usability. The evaluation uses diverse and comprehensive datasets.
* The paper is well-written and easy to follow.
This paper introduces an object-centric framework for extracting object representations without the need for training. The proposed approach involves applying simple k-means clustering to features from off-the-shelf backbones, specifically DINOv2 [1]. The centroids from clustering, computed with different numbers of clusters (k values), are stored to capture varying levels of granularity. The resulting set of centroids is then treated as the object-centric representation of the image. These representations are evaluated on downstream tasks such as classification and video action recognition to assess their quality, and on image segmentation to determine how well they visually align with objects.

---
[1] Maxime Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, arXiv
* Applying clustering to pretrained encoder features is a well-known concept. Previous work in object-centric learning, such as DINOSAUR [1] and SOLV [2], has reported k-means clustering results in their ablations. Furthermore, clustering DINO [3] features has been used for segmenting objects in videos [4] and images [5, 6, 7]. This paper currently provides an evaluation of this well-established clustering approach on different benchmarks, but it is unclear how the framework represents a significant improvement or introduces novelty.

* In section 3.2, assuming overlapping masks across different granularities is an advantage over current object-centric methods, it’s worth noting that SAM [8] can extract overlapping masks for any object at any scale with pixel-level precision. This approach offers higher precision than patch clustering and can be used to extract object-centric representations in a zero-shot manner. What is the advantage of the proposed model compared to SAM in this regard?

* Since the CLS token is included in the $S_N$ representation set, it is expected to perform similarly in single-object classification tasks (e.g., ImageNet). For multi-object datasets like COCO, centroids should enhance performance by adding more granularity, whereas the CLS token summarizes the entire scene. Therefore, the comparison with the CLS token in Table 1 doesn’t provide much insight. Moreover, the approach uses multiple levels of granularity ($k = \\{1, 2, 4, 8\\}$), whereas SPOT [9] is limited to a single scale (7, 8, or 16), placing SPOT at a disadvantage. For a fair comparison, SPOT should include multi-scale representations, similar to the evaluations in Tables 2 and 3.

* For video action recognition (Figure 3), how does dropping some portion of tokens in the “all-patches” setting affect the tradeoff between token count and performance? Redundant tokens are common in video understanding tasks [10]. This comparison is important to highlight the efficiency of clustering centroids.

* The proposed approach leverages DINOv2 [11], which has been shown to significantly improve performance in object-centric tasks [2], while other models in Table 3 rely on the original DINO. Despite this advantage, both DINOSAUR and SPOT outperform the proposed method when using a single scale, i.e., a fixed number of slots/centroids, suggesting that the performance gain may primarily stem from the use of multi-scale representations. Given this, a multi-scale version of prior object-centric representations, such as a collection of DINOSAUR slots with $S = \\{1, 2, 4, 8\\}$, is expected to provide richer information than k-means clustering.

---
[1] Maximilian Seitzer et al., Bridging the Gap to Real-World Object-Centric Learning, ICLR 2023

[2] Görkay Aydemir et al., Self-supervised Object-centric Learning for Videos, NeurIPS 2023

[3] Mathilde Caron et al., Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021

[4] Georgy Ponimatkin et al., A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation, WACV 2023

[5] Paul Engstler et al., Understanding Self-Supervised Features for Learning Unsupervised Instance Segmentation, arXiv

[6] Gyungin Shin et al., Unsupervised Salient Object Detection with Spectral Cluster Voting, CVPRW 2022

[7] Xudong Wang et al., Cut and Learn for Unsupervised Object Detection and Instance Segmentation, CVPR 2023

[8] Alexander Kirillov et al., Segment Anything, ICCV 2023

[9] Ioannis Kakogeorgiou et al., SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive, CVPR 2024

[10] Tengda Han et al., Turbo Training with Token Dropout, BMVC2022

[11] Maxime Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, arXiv
Clustering DINOv2 features with k-means produces better object-centric representations than sota slot-based models, eg for scene and video classification. Object masks are more general (overlaps), capture objects/parts, but are less pixel-perfect.
@inproceedings{
anonymous2024a,
title={A Clustering Baseline for Object-Centric Representations},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Z56fPyx7GL},
note={under review}
}
Object-centric learning aims to discover and represent visual entities as a small set of vectors and masks, which can be later used for downstream tasks.
Recent methods for object-centric learning build upon vision foundation models trained with self supervision because of the rich semantic features they produce.
However, these methods often require additional training, offer limited flexibility, and are optimized for a specific granularity of objects, e.g. to maximize segmentation metrics on a test dataset.

In this work, we demonstrate how to discover objects and parts with a simple clustering algorithm applied to the features of an off-the-shelf backbone.
K-means, as the clustering method of choice, is fast and flexible, produces interpretable masks, preserves the quality of the backbone embeddings, does not require additional training, and can be applied multiple times to capture different part/whole structures, e.g. a laptop and its keyboard.
The resulting representation is a small set of object tokens that compactly summarizes the image contents.

We evaluate the quality of such a representation on a variety of downstream tasks including scene classification and action recognition in videos, showing that it surpasses the performance of fine-tuned object-centric learning methods.
Furthermore, we evaluate the corresponding object masks for unsupervised segmentation on annotated datasets.
Though not as pixel-perfect as specialized methods, we show that they effectively capture real-world objects and parts at various granularity.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
object-centric representations self-supervised learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ea2548591f3c21d4b264b49f34083c2cbc167be5.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
A Clustering Baseline for Object-Centric Representations
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission