MORPHING TOKENS DRAW STRONG MASKED IMAGE MODELS
Self-supervised learning
Yes
No ethics review needed.
- For the Token encoding section of "4.1 Preliminary" section, consider adding more introduction/references to the concept introduced, for example "online encoder" and "target encoder.
- It will be easier for readers to follow if there is an additional figure about how to obtain the morphing matrix M.
- The paper achieves SOTA results on ImageNet-1k. 
- The proposed method is applicable to various SSL frameworks such as MAE, BEiT and BYOL.
- The paper is very well written and has extensive experiments and visualizations.
- The "Pilot Study" section gives very good narrative to the paper and provides intuitive motivations.
The paper proposes a novel masked image modeling method. The paper identified the issue of the existing pre-training models: the spatial consistency in target representations despite proximity and contextual similarity. Then the author proposed a context-aware token aggregation method called Dynamic Token Morphing (DTM). The experiments show that the proposed DTM method is general enough to help improve various SSL frameworks and achieve SOTA ImageNet-1k performance. In addition, the computation complexity add-on of the proposed method is not large.
The paper is very well-written and achieves SOTA results, I don't have weakness to add.
Yes
No ethics review needed.
In addition to weakness, there are also some questions:
1. In line 239, "we do not employ a superpixel-based clustering for our method, which is inefficient as well." Is the decision to avoid superpixel-based clustering solely due to its low performance? Could the reasons be analyzed further?
2. Please explain why the absence of a dynamic mechanism results in lower performance than the baseline in Table 7.
1.This paper provided an extensive discussion and demonstration of the spatial inconsistency in pre-trained representations, illustrating the significance of this issue in the domain of self-supervised learning.
2. DTMs re-characterize visual features, mitigating the inflexibility associated with patch-based representations.
3. This paper integrated DTM with multiple baseline methods, such as MAE, BEiT v2, BYOL, etc., demonstrating the effectiveness and convenience of the proposed approach.
This paper investigates the issue of spatially inconsistent target representations during pre-training. The authors conducted quantitative and qualitative experimental observations of this phenomenon to discuss the impact of such spatial inconsistencies, particularly on the performance of downstream classification tasks. To address this issue, based on the MIM baseline, they propose an advanced token aggregation method termed Dynamic Token Morphing (DTM), which integrates tokens with contextual associations. The paper reports the classification results on ImageNet-1k and the segmentation performance on ADE20k, demonstrating consistent improvements with DTM across the SSL framework and the SLIP model.
1. Although DTM presents an intuitive feasibility in addressing the spatial inconsistency of visual representations, the explanation in the methods section is not very intuitive or detailed, which may lead to comprehension errors. For instance, the process of how to perform k iterations based on bipartite matching to obtain the final token morphing matrix M, and how to derive the aggregated results from M, is not clear. Perhaps an algorithm flowchart would be beneficial.
2. Some details in the writing of the article could be further optimized. For example, on line 264, whether generating vi with xiM should not include masked tokens, as there is an inconsistency with Figure 4 (the input of the target encoder does not include masked tokens). Additionally, in Figure 4, the lines connecting the Morphing Matrix to Vi and Ui are different; does this indicate different operations?
3. The authors emphasize in the abstract that DTM brings about faster training, but when comparing with other methods, especially the baseline method, the best results are achieved using the same number of epochs. Does this imply that the overall training time was reduced?
Yes
No ethics review needed.
See the second point of Weaknesses.
1.A interesting insight on masked image modeling pre-training methods and good results are achieved across all scale models including ViT-S/B/L.
2.The proposed method can be used for both MIM and contrastive learning mothods
This paper proposes Dynamic Token Morphing (DTM) to mitigate the identified spatial inconsistencies discriminative problem. This method can be used fot both masked image modeling methods and contrastive methods and accelerating the training. The achieved results look good and I think the novelty is also good. But I concern several weaknesses as I noted below. I may change the rate after reviewing other reviewers' review.
1.Why not compare your method with hierarchical ViTs, such as ConvMAE [1], HiViT [2], iTPN [3], GreenMIM [4]. Is your method still effective on these methods? I suggest conducting an experiment on maybe one Small scale model.
2.The authors argue that the current methods meets the trouble of spatial inconsistencies, but existing ones actually have achieved very good results, such as EVA [5], EVA02 [6], Fast-iTPN [7]. How the authors expain that? If you think the proposed method can still improve their performance, I think conducting experiments is necessary.
3.Missing COCO experiments

[1]ConvMAE: Masked Convolution Meets Masked Autoencoders, [NeurIPS 2022]
[2]HiViT: A Simpler and More Efficient Design of Hierarchical Vision Transformer, [ICLR2023]
[3]Integrally Pretrained Transformer Pyramid Networks, [CVPR2023]
[4]Green Hierarchical Vision Transformer for Masked Image Modeling, [NeurIPS 2022]
[5]EVA: Exploring the Limits of Masked Visual Representation Learning at Scale [CVPR2023]
[6]EVA-02: A Visual Representation for Neon Genesis
[7]Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migration [TPAMI2024]
We identify spatial inconsistencies of Masked Image Modeling targets tokenized from pre-trained models and resolve the issue through our proposed Dynamic Token Morphing
@inproceedings{
anonymous2024morphing,
title={{MORPHING} {TOKENS} {DRAW} {STRONG} {MASKED} {IMAGE} {MODELS}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=d7q9IGj2p0},
note={under review}
}
Masked image modeling (MIM) has emerged as a promising approach for training Vision Transformers (ViTs). The essence of MIM lies in the token-wise prediction of masked tokens, which aims to predict targets tokenized from images or generated by pre-trained models like vision-language models. While using tokenizers or pre-trained models are plausible MIM targets, they often offer spatially inconsistent targets even for neighboring tokens, complicating models to learn unified and discriminative representations. Our pilot study identifies spatial inconsistencies and suggests that resolving them can accelerate representation learning. Building upon this insight, we introduce a novel self-supervision signal called Dynamic Token Morphing (DTM), which dynamically aggregates contextually related tokens to yield contextualized targets, thereby mitigating spatial inconsistency. DTM is compatible with various SSL frameworks; we showcase improved MIM results by employing DTM, barely introducing extra training costs. Our method facilitates training by using consistent targets, resulting in 1) faster training and 2) reduced losses. Experiments on  ImageNet-1K and ADE20K demonstrate the superiority of our method compared with state-of-the-art, complex MIM methods. Furthermore, the comparative evaluation of the iNaturalists and fine-grained visual classification datasets further validates the transferability of our method on various downstream tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Self-supervised learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b586a89e95ce331fe456da84befda61212f3f663.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
MORPHING TOKENS DRAW STRONG MASKED IMAGE MODELS
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission