From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
Multimodal Large Language Models;Image Tokenizer;Token Merge
Yes
No ethics review needed.
1. Why is there a 0.99 in L770 and L253? Is this made up?
2. For Figure 2 (b) (c), all settings converge after ~150 iterations. I don't think they make any difference.
3. Why vocabulary changes from 8k-> 16k, there is a performance drop? I can not find any evidence in the proof that can demonstrate this point.
4. The proposed BPE is very similar to super pixel or clustering algorithm. Authors should discuss the difference and compare the performance. 
5. In table 5.1, authors can add another classical setting: During SFT, visual encoder is frozen.
1. From the results shown to us, there is some improvement.
2. Experiment settings are clear
This paper tried to use BPE for image tokenization. From the results shown to us, there is some improvement.
1. Performance is poor compared to any CLIP style or even DINO style MLLM as the visual encoder.
2. There is no projector in the experiments. This could be an extreme unfair setting compared to classical pipeline.
3. I do not think proofs are helpful to understand what is going on in the experiments.
Yes
No ethics review needed.
1. The LLM+VQ+BPE is also supervised finetuned on LLaVA-One-Vision, etc data, however, is far behind LLaVA-OneVision and other models that trained on these data. Then what's the benefit of this VQ+BPE compared with previous MLLM practices?
2. Is this VQ+BPE applied to other LLM beyond LLaMA-3.1-8B and get similar observations?
1. This BPE image tokenization approach is novel and that potentially help the transformer better understand alignment between text and image with a semantic image token.
2. There is a theoretical analysis on how BPE tokenize benefits transformers learning in Section 3.
3. The scaling of BPE is reflected in that the model improves when adding larger scale of data such as ShareGPT4, etc.
This paper proposes to apply Byte-Pair Encoding to visual data, which first encode image into discrete token IDs, and then train the BPE image tokenize to get image tokens with semantic prior (e.g., previously image tokens for 'a white cat' is separated in the sequence of image tokens, after BPE, one token is representing 'a white cat'). The experiments are mainly based on applying BPE image tokenizer training to LLaMA-3.1 and compare on VQAv2, MMBench, etc.
1. The experimental evidences are kind of weak. First, it's far behind current MLLMs SOTA on public benchmarks. For example, the best presented number of proposed model is LLM+VQ+BPE with Additional scaling (SFT) , which achieves 60.6 on VQAv2, 44.0 on MMBench, and 48.2 on VizWiz, which is far behind similar size 7B LLaMA-based MLLMs.
2. Second, the ablation is not sufficient to show the benefit of BPE image tokenizer. Only one Table results compare LLM+VQ and LLM+VQ+BPE. The details of these two models are not illustrated, e.g., what is exactly implemented for LLM+VQ.
Yes
No ethics review needed.
1.Applicability of the Theoretical Model: How does a simplified 2D Markov process adequately capture the complex structure of real-world image data?

2.Sensitivity to Information Loss: How is the potential impact of information loss in tokenization, especially for detail-sensitive tasks, theoretically assessed?

3.Task Representativeness and Generalization: How can results on VQA-like tasks ensure performance on precision-demanding tasks like image caption?
1.This paper creatively adapts byte-pair encoding (BPE) for images, aiming to make visual data work more seamlessly with text in multimodal models.

2.The approach integrates structural information directly into image tokens, which could help models better understand and align visuals with text, showing solid potential in cross-modal tasks.
This paper presents a novel BPE image tokenizer that brings byte-pair encoding (BPE) to image tokenization, enhancing multimodal large language models (MLLMs) in aligning visual and textual information. By embedding structural priors into image tokens, the method shows promise in cross-modal tasks and scalability, offering a new approach to unified visual and text processing.
1.Theoretical framework has several notable limitations:
  1.1Lack of Multimodal Fusion Analysis: The paper’s theoretical analysis is focused on 2D image data alone and does not delve into how the BPE tokenizer facilitates the fusion of visual and textual information. Multimodal tasks typically require deep semantic and structural alignment across modalities, which is not sufficiently addressed in this analysis. This omission limits the theoretical support for the tokenizer’s efficacy in a multimodal model context.
  1.2Absence of Analysis on Information Loss in Tokenization: The paper lacks a theoretical exploration of the potential information loss from BPE tokenization, such as the simplification of high-frequency visual details. There is no quantification of how the loss of these details might impact overall model performance. This gap in the analysis leaves the question of how well the BPE tokenizer preserves image details unanswered.

2.A notable limitation of this paper is its focus on evaluating the BPE image tokenizer primarily through VQA-like tasks, which generally require only broad semantic alignment across modalities. While effective for assessing general multimodal comprehension, these tasks may not fully capture the demands of applications like image segmentation or image caption, where finer-grained visual detail and spatial relationships are crucial. Without evaluation on these more intricate tasks, it remains unclear how well the method handles scenarios that require detailed visual representation, potentially limiting its applicability to real-world multimodal use cases that demand high visual fidelity.
We introduce a novel BPE Image Tokenizer that applies BPE tokenization to visual data, enabling more effective integration of visual information in MLLMs and improving their performance.
@inproceedings{
anonymous2024from,
title={From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=3TnLGGHhNx},
note={under review}
}
Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities. We introduce a novel image tokenizer that bridges this gap by applying the principle of Byte-Pair Encoding (BPE) to visual data. Unlike conventional approaches that rely on separate visual encoders, our method directly incorporates structural prior information into image tokens, mirroring the successful tokenization strategies used in text-only Large Language Models. This innovative approach enables Transformer models to more effectively learn and reason across modalities. Through theoretical analysis and extensive experiments, we demonstrate that our BPE Image Tokenizer significantly enhances MLLMs' multimodal understanding capabilities, even with limited training data. Our method not only improves performance across various benchmarks but also shows promising scalability, potentially paving the way for more efficient and capable multimodal foundation models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multimodal Large Language Models Image Tokenizer Token Merge
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/262fa2eb02a6924e7d350ce9a33d207c650680fb.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission