VeLAR: Vision-oriEnted Language-Attentive token Reduction for multimodal large language models
Multi-modal Large Language Models;Token Reduction;Model Aceleration;Foundation Models;Vision-Language Learning;Instruction Tuning
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
We propose a token reduction framework for MLLMs that reduces vision token redundancy in vision-language learning, cutting computational costs by up to 42% while maintaining and even surpassing the original model performance.
@misc{
sun2024velar,
title={Ve{LAR}: Vision-oriEnted Language-Attentive token Reduction for multimodal large language models},
author={Yizheng Sun and Yanze Xin and Hao Li and Chenghua Lin and Riza Batista-Navarro},
year={2024},
url={https://openreview.net/forum?id=oXxIPDjM5R}
}
Multi-modal large language models (MLLMs) have made significant strides by integrating visual and textual modalities. However, architectures that pass all vision tokens to the language model (LLM), such as LLaVA, incur high computational costs due to the large number of vision tokens. While approaches using Q-formers as vision-language connectors reduce computational overhead by generating fewer vision tokens, they often experience performance degradation. In this paper, we propose a progressive token reduction method, called VeLAR, that retains the performance of LLaVA-based MLLMs while substantially reducing computational load. We introduce a lightweight cross-attention decision module where vision tokens attend to language tokens. This module is inserted into various layers of the LLM to compute a relevance score for each vision token, dynamically determining whether to prune it. During training, we apply a targeting pruning ratio with Gumbel-Softmax activation to maintain differentiability in the pruning process by attention masking, while in inference, the pruning ratio can be flexibly adjusted to consider different computational trade-offs without re-training. By progressively pruning redundant vision tokens throughout the LLM backbone, our method can reduce 87.5% vision tokens by the final layer and achieve up to a 42% decrease in FLOPs. Across 12 multi-modal benchmarks, the average performance loss is less than 1%, with superior performance observed in 7 of them.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Yizheng_Sun1 ~Yanze_Xin1 ~Hao_Li25 ~Chenghua_Lin1 ~Riza_Batista-Navarro1
Yizheng Sun Yanze Xin Hao Li Chenghua Lin Riza Batista-Navarro
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multi-modal Large Language Models Token Reduction Model Aceleration Foundation Models Vision-Language Learning Instruction Tuning
I certify that there is no acknowledgement section in this submission for double blind review.
sun|velar_visionoriented_languageattentive_token_reduction_for_multimodal_large_language_models
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
VeLAR: Vision-oriEnted Language-Attentive token Reduction for multimodal large language models
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission