On Exploring Visual Attention Shrinking for Accelerating VLMs for Video Understanding
Visual Language model;Inference Acceleration;Visual Attention Shrinking
Yes
No ethics review needed.
See weaknesses.
1. Tuning-free and plug-and-play: Being tuning-free and plug-and-play, The proposed method can be seamlessly integrated with existing VLMs without the need for extensive modifications or retraining, facilitating broader adoption.
2. Efficient Token Usage. Removing less important visual tokens during inference is an intuitive motivation.
3. Well-structured presentation. The presentation of this paper is clear and easy to understand.
This paper proposes a method to improve the efficiency of vision-language models for video tasks by reducing redundant visual tokens during inference. The authors introduce an algorithm to detect and remove low-importance tokens along spatial and temporal dimensions. Their approach requires no parameter changes, is compatible with KV caching, and achieves substantial speedups, using only 47.2% of tokens without degrading task performance, especially benefiting large models like VILA1.5-40B.
1. Lack sufficient experimental support. It would be beneficial to include evaluations on other challenging video benchmarks, such as long video datasets, to validate effectiveness and enable a more comprehensive comparison.
2. Limited novelty. The concept of removing redundant tokens based on attention scores is not entirely new and has been explored in other VLM and transformer-based model optimizations. This approach may not offer a substantial advancement beyond existing methods.

Minor issues: These are some typos in the paper, such as Line 464 and the captions of Table 1.
Yes
No ethics review needed.
The paper mentions that the token reduction algorithm performs token reduction at certain layers, but what is the basis for selecting which layers to apply token reduction?
Can the dynamic visual token reduction algorithm you proposed be extended to other types of multimodal data beyond video?
The paper introduces the phenomenon of Visual Attention Shrinking (VAS) and develops a dynamic token removal algorithm based on this phenomenon, providing a new approach to improve the inference efficiency of video VLMs.
The proposed algorithm does not require parameterized modifications to the original VLM and is compatible with the prevalent KV cache strategy, demonstrating good generality.
The proposed method can perform token removal in both temporal and spatial dimensions, enhancing inference speed.
This paper introduces a dynamic token removal method based on the phenomenon of Visual Attention Shrinking (VAS), which identifies less important visual tokens during inference. The algorithm operates without modifying the original VLM and is compatible with KV cache strategies.
The paper primarily demonstrates the effectiveness of VAS through experimental results, but it lacks more intuitive visual analyses. It is recommended to use visualization tools (such as heatmaps, attention maps, etc.) to clearly illustrate the changes in attention distribution of the model before and after token removal, as well as the impact on the model's inference process.

The proposed method does not perform as well on certain specific tasks (such as Egoschema and MLVU) and fails to adequately explore the reasons for this.
Yes
No ethics review needed.
1.  Is the shape of $T$  $m$ or $n$? If it is $m$, then in Line 8 of Algorithm 1, $T$ should be summed over dimension 1. It appears there is a conflict between Line 8 and Line 10 in Algorithm 1.
2. How about multi-turn conversations?
1. The phenomenon of Visual Attention Shrinking where some frames or positions consistently exhibit a downward trend in attention scores, is interesting.
2. In the current field of video VLM, a large number of visual tokens are commonly used to represent videos. It is very meaningful to explore token reduction in this field.
3. The experiments are comprehensive, including multiple Benchmarks and video VLM models, and the method shows a significant acceleration for the models.
This paper identifies an interesting phenomenon termed as Visual Attention Shrinking (VAS), wherein certain visual tokens receive progressively diminishing attention during the processing stages of the model, and develop a robust algorithm to detect attention shrinking at each layer of the model using states from preceding layers.

The contributions are summaried as follows:
1. observe an interesting phenomenon called Visual Attention Shrinking.
2.  develop an algorithm that utilizes the attention states from previous layers to robustly detect attention shrinking, and continuously removes visual tokens based on the detection results. 
3. extensive experiments on various VLMs, across multiple video benchmarks, validate the effectiveness of propsoed method.
1. In certain settings, the proposed method experiences a higher drop in performance, such as with LongVA-7B and VILA1.5-40B.
2. Although the article focuses on accelerating the inference of video VLM models, it lacks substantial design specific to video.
Yes
No ethics review needed.
Please revise the Weaknesses section point by point. This is a paper with great potential. If the authors can provide additional responses to certain issues, discuss related work more thoroughly, and include more experiments and observations, I would be very happy to raise my score. Additionally, please redraw Figure 1, as it took me twice the time to understand what VAS is actually doing!
1. The efficiency and performance of the proposed method are competitive compared to ToMe and FastV.

2. This is an architecture-agnostic approach, meaning it can be widely applied to various LMMs.

3. Although the pipeline figure in the paper is somewhat confusing, the textual organization is well-structured and makes it understandable.

4. In addition to the basic experiments, the authors included some observations (such as Figure 2) and ablation studies (Tables 3 and 4) to make the work more comprehensive.

5. The motivation of the paper is reasonable, and I believe this is indeed an urgent issue that needs to be addressed in LMMs.
The large number of visual tokens creates significant inconvenience for both the training and inference of large multimodal models (LMMs). Therefore, reducing and removing unnecessary visual tokens is an important direction for development. This paper observes that certain visual tokens receive progressively less attention during the processing stages of the model. Based on this observation, token removal is performed along the temporal and spatial axes without requiring architectural modifications. This strategy can reduce the number of visual tokens by approximately 50%, with almost no performance degradation, while significantly speeding up inference.
1. The proposed VAS method does not actually show a sufficiently significant performance difference compared to ToMe [1] and FastV [2]. As shown in Figure 1, when VAS is applied to PLLaVA-7B, the performance on MVBench, VideoMME, and MLVU does not demonstrate a notable advantage (e.g., 46.6 vs 46.9, 48.5 vs 49.0). Furthermore, when VAS is applied to LongVA-7B, it consistently falls behind ToMe across all benchmarks.

2. There are many instances of "Failed" in Table 1, and the authors lack a reasonable explanation for this. The only mention of it is in Section 4.2.2, where it is briefly noted that "FastV’s aggressive reduction approach proves to be overly drastic, often resulting in model abnormal responses." However, this is merely a superficial description of the phenomenon.

3. The authors lack a direct comparison of the proposed VAS method with prior works, such as ToMe [1] and FastV [2], in terms of motivation and technique. Compared to FastV [2], both methods evaluate the importance of visual tokens based on attention weights, identifying tokens that can be discarded without sacrificing model performance. Both approaches use gradual token reduction strategies, ensuring that important tokens are preserved during the early stages of inference to prevent the loss of critical information.

4. The authors lack citations and discussions of other works focused on token merging, such as LlamaVid [3] and MovieChat [4], among others. Additionally, the authors' discussion of Chat-UniVi is inaccurate. The claims that it "attempted to adjust the architecture of the visual encoder" and "introduced additional training requirements" do not align with the original paper's descriptions.

5. Although the authors emphasize "lengthy videos" in the abstract, the experiments lack analysis and emphasis on VAS's performance and observations in long video sequences. For example, does attention dispersion become more severe with longer sequences? It would have been better if the authors had conducted more detailed observations on benchmarks like MovieChat-1K [4] or LongVideoBench [6].

[1] Bolya, Daniel, et al. "Token merging: Your vit but faster." arXiv preprint arXiv:2210.09461 (2022).

[2] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." arXiv preprint arXiv:2403.06764 (2024).

[3] Li, Yanwei, Chengyao Wang, and Jiaya Jia. "Llama-vid: An image is worth 2 tokens in large language models." European Conference on Computer Vision. Springer, Cham, 2025.

[4] Song, Enxin, et al. "Moviechat: From dense token to sparse memory for long video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Jin, Peng, et al. "Chat-univi: Unified visual representation empowers large language models with image and video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[6] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." arXiv preprint arXiv:2407.15754 (2024).
@inproceedings{
anonymous2024on,
title={On Exploring Visual Attention Shrinking for Accelerating {VLM}s for Video Understanding},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=oS79Tw3G0c},
note={under review}
}
Vision-language models (VLMs) have shown promise in a variety of challenging video comprehension tasks. VLMs typically extract frames from the source video and take the corresponding encoded visual tokens as input. A rapid increase in the number of visual tokens, e.g., when handling lengthy videos, can swiftly lead to a long-context dilemma during the inference process of VLMs, posing an efficiency challenge for real-world applications. Given that significant redundant and task-irrelevant information may exist in the visual tokens across both spatial and temporal axes, we advocate removing less important visual tokens during the prefilling phase of the inference procedure to improve the computation and storage efficiency of VLMs. We first identify an interesting phenomenon termed as \emph{Visual Attention Shrinking (VAS)}, wherein certain visual tokens receive progressively diminishing attention during the processing stages of the model. This implies that the model itself knows what to care about and what to discard. With this understanding, we develop a robust algorithm to detect attention shrinking at each layer of the model using states from preceding layers. Based on the detection results, we perform token removal in both temporal and spatial axes. Our approach does not require parameterized modifications to the original VLM and is compatible with the prevalent KV cache strategy. Through extensive experiments across different VLMs, our approach witnesses an average speedup of $1.98\times$ in generating the first response token, utilizing only 47.2% of the visual tokens, without compromising the task performance. Additionally, when applied to the huge VILA1.5-40B, our method can achieve up to $4.16\times$ speedup compared to the vanilla model.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Visual Language model Inference Acceleration Visual Attention Shrinking
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5617f82e561ce7c62ea51f5daf90c9db84713976.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
On Exploring Visual Attention Shrinking for Accelerating VLMs for Video Understanding
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission