Visual Context Window Extension: A New Perspective for Long Video Understanding
Large Multimodal Models;Long Video Understanding;Visual Context Window
Yes
No ethics review needed.
1. Regarding the visual context window extension, some clarification is needed to prevent misunderstanding. In the context of video question-answering, which of the following interpretations is correct?
- Interpretation 1: The visual context window and language context window coexist. In other words, during video question-answering, RoPE encoding for the visual part uses the visual context window for interpolation, while the textual part uses the language context window.
- Interpretation 2: The same visual context window is used for interpolation for both the visual and textual parts during video question-answering.

If Interpretation 1 is correct, how is attention computed between the visual and textual parts given the difference in relative position encoding?

2. According to the citation of allava dataset, it seems to be an image dataset. How can it be used to fine-tune the video understanding model in the paper?

3. In the ablation study of progressive pooling strategy, why does (2,4),4 perform better in some cases compared to (2,2),0?

4. Why does (2,8),4 outperform (2,4),4? Moreover, given that 8 is a very high compression ratio, if the ViT patch size is 14×14, this would compress a 224×224 image frame down to a length of $224/14/8=2$ per side, equating to 4 tokens. Since inter-frame pooling is independent, there is a possibility that dynamic information is not captured.
Could it be that the highly compressed frames are actually not contributing? A simple experiment would be, comparing with (2,2), 0 with 64 sampled frames. 

5. Will the code and model checkpoints be open-sourced?

If all the questions above are solved, I would raise my score.
1. Effective adaptation and optimization of YarN for long video understanding. The paper highlights the need for different context windows for vision and language, enhancing long video comprehension.

2. The progressive pooling strategy with dual compression ratios reduces training and inference costs while retaining dynamic information.
This paper investigates the application of large language models (LLMs) for long video understanding. It first identifies that discrepancies between visual and language modalities lead to different context windows for visual and language tokens. Inspired by the YarN approach, the paper proposes an improvement to the visual context window for video understanding models using RoPE encoding. Additionally, to address the issue of high token count and the associated training and inference overhead for long videos, a progressive pooling strategy is introduced, reducing memory consumption by 45%. The proposed model achieves state-of-the-art results on benchmarks such as MLVU.
1. More ablation study and analysis should be done regarding progressive pooling strategy, raising doubts about the role of high-compression frames (see questions 3 and 4).

2. If the answer to Question 1 is "the same context window is used for both language and video parts in video question answering tasks", this part seems more like parameter tuning specific to a particular model and training data setting. It has some contributions but may lack novelty.
Yes
No
No ethics review needed.
1. The progressive pooling strategy may also result in some loss of information.

2. The paper lacks a detailed overall framework diagram to showcase the entire process.

3. The paper may not be entirely fair in comparison with methods that use fewer frames, as a higher number of frames inherently contains more information.
This paper demonstrates a significant strength in addressing the limitations of Large Multimodal Models (LMMs) in long video understanding by proposing the Visual Context Window Extension, which leverages the inherent differences between visual and language modalities. 
Another strength is the introduction of a progressive pooling strategy that effectively reduces memory consumption by approximately 45% without sacrificing performance, making the approach more feasible for practical applications.
 Lastly, the paper's approach shows consistent improvements in performance as the number of video frames increases, outperforming other models on benchmarks like MLVU, which is a substantial contribution to the field of long video understanding.
The paper introduces a novel approach called Visual Context Window Extension for enhancing long video understanding using Large Multimodal Models (LMMs). It addresses the challenge by extending the visual context window without the need for retraining on long video datasets, leveraging the differences between visual and language modalities. The paper also proposes a progressive pooling strategy to reduce memory consumption, leading to improved performance and efficiency in long video understanding tasks.
1. The progressive pooling strategy may also result in some loss of information.

2. The paper lacks a detailed overall framework diagram to showcase the entire process.

3. The paper may not be entirely fair in comparison with methods that use fewer frames, as a higher number of frames inherently contains more information.
Yes
No ethics review needed.
The authors fixed the input frames to 256/512. Could this approach be extended to handle more frames?

What are the results of fine-tuning on 256 frames and then extending to a higher number of frames?

Could the authors provide results on MVbench? I'm interested in whether this visual context extension impacts the model's ability to understand short videos.
1.  The motivation is clear. Figure (a) effectively illustrates the challenge of long video understanding due to the limited visual context window.

2.  The proposed visual context window extension significantly improves baseline performance without additional tuning.

3.  The paper is well-written and easy to follow.
The paper focuses on adapting LMMs for long videos understanding. The authors propose extending the visual context window to adapt LMMs for long videos. They introduce a pooling strategy to reduce memory usage by selectively lowering the resolution of frame embeddings. This approach improves performance on long video tasks, while cutting memory usage by 45% without losing accuracy.
Overall, the technical contribution appears limited, as it primarily involves modifications to YaRN.

Additionally, Table 3 shows that the proposed method performs worse on short videos.
Yes
No ethics review needed.
1. In Figure 1(b), the diagram illustrates the modality gap between visual and language embeddings, which leads to the visual context window extending beyond its range, resulting in a decrease in performance. Could you clarify how extending the visual context helps to narrow this gap? 
2. How does the clustering visualization of visual and language embeddings change after implementing the proposed window extension strategy?
3. In line 100, the author states that they "believe the first frame preserves rich information" and in line 264, they "assume each uniform group represents an event." These statements do not seem technically sound, and the author does not provide substantial evidence to support them. Given that a video contains dynamic content, a uniform sampling strategy might not effectively separate distinct events. Could you explain why the first frame is considered to preserve rich information?
4. Does this visual/text context extension support the interleaving of visual and text features?
5. How does the LLaVA-OneVision model perform when utilizing the full context length of Qwen2, instead of uniformly sampling 32 frames on MLVU and LongVideoBench?
+ The method adapt YaRN, originally designed for extending language context windows, to handle visual context windows separately.
+ The performance is impressive on several long-video benchmarks such as VideoMME, MLVU and LongVideoBench
+ The finetuning process does not require long video datasets.
This paper proposes visual context window extension of MLLMs for long video understanding, allowing them to process longer video sequences without retraining on long video datasets. Additionally, the paper uniformly pooling several frame tokens for memory concern. The method is evaluated across several benchmarks, showing competitive performance, surpassing GPT-4o on the MLVU benchmark.
+ Several theoretical insights are missing. Specifically, it is unclear why a uniformly sampled group is assumed to represent an event, and the rationale behind choosing to maintain the first frame at a high resolution is not explained. For further details, refer to questions 1, 2, 3.
+ The contribution is limited in terms of progressive pooling. Firstly, there is no progressive process. The method simply maintains the first frame at high resolution while spatially pooling the subsequent frames to a lower resolution. 
+ The proposed strategy seems hard to handle the scenarios when the input consists of interleaved visual and text embeddings.
This paper enhances long video understanding in Large Multimodal Models by extending visual context windows and introducing a progressive pooling strategy.
@inproceedings{
anonymous2024visual,
title={Visual Context Window Extension: A New Perspective for Long Video Understanding},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=X4Rcxi9588},
note={under review}
}
Large Multimodal Models (LMMs) have demonstrated impressive performance in short video understanding tasks but face great challenges when applied to long video understanding. In contrast, Large Language Models (LLMs) exhibit outstanding capabilities in modeling long texts. Existing work attempts to address this issue by introducing long video-text pairs during training. However, these approaches require substantial computational and data resources. In this paper, we tackle the challenge of long video understanding from the perspective of context windows, aiming to apply LMMs to long video tasks without retraining on long video datasets. We first conduct an in-depth analysis of why pretrained LMMs struggle to understand lengthy video content, identifying that discrepancies between visual and language modalities lead to different context windows for visual and language tokens, making it difficult to directly extend the visual tokens to match the language context window. Based on this, we propose to adapt LMMs for long video understanding tasks by extending the visual context window, eliminating the need for retraining on large-scale long video datasets. To further mitigate the significant memory consumption caused by long sequences, we introduce a progressive pooling inference strategy that selectively adjusts the spatial resolution of frame embeddings, reducing the number of visual tokens while retaining important spatial information. Across multiple long video understanding benchmarks, our method consistently improves the performance as the number of video frames increases. On the MLVU benchmark, our method outperforms GPT-4o, even though our model size is only 7B. Additionally, in the 256-frame setting, our method reduces memory usage by approximately 45% compared to the baseline, without introducing any performance loss.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Multimodal Models Long Video Understanding Visual Context Window
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f0d738519d4612206a0c48ce884ae471d08fbd6f.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Visual Context Window Extension: A New Perspective for Long Video Understanding
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission