Understanding Long Videos with Multimodal Language Models
long-video;visual question answering;interpretability
Yes
No ethics review needed.
As mentioned in the weakness.
+ It is interesting that this work introduces three key attributes essential for video understanding: Global Object Information, Object Spatial Location, and Object Motion Trajectory. These attributes contribute significantly to a more comprehensive analysis of video content.
This paper introduces two baselines and a novel approach called the Multimodal Video Understanding (MVU) framework for video understanding tasks. The baselines explore using either a single frame or no visual input at all. In contrast, MVU aggregates multimodal information relevant to the video to enhance understanding.
- Although this paper focuses on long-video understanding, it lacks specific design elements to address long-video scenarios. Challenges such as context length limiting input frames and modeling long-range temporal information are not directly addressed. The attributes introduced—Global Object Information (GOI), Object Spatial Location (OSL), and Object Motion Trajectory (MOT)—are not tailored to tackle these issues in long-video understanding.

- In Table 1, the comparisons may be weaker due to the differing training setups and base models used across experiments. As such, the superior performance of SF-LLM over the state-of-the-art does not necessarily imply that the number of frames is irrelevant for video understanding.

- Since the model is based on LLaVA-v1.5-13B, an important baseline is missing: the use of multiple frames as input for LLaVA-v1.5-13B.

- Certain details are lacking, such as the method for using LLaVA-v1.5-13B for frame sampling, as mentioned in Figure 3.

- Likelihood Selection is widely used in the MCQ benchmark as an additional track. For a fairer comparison, other methods should also incorporate this strategy for comparison results.

- The benchmark in this paper only includes mid-length videos, roughly under three minutes. To more competitively demonstrate MVU's capabilities in long-video understanding, it would be beneficial to evaluate on benchmarks like VideoMME and LongVideoBench.
Yes
No ethics review needed.
Please refer to the weaknesses.
1. The proposed likelihood selection approach offers a good way to speed up the inference in autoregressive LLMs. 
2. Going beyond existing video datasets, this paper further evaluates the generalization ability on Open-X-Embodiedment.
3. The paper is well written and easy to follow.
This paper proposes a training-free approach to understanding long-form videos by extracting explicit image-/object-level information. The extracted information is translated into natural language descriptions to make LLM ‘see’ the visual input. Experimental results on several videoQA benchmarks demonstrate the superiority.
1.	It seems that using likelihood as a selection criterion still focuses on the exact match between the generated text and the answer candidates in a per-logit manner, without considering the semantic meaning. For example, ‘C is washing plates’ vs ‘C is cleaning dishes’. 
2.	Unlike prior approaches where all answer candidates are fed together to the language model, the proposed likelihood selection method organizes the Q-A pairs in a batch dimension. In this way, it seems that LLM fails to analyze the relationship between answer candidates, increasing the difficulty of QA.
3.	If the frames are uniformly sampled across the entire long video, how can you ensure the consistent occurrence of objects? In certain cases, the appeared objects in each frame are completely different. Another related question is whether using X_{OSL} and X_{OMT} extracted from densely sampled frames (i.e. with better object/trajectory consistency) would lead to performance gain. 
4.	The authors are encouraged to evaluate the model on more long-video QA benchmarks, especially those designed to mitigate the language bias of existing long-video QA benchmarks (e.g., EgoSchema). 
5.	In Tables 2 and 3, a series of recent state-of-the-art approaches are not compared. [1][2]

[1] Juhong Min et al. MoReVQA: Exploring Modular Reasoning Models for Video Question Answering, CVPR 2024.

[2] Xiaohan Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent, ECCV 2024.
Yes
No ethics review needed.
See the fifth point of weakneses.
The proposed method seems novel and I think that makes sense for video understanding. But I think the key frame selection plays the most important role for video understanding -- it seems the authors do not propose new method on this.
This paper proposes to inject video-specific information into an LLM based framework for video understanding. Specifically, the authors utilize off-the-shelf vision tools to extract three object-centric information modalities from videos and then leverage natural language as a medium to fuse the information. I think the proposed method seems novel and interesting. But I think this paper leaks a comparison with recent papers and the performance is not good enough.
1.Figure 1 caption: “(left-right)” -> “(left-bottom)”;
2.I think the authors used both \cite{} and \citep{} in their writing;
3.Missing comparison with recent methods, for examples the publish papers in 2024. I think that is necessary to better your paper;
4.The performance seems not so good, only a marginly superiority compared to the counterparts even without comparing the recent methods. I think the performance is not good;
5.How to select the frames for most relevant frames? Is there any novelty to do that -- I think this is the most important part for video understanding.
Investigates effects of LLM strengths on Long Video QnA tasks. Introduces Multimodal Video Understanding (MVU) framework that incorporates object-centric data from pre-trained models and sets a new state-of-the-art in long-video tasks.
@inproceedings{
anonymous2024understanding,
title={Understanding Long Videos with Multimodal Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=OxKi02I29I},
note={under review}
}
Large Language Models (LLMs) have allowed recent LLM-based approaches to achieve excellent performance on long-video understanding benchmarks. We investigate how extensive world knowledge and strong reasoning skills of LLMs influence evaluations on standard long video benchmarks. Surprisingly, we discover that LLM-based models yield surprisingly good accuracy on long-video tasks with limited video information, sometimes even with no video specific information. 
Building on this, we inject video-specific object-centric information extracted from off-the-shelf pre-trained models into an LLM-based setup. We utilize natural language as a medium for fusing this information. Our resulting Multimodal Video Understanding (MVU) framework demonstrates state-of-the-art performance across long-video understanding benchmarks as well as on robotics domain tasks. Our code will be released publicly.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
long-video visual question answering interpretability
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2afa550cac702f21c0c00bdd5f2097d438752ee1.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Understanding Long Videos with Multimodal Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission