MSR-ViR: Modularized Self-reflected Video Reasoner for Video Question Answering
Video Question Answering;Multimodal LLM;Modular Network;Self-reflected Training
Yes
None
No ethics review needed.
To generate reasoning paths for VideoQA, would it be more effective to design a Chain-of-Thought dataset and perform supervised fine-tuning (SFT)? The O1 model currently adopts this approach, achieving clear reasoning paths through an end-to-end structure.
1. The MoST-Grounding module introduces two standard modules—temporal localizer and spatial localizer—which can be flexibly assembled in sequence based on the question parser. This structure is robust and allows for reliable generation of reasoning paths.

2. The authors present a clear motivation: to create a framework for generating reasoning paths for the black-box nature of VideoQA tasks. The comprehensive visualization of reasoning paths further demonstrates the effectiveness of the model.
The paper addresses the use of multimodal Large Language Models in understanding tasks across various multimodal scenarios, specifically focusing on applications in Video Question Answering. However, current Multimodal Large Language Models are largely black-box systems for VideoQA tasks, lacking the ability to provide an understandable reasoning path and, thus, suffer from limited interpretability. To address this, the authors propose MSR-ViR, which constructs a modular reasoning structure designed to generate reasoning paths, and incorporates a reinforcement learning framework to prevent the model from generating unreasonable reasoning paths.

While the proposed approach is interesting, it relies on the integration of four existing models. This ensemble-based structure shows only marginal performance improvements (1-2%), and the manuscript does not discuss crucial aspects such as reasoning time costs or memory overhead.
1. The question parser and LLM reasoning components both rely on LLM structures, leading to high computational costs.

2. Both the temporal localizer and spatial localizer use existing models, specifically UniVTG and YOLO-World, which contribute to significant parameter overhead. As the complexity of VideoQA tasks increases, relying on these two models alone may not only limit predictive accuracy but also compromise the completeness of reasoning paths. Future work may need to explore additional modules to support diverse combinations (see [1] for reference).

3. The ablation study lacks comprehensiveness. While the authors assess model performance on QA and grounding tasks and provide an effectiveness analysis of each module, they do not evaluate inference speed, parameter count, or other metrics compared to end-to-end models. Given that the proposed framework integrates multiple existing large models, an analysis of inference speed is both important and currently missing.

[1]. Neural Module Networks
Yes
No ethics review needed.
see weaknesses.
1.	Successfully develop a ground-then-answer framework for interpretable video question parsing. The question parser policy is able to be optimized via answer feedback. 

2.	The approach is well presented and easy to understand.
This paper aims to improve the interpretability of Multimodal LLMs in performing VideoQA. To achieve the goal, the authors design a modular self-reflection framework MSR-ViR. The framework primarily comprises a spatial-temporal grounding module and a self-refection learning mechanism based on DPO. MSR-ViR basically decouples video grounding from videoqa, enabling the interpretaion of intermediate results to understand the answers. The experiments on related datasets have demonstrated the strength of the approach in both accuracy and interpretability (grounded accuracy).
1.	The paper does not improve video grounding but just uses existing method UniVTG. According to table 2, the grounding performance in terms of IoP@0.5 is worse than previous VLMs (VGT and TempCLIP). This severely limits the improvements of QA performance. 
2.	According to the model ablation results in Table 3,  the global representation g_v (which opens back door for grounded QA) seems more crucial than other components. Such results slightly depart from the major claim of interpretable VQA where correct answers are anchored on correct visual content.
3.	Should compare with SeViLA which also finetunes a localizer on QVHighlight (like UniVTG) for grounded VideoQA.
Yes
No ethics review needed.
See Weaknesses
1) The paper is well written and the motivation is clear, with a strong focus on the interpretability challenge in VideoQA, making it highly relevant to the field.
2) The MoST-Grounding module integrates multiple submodules to effectively localize temporal and spatial regions, improving transparency in the input to the MLLM.
3) The Alternate Self-Reflection strategy introduces a novel reinforcement-based method to align the question parser and the MLLM, enhancing performance and consistency.
This paper addresses the black-box problem of multimodal large language models (MLLMs) in VideoQA by proposing MSR-ViR, a novel framework. MSR-ViR introduces two core components: (1) the MoST-Grounding module, which localizes relevant temporal segments and spatial regions in videos, and (2) an Alternate Self-Reflection Training Strategy, which iteratively enhances the question parser and the MLLM. Evaluations on datasets such as NExT-QA, STAR, and NExT-GQA demonstrate that MSR-ViR achieves competitive performance on VideoQA tasks and improves interpretability by providing visual evidence for answers.
1) The framework is relatively heavy, relying on multiple external tools for tasks and additional operations such as resizing and truncation, which increases computational overhead. Moreover, what if these external tools are unreliable, which can lead to further exposure bias? It's necessary to further investigate the choice of the sub-modules in the MoST-Grounding module.
2) While the approach improves the selection of input information, it does not make the internal reasoning process of the MLLM more interpretable. It still focuses on the process of 'input' to decide which information should be fed into the MLLM as soft prompts. 
3) The paper misses references with related works such as SeViLa [1] and GCG [2], which also focus on VideoQA with grounding elements. Including these baselines would strengthen the empirical validation.

[1] Yu et al. "Self-Chained Image-Language Model for Video Localization and Question Answering", 2023 NIPS
[2] Wang et al. "Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering", 2024 ACM MM
@inproceedings{
anonymous2024msrvir,
title={{MSR}-ViR: Modularized Self-reflected Video Reasoner for Video Question Answering},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=4Sv5MQ931E},
note={under review}
}
Recently, multimodal large language models (multimodal LLMs) have been applied to a wide range of video understanding tasks, particularly for Video Question Answering (VideoQA). However, existing multimodal LLMs suffer from the following challenge: the classic end-to-end training strategies of multimodal LLMs for VideoQA tasks are black-box, thus lacking interpretability as they can neither present a reasoning path nor indicate where the answer is derived from the video. To tackle this challenge, we propose MSR-ViR (Modularized Self-Reflected Video Reasoner), a self-reflected framework that introduces a Modularized Spatial-Temporal Grounding (MoST-Grounding) module to multimodal LLMs for VideoQA tasks. MoST-Grounding utilizes a question parser LLM to generate execution policies, which serve as a reasoning path from questions to answers providing interpretability for our VideoQA framework. Based on the execution policies, MoST-Grounding invokes various small modules to localize temporal segments and spatial regions in videos which provide multimodal LLMs with most relevant visual information, while presenting visual evidence of our final answers. To avoid the question parser LLM generating unreasonable policies, we further propose a reinforcement learning-based Alternate Self-reflection training strategy to optimize the Multimodal LLM and the question parser LLM. Experiments on VideoQA datasets (NExT-QA and STAR) and grounded VideoQA dataset (NExT-GQA) demonstrate that our method significantly improves video understanding capabilities of multimodal LLMs, while providing interpretable reasoning paths together with temporal and spatial localization evidence within the video.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Video Question Answering Multimodal LLM Modular Network Self-reflected Training
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/bc2578be11dcf758ee1e9d5318f4727faf706324.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
MSR-ViR: Modularized Self-reflected Video Reasoner for Video Question Answering
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission