Can Multimodal Foundation Models Perform Visual Temporal Reasoning?
visual temporal reasoning;video understanding;benchmark;vision-language benchmark;video-language models;evaluation
Yes
No ethics review needed.
Audio inputs (speech) are not discussed in this benchmark. What happens if audio (speech) is included as part of video inputs to the models? Will the results on principles and and on TVBench change?
The paper is written well and easy to follow. It addresses an important area in video understanding research, where several benchmarks are known to have shortcomings in actually requiring all the frames of a video [1].
- The benchmark is manually curated in a three-stage process, potentially yielding very high-quality QA annotations
- Several models including proprietary ones lag significantly behind humans. This verifies that the benchmark is indeed difficult for existing models.
- A commendable effort is made in conceptualizing the principles defining how to check whether a benchmark requires temporal reasoning, and its shown that TVBench is way ahead of other benchmarks in adhering to these principles as shown in Tables 1-4
- Some of the ablations / analysis about existing models' capabilities are interesting, i.e. the plateauing at 8 frames compared with humans, exploiting common sense shortcuts etc.

[1]: Revealing Single Frame Bias for Video-and-Language Learning
This paper tackles the problem of benchmarking temporal reasoning using multimodal foundation models. First, the paper establishes some principles that temporal reasoning benchmarks must follow in order to test temporal reasoning capabilities, and shows that existing benchmarks fail to cater to these principles. Second, a new manually annotated benchmark, comprising six question types with some counterfactual modifications, is proposed. Third, several existing models are benchmarked demonstrating a big difference between human performance and best model's performance. And finally, some ablation studies are presented that show how one can improve models capabilities on this benchmark.
The main weakness in my opinion is that the defined principles and ways of measuring benchmarks' adherence to these principles is not very rigorous.

1: Single frame insufficiency: 
- Why is the definition limited to a single frame? What about QA pairs that can be answered in just a few frames, say 2 frames? In the abstract, it's argued that such a scenario is bad for temporal reasoning benchmark, however, this principle will consider such a question as good.
- In the second implementation, a single hand-picked frame is compared with uniformly sampled 16 frames. This seems a bit unfair to the 16 frame case. Shouldn't we compare hand-picked 16 frames with hand-picked 1 frame to get a true sense of single frame insufficiency? 

2: Frame order sensitivity:
The definition assumes that frame order sensitivity is essential for a good temporal reasoning benchmark. However, a powerful model human can easily figure out the temporal order in a lot of cases. So, I don't know if this needs to be a hard requirement.

 
3: Frame contribution parity:
Implementation-wise, again why only a single frame is considered? The proposed implementation will not give any signal for a QA pair requiring 2 frames.

For example, a benchmark that simply contains state transition questions requiring exactly 2 frames will get perfect scores on these three principles. However, such a benchmark is not very valuable. In effect, these principles can be considered necessary conditions (second one is a bit questionable), but not sufficient conditions for a benchmark to be considered as requiring temporal reasoning.

Lastly, the way of evaluating these principles using pre-existing models creates a chicken and egg problem, as the models themselves may not be capable of answering the said questions. For example, a very hard benchmark that GPT-4o cannot answer will fail all the principles if we use GPT-4o to evaluate the principles.

A couple of unrelated issues are:
- Many of the proposed question types already are part of existing works, so it’s unclear how this benchmark would be any different in satisfying the principles. Is the difficulty coming from the annotation process and counterfactuals?
- The benchmark may contain a lot of unrealistic questions because of (i) counterfactuals and (ii) simulated data that are used to artificially make the benchmark harder. These may affect the usefulness in real-world applications.
Yes
There are sourced YouTube videos in the dataset, which might have some implications in terms of privacy, security and safety.
Yes, Privacy, security and safety
1. What would happen if corresponding accuracy is zero in the denominator for the three metrics? Might need an epsilon term, or it might just be sufficient to subtract the two accuracies.
2. Why might there be a performance plateau beyond 8 frames? Is it in the design of common foundation models? Is it likely due to there being too much information to process (for example, the way the models process more than 8 images) ? Do various models show the plateau behavior exactly at 8 frames? What about at smaller frames? 
3. How does the performance of zoomed-in views differ when only tested on simulation videos with less noise?
- The paper tackles an important aspect of video understanding, which with benchmarks that are not carefully designed, might be overestimated. 
- The dataset seems to consist of well designed and curated data, as their design philosophy matches the results on the three metrics they propose. This is likely due to the rigorous quality check, and the diversity of videos (including simulations) that the authors include in the data.
- Intensive experiments and the analyses provide valuable insights into the performance of multimodal foundation models on video data.
This paper investigates whether current multimodal foundation models can perform visual temporal reasoning, which is a complex task requiring models to incorporate visual and temporal information simultaneously. The authors show that current benchmarks on video understanding may be insufficient to truly test the model's capabilities in visual temporal reasoning and introduce TVBench which consists of challenging video and question pairs that expose a significant gap between multimodal foundation models and humans, hence exhibiting limitations in current multimodal models.
- If the authors could include an experiment to test the effect of language prior - i.e. the model's accuracy when only given the text part of the dataset questions, and report the difference, that might also help reinforce the soundness of their benchmark design.
- If the authors could provide some information on the resolution and scale of the videos, that might be helpful especially because the authors discuss models not benefiting from more than 8 frames.
Yes
No ethics review needed.
* The author seem not mention it in the paper - will the benchmark be made publicly available at some point?
* The author selectively evaluate some proprietary MFMs in Table5. It would be interesting to report the results of different model sizes in the same model family  (e.g. Gemini 1.5 pro v.s. Gemini 1.5 flash; Claude 3.5 Sonnet v.s. Opus) to understand the scaling effect of current SOTA MFMs on the proposed benchmarks.
The paper is easy to follow and the key concepts are delivered clearly. The authors start by setting three principles and studied the existing video benchmarks around these three principles quantitatively, revealing the opportunities and curating datasets around them, I think the approach to the problem and processes are rigorous and effective. Their final results on evaluating MFMs on their datasets also show big opportunties for current system to improve in terms of temporal reasoning.
This paper present a new benchmark for visual temporal reasoning. They start by defining three principles for video temporal understanding, namely SINGLE FRAME INSUFFICIENCY, FRAME ORDER SENSITIVITY and  FRAME CONTRIBUTION PARITY. Three principles are then quantified by a metric to reflect the headroom provided by indivisual benchmarks. Finally, a new new benchmark named TVBench with 1484 carefully curated examples are presented and a diverse set of general purpose MFMs are benchmarked against TVBench.

The paper is easy to follow and the key concepts are delivered sound and clear. I appreciate the details the authors provided in the appendix about the analysis and the dataset.
I have some concerns and questions for discussion.

* Clarification on Table 6. It is not very clear to me what "True" & "False" stand for in the table. More explanation and details on the experiment setups are needed. This leads to the question of how the conclusion "More Capable Models Are More Likely to Exploit Shortcuts Through Common Sense." is drawn. Does "True" and "False" mean different split of datasets? If so, those videos and their QAs are different, how could we compare their relative performance drop and reach the conclusion?

* L479 "Models Excel in First-Person Over Third-Person Perspective Temporal Reasoning Video Understanding." The conclusion of model excels FPV over TPV videos on comparing experiment results of 88 FPV QA v.s. 668 TPV QA. Since these are two set of different videos and QAs, it is hard to compare the final score side by side and draw the above conclusion in my opinion. More rigorous study or careful experiment design is needed, for instance, creating a matched set of FPV and TPV videos covering the same scenarios would make more sense.

* References to the visual temporal reasoning principles are needed. Some principles discussed in the manuscript for temporal reasoning in video understandings have been studied and revealed in previous literatures, e.g. "Single Frame Insufficiency" has been discussed in [1]; "Frame Order Sensitivity" has been explored/discussed in many literature, e.g. [2]; [3][4] studied "FRAME CONTRIBUTION PARITY" as well. Having a brief literature review section for each principle, and discuss the difference and contributions of current work could strengthen the manuscript. 

[1] Lei, J., Berg, T.L. and Bansal, M., 2022. Revealing single frame bias for video-and-language learning. arXiv preprint arXiv:2206.03428.

[2] Qian, R., Li, Y., Yuan, L., Gong, B., Liu, T., Brown, M., Belongie, S., Yang, M.H., Adam, H. and Cui, Y., 2021. Exploring temporal granularity in self-supervised video representation learning. arXiv preprint arXiv:2112.04480.

[3] Huang, De-An, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles. "What makes a video a video: Analyzing temporal information in video understanding models and datasets." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7366-7375. 2018.

[4] Liu, X., Pintea, S.L., Nejadasl, F.K., Booij, O. and Van Gemert, J.C., 2021. No frame left behind: Full video action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14892-14901).
Yes
No ethics review needed.
- Will the benchmark be publicly available?

- How do you generate the queries for each task in the benchmark? Please include this information in the main paper, which may be inspiring to the readers.

- Why not reporting the results of open-source models after fine-tuning them on the temporal reasoning questions. e.g, a training set of the proposed benchmark? It is possible that the model learns to reason from dynamics in the video after fine-tuning. Please justify this.
- Well-Established Benchmark: The benchmark is rigorously constructed with three quantitative principles that emphasize the complexity of temporal reasoning tasks, ensuring a challenging and relevant evaluation.

- Extensive and Comprehensive Coverage: The benchmark includes a wide range of tasks and scenarios, providing thorough coverage of diverse temporal reasoning challenges, which enhances the evaluation scope and relevance.

- In-Depth SOTA Model Comparison: The paper presents an extensive and detailed comparison of state-of-the-art video foundation models, providing valuable insights into their capabilities and limitations for video question answering.

- Clear and Cohesive Writing: The paper is well-written, presenting its ideas and findings in a clear and organized manner, making it accessible and informative for a broad audience.
The paper introduces a novel video benchmark specifically designed to assess the temporal reasoning capabilities of multimodal foundation models. The benchmark is thoughtfully formulated, encompassing a wide range of dynamic and continuous reasoning scenarios over time. By evaluating state-of-the-art video foundation models, the paper highlights fundamental challenges these models face in Video Question Answering (VideoQA). This work provides valuable insights into the limitations of current multimodal models in handling temporal visual tasks, promising to advance research in both the video and large language model (LLM) communities. However, despite these contributions, I still have some concerns and will adjust my rating depending on whether they can be further addressed.
- Sensitivity in Mathematical Formulation: The mathematical formulation of the three principles appears sensitive to certain parameters, particularly the choice of human-picked frames and the number of input frames. For instance, using 16 frames as the multiple-frame input seems arbitrary; different frame counts could significantly impact evaluation scores and model comparison across benchmarks. This choice requires further justification, along with ablation studies on multi-frame input numbers within the three principles, to confirm robustness and fairness.

- Data Imbalance Across Tasks: The sample sizes across the six tasks are imbalanced, with some categories, like Visual Cues, containing only 70 questions—significantly fewer than other categories. This imbalance could impact the benchmark's representativeness and reliability. Addressing this imbalance would improve the benchmark's overall consistency and rigor.

- Fairness in Benchmark Comparisons: Comparisons with other benchmarks may not be entirely fair due to the random sampling of 200 QAs for evaluations. This approach may inadvertently filter out complex temporal reasoning questions. To ensure fairness and maintain the benchmark's integrity, evaluating all QAs from other benchmarks would be preferable.

- Insufficient Text-Only Baseline in Table 5: The current text-only baseline presented in Table 5 may not fully capture the model's reasoning capabilities. Converting video content into textual descriptions for input into a model like GPT-4 would offer a more meaningful comparison with visual inputs. Including this enhanced baseline would provide deeper insights and enrich the discussion in Table 5.
@inproceedings{
anonymous2024can,
title={Can Multimodal Foundation Models Perform Visual Temporal Reasoning?},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=fCi4o83Mfs},
note={under review}
}
State-of-the-art Multimodal Foundation Models (MFMs) typically interpret videos as sequences of individual frames, and achieve relatively high accuracy on existing temporal reasoning video benchmarks. However, are these models truly comprehending the videos as continuous sequences, or just isolated single frames? To our surprise, many questions posed by existing temporal reasoning video benchmarks can be accurately answered using a single, a few, or even out-of-order frames. This suggests that the visual temporal reasoning capabilities of current models are overestimated. To address this limitation, we introduce TVBench, a novel benchmark crafted to rigorously assess MFMs' temporal reasoning capabilities in video understanding. Unlike prior benchmarks, our benchmark strictly demands reasoning about the transitions across all frames by focusing on dynamic, continuous processes that unfold over time. TVBench comprises 1,484 carefully curated, human-annotated questions spanning six tasks (i.e. action count, direction, rotation, shape&trend, velocity&frequency, and visual cues), applied to 1,417 videos that encompass human-centric, real-world, and simulated scenarios. We present a comprehensive evaluation of 23 MFMs, including both open-source and proprietary models. Our benchmark exposes a more significant performance gap between human-level and MFM-enabled temporal reasoning video understanding than was previously known. This performance gap highlights the substantial challenges posed by the benchmark. Furthermore, we provide a thorough analysis across different subsets of evaluation results, highlighting key research challenges and potential directions for future MFMs. We believe TVBench will serve as a crucial testbed for evaluating the next-generation MFMs and as a call to the community to develop AI agents capable of comprehending the human world dynamics through the video modality.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
visual temporal reasoning video understanding benchmark vision-language benchmark video-language models evaluation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/afd8452b417b380a0460f76c7bd0140f52484f43.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Can Multimodal Foundation Models Perform Visual Temporal Reasoning?
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission