InfiniBench: A Comprehensive Benchmark for Large Multimodal Models in Very Long Video Understanding
video understanding;benchmark;long video benchmark;long video understanding
Yes
No ethics review needed.
1. Add references and discussions of related work.
2. It would be better to evaluate more long-video models (e.g., Qwen2VL) and different input frame rates (1, 8, 32, 128, and more).
3. Since most question-answer pairs are generated by GPT-4o, could this lead to inflated evaluation results for GPT-4o? Analysis is needed regarding dataset quality, hallucination rates, and potential information leakage issues.
1. The questions are comprehensive and well-structured, covering multiple dimensions and employing diverse construction strategies for different types of questions.
2. The evaluation methods are reasonable, adopting different assessment metrics for multiple-choice and open-ended questions.
This paper introduces InfiniBench, a video understanding benchmark dataset featuring the longest video duration (average 52.59 minutes per video) and the largest number of question-answer pairs (108.2K) to evaluate 9 different video understanding tasks.

The authors conducted comprehensive evaluations of existing large multimodal models (including commercial models like GPT-4V, Gemini 1.5 Flash, and open-source models). Experiments show that even leading AI models still face challenges in long video understanding, with the best models GPT-4V and Gemini 1.5 Flash achieving average accuracy rates of only 49.16% and 42.72% respectively.
1. The paper lacks discussion of related work. For example, benchmarks proposed in Video-MME, LVBench, and Long VideoBench published in June 2024 are very similar to InfiniBench.

2. Most of the question-answer pairs are generated by GPT-4o. Although multiple information sources were used as input, it's difficult to guarantee the quality of the dataset.

3. Part of the data comes from IMDB content, which likely appeared multiple times in the training corpus of LLMs used by video models, potentially leading to dataset leakage issues.
Yes
No ethics review needed.
1. How is GPT-4V's scoring aligned with human evaluation?
2. Why weren't the latest models tested, and why wasn't there comparison and discussion of the latest benchmarks?
1. InfiniBench provides a comprehensive evaluation of large multimodal models' capabilities in long video understanding through including the longest video duration and a large number of question-answer pairs, as well as designing diverse question types (multiple-choice and open-ended questions) covering nine different skills, thus thoroughly examining models' performance across multiple dimensions of long video understanding.

2. By evaluating various models including both commercial and open-source models, InfiniBench reveals the challenges and limitations of existing models in long video understanding, especially in tasks requiring deep contextual understanding and critical thinking. This in-depth assessment helps identify model deficiencies and provides clear directions for future research and model improvements.

3. InfiniBench's design not only tests models' technical capabilities but also drives models toward more human-like understanding and reasoning abilities. Through proposing human-centric questions, such as movie spoiler questions, it promotes model performance improvement in long video understanding tasks, which is significant for achieving more advanced AI applications and advancing the field of artificial intelligence.
This paper introduces InfiniBench, an innovative and comprehensive benchmark focused on evaluating large multimodal models' performance in understanding very long videos. InfiniBench is notable for its ultra-long video duration (averaging 52.59 minutes per video) and massive question-answer pairs (108.2K), covering nine different skills including multiple-choice and open-ended questions. These questions are designed to be both diverse and human-centric, with videos primarily sourced from movies and TV shows. Experimental results show that even leading AI models like GPT-4V and Gemini 1.5 Flash face significant challenges in long video understanding, achieving average accuracies of only 49.16% and 42.72%, with mean scores of 3.22 and 2.71 (out of 5) respectively. This indicates that while these models perform relatively well on local skills, they still have limitations in skills requiring global reasoning and deep contextual understanding, such as scene transitions and movie spoiler questions. Open-source models generally perform below random chance on multiple-choice questions, highlighting long-sequence global reasoning as a major challenge for existing models. Additionally, models relying on both video and text information perform poorly without caption input, emphasizing the importance of processing both visual and textual information for long video understanding. The introduction of InfiniBench aims to fill the gap in long video understanding benchmarks, drive the development of open-source large language models, and motivate multimodal large models toward more human-like long video understanding and reasoning capabilities, despite current limitations such as video source restrictions and script dependency.
1. The benchmark only uses movies and TV shows for testing, which is too limited. It should include more types of videos that show different parts of real life, like nature documentaries or home videos. The problem is that movies and TV shows follow certain storytelling patterns, so AI models might just learn these patterns instead of truly understanding the videos. They should add more casual videos like vlogs and livestreams to make the testing more realistic.

2. The benchmark needs written scripts to create its questions and answers. This is a big problem because most real-world videos don't come with scripts. Without scripts or captions, the benchmark can't test how well AI models understand regular videos that people actually watch and share online.

3. InfiniBench's testing does not cover current mainstream open-source models such as Qwen2VL, LLaVA-Onevision, and InternVL2. This makes it difficult to obtain a more comprehensive and in-depth comparison between open-source and closed-source models.

4. In Table 1, the benchmark comparison is insufficient, especially regarding some recent video benchmarks such as Video-MME and LongVideoBench. Additionally, the authors' definition of "very long" is problematic - MLVU and MovieChat have only a 3-minute gap, yet MLVU is defined as very long. This is not reasonable.
Yes
I have a concern about potential copyright infringement in this work. The proposed dataset is based on copyrighted content (video frames and subtitles of movies and TV shows) that authors have downloaded and used for experiments. The paper also includes figures of frames from TV shows. It is unclear whether the authors obtained permission from copyright owners for their use of the data. Authors do not mention whether they intend to release the dataset publicly, but if they do, this would raise further concerns.
Yes, Legal compliance (e.g., GDPR, copyright, terms of use)
Given the concerns listed above, I have doubts that this paper is suitable for publication at ICLR. I hope that authors can provide evidence to address my concerns as well as answers to the following questions.

* Could authors provide evidence of transcript quality? How accurate and complete are they? How much focus do they have on vision? Could authors provide examples?  
* Why are multiple-choice questions evaluated by asking the model to generate an answer and then using GPT to match this answer to the options? Authors state in the appendix that the reason is that models often do not follow the prescribed answer format, but from my experience at least the larger VLMs are good at following instructions about the answer format.  
* I am worried that using GPT for option matching introduces additional bias. I believe this could be measured by evaluating GPT or Gemini again by giving it the answer options in the prompt and asking it to respond with only the answer letter. Results could then be compared against the GPT-matched results.  
  * Also to the above point, did authors verify that event ordering type questions get matched correctly with GPT? These answers only differ in their ordering of options, so I am wondering whether GPT matches them correctly.  
* The benchmark was constructed using GPT, and GPT is the best performing model across all tasks. It would be interesting to quantify if there is bias towards GPT, e.g. by generating part of the data with Gemini and checking if relative model performance is consistent with the original benchmark.  
* How are copyright concerns handled? Did authors obtain permission from the copyright owners to use the video material for this purpose and to reproduce this content in a publication? If the dataset will be publicly released, how are copyright concerns handled?  
  * l. 198: “To address this limitation, we transformed the TVQA dataset from a collection of short clips into a long video dataset by gathering and sequencing the clips corresponding to each episode thereby reconstructing the full episode frames.“ How was this done and what data source was used?  
* Appendix l. 12: “The remaining two skills, i.e., local visual questions and summarizing, do not need human verification, as the first one is adopted from the TVQA dataset, and the latter is scrapped from human responses on the web.” I do not fully agree with this statement since existing benchmarks and the humans writing the summaries that were pulled from the web could still contain errors. Do authors have evidence to the quality of TVQA annotations and summaries obtained from the web?  
* How does the number of video frames provided affect the model accuracy?  
* Appendix B is quite important to understand the evaluation results presented, so I think it would be better suited to be in the main text.  
* Appendix B mentions that the benchmark videos have no audio, so video and subtitles are provided to the model separately. Does this mean that alignment between frames and subtitles is missing? Did authors measure the effect of this?  
* Could authors explain how spoiler questions are generated and provide the prompt used?  
* How does the “I don’t know” option affect results? How accurately does GPT match model answers to this option?  
* Fig. 5 (left) is redundant with Tab. 3, so one of them should be removed.  
* l. 363: The explanation of local vision and text questions is not clear. It is not explained what these questions are nor how they were generated.  
* It would be good to have random accuracy in Tab. 5 for direct comparability. Then, Tab. 4 could be omitted.  
* l. 482: “As shown in the table 5, MiniGPT4-video and LLaVA-NeXT-Interleave match lower than the random performance” What random performance is being compared to here? It would help to add this to the table as suggested above.  
* l. 482, l. 505: How can a model’s performance be lower than random?  
* l. 488: “One reason may be that eliminating the noisy information and focus on only the related information helps more in answering the questions“ How does the Goldfish model eliminate noisy information?  
* For the human verification, how were human responses on open-ended questions evaluated?

Minor points

* Tab. 1: I would not agree with the “human” checkmark for InfiniBench since questions were generated fully automatically.  
* Tab. 2 is never referenced.  
* Appendix B: It would be helpful to express this in tabular form so readers can see at a glance how many frames and what modalities were used in each model.  
* Tab. 5.: I would suggest to organize this into one big table with one column per task type. Also would be nice to visualize as a radar chart.  
* It would be helpful to annotate question types in Sec 3.2.2 and Fig. 1 with whether they are MCQ or OE.  
* It would be helpful to see a listing of modalities (vision, summary, transcript) used to generate each question.  
* Please use \\citep for citations to place citations in parentheses.  
* In tables, please right-justify numerical columns and use a consistent number of digits after the decimal point.  
* Fig. 4: The font size in these charts is very small in print. I suggest increasing it. Also I would suggest to change the pie chart into a bar chart for easier readability.  
* Fig. 5: Same concern as above about the font size.  
* l. 373: Here, the reference to Fig. 4 is repeated, but Fig. 5 is wrongly referenced. Suggest correcting this sentence to refer to Fig. 3\.  
* l. 406: Broken reference.  
* l. 413: The reference should point to Sec. B in the supplementary material.
* The presented benchmark has an impressive scale with 108.2k questions on 1,219 videos that average 52.59 minutes in length.  
* There are 9 different question types that test long video understanding models across a variety of skills.  
* The paper presents results of 8 long video models and draws interesting conclusions on their performance.  
* There is a large gap between human performance and model performance, suggesting the benchmark has ample room for improvement.  
* The paper has a good in-depth discussion of related work.
The paper proposes InfiniBench, a novel benchmark for long video understanding based on movies and TV shows. The benchmark has 108.2k question-answer pairs on 1,219 videos that average 52.59 minutes in length. The benchmark tests 9 different reasoning abilities including visual, long-context and local reasoning. This makes InfiniBench the largest-scale long video understanding benchmark to date. InfiniBench was constructed by combining and augmenting from two existing video benchmarks, TVQA and MovieNet. Most question types were generated by prompting GPT-4 with the transcript of the video while a custom pipeline was used to generate questions on changes in character appearance. The paper presents benchmark results of 8 long video understanding models, including 6 open source ones and 2 commercial ones, and discusses insights into their performance across various tasks.
* The question-answer pairs in the benchmark were generated fully automatically without any human intervention. This raises questions about soundness and of the questions and potential bias. A human evaluation is performed on a subset of the data, but good human performance is no proof that questions are well-formed and free of hallucinations.  
* Most questions are generated from transcripts that authors obtained online, but it is unclear what information these transcripts contain, whether they are complete and error-free. It is also unclear how much visual information the transcripts contain and therefore it is unclear to what degree this is a multimodal benchmark.  
* The use of movies and TV shows raises questions about generalizability. Most MLLMs likely know the plots of popular movies and shows because their summaries or transcripts were part of their training data. So, they may be able to answer the questions in the dataset without any context, which is not the case for most videos from the web. The effect of this is not examined.  
* It is unclear how much the benchmark relies on multimodal reasoning. Questions about movies and TV shows could often be answerable from subtitles alone, which are provided as context in the evaluation. It would be interesting to see an ablation that uses (1) No context, only the question itself (2) Only the question and subtitles (3) the question, subtitles and video frames.  
* The copyright implications of using movies and TV shows and possibly releasing the dataset are not discussed and raise ethical concerns.  
* Since the dataset has \~100 questions per video, it is likely that there are (near) duplicate questions. However there is no analysis of this and no mention of a filtering stage to remove duplicates.  
* There are several issues with the presentation such as redundant figures, tables that are not referenced, and wrong references. The limitations section also exceeds the 10-page limit.
Yes
No ethics review needed.
Please see weakness.
1 Important Topic. Long-form video understanding is a challenging but important problem. Hence, how to develop benchmark to evaluate this problem is critical.

2 Experiments. The experimental results are sufficient to support the claim of benchmark.
In this work, the authors propose an InfiniBench for very long video understanding. To contain local/global events and understand visual/contextual content, they define a long video understanding covering nine skills through four critical aspects in movies and tv shows.
1 Similar work has been proposed in the literature. For example,  [MoVQA: A Benchmark of Versatile Question-Answering for Long-Form Movie Understanding, arXiv:2312.04817].  Please clarify the difference. 
 
2 The writing and paper organization is not good. Please refine it for easy reading.
Yes
No ethics review needed.
Can the authors comment on the limited variety of TV shows? What about sports events like NBA, NFL, Tennis, etc. 

On GPT-4o evaluation, only 250 frames are selected. Are the 250 frames selected uniformly? Have you tried to reduce the frame size and squeeze more frames into GPT-4o?

Will all the videos be released to public? Are there any legal issues?

Are there text scripts (screenplay) associated with all the videos (movies and TV shows)?
The videos are very long with an average length 52 minutes.

The number of (question, answer) pairs is large (108k)

Some of the questions are unique such as spoiler questions, global appearance, and scene transitions. 

Compared to the existing benchmarks, this benchmark contains much longer videos and contains some new interesting types of questions. It'll be very useful to the researchers who work on long video understanding.
The paper presents a benchmark, called InfiniBench, for the evaluation of long video understanding. The dataset consists of 1219 videos. The average length of the videos is 52.59 minutes. There are 108.2K (video, question) pairs. The questions are divided into 9 categories. Some categories require the ability to make associations across a longtime span. Some categories require in-depth understanding and reasoning capabilities. It is a very interesting new benchmark for long video understanding.
The variety of TV show sources is limited since there are only 6 different TV shows.
@inproceedings{
anonymous2024infinibench,
title={InfiniBench: A Comprehensive Benchmark for Large Multimodal Models in Very Long Video Understanding},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=2D0uXQbntW},
note={under review}
}
Understanding long videos, ranging from tens of minutes to several hours, presents unique challenges in video comprehension. Despite the increasing importance of long-form video content, existing benchmarks primarily focus on shorter clips. To address this gap, we introduce InfiniBench a comprehensive benchmark for very long video understanding which presents 1)The longest video duration, averaging 52.59 minutes per video; 2) The largest number of question-answer pairs, 108.2K; 3) Diversity in questions that examine nine different skills and include both multiplechoice questions and open-ended questions; 4) Human-centric, as the video sources come from movies and daily TV shows, with specific human-level question designs such as Movie Spoiler Questions that require critical thinking and comprehensive understanding. Using InfiniBench, we comprehensively evaluate existing Large Multi-Modality Models (LMMs) on each skill, including the commercial models such as GPT-4o and Gemini 1.5 Flash and the open-source models. The evaluation shows significant challenges in our benchmark. Our findings reveal that even leading AI models like GPT-4o and Gemini 1.5 Flash face challenges in achieving high performance in long video understanding, with average accuracies of just 49.16% and 42.72%, and average scores of 3.22 and 2.71 out of 5, respectively. We hope this benchmark will stimulate the LMMs community towards long video and human-level understanding. Our benchmark can be accessed at (https://infinibench.github.io/Infinibench-website/) and will be made publicly available.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
video understanding benchmark long video benchmark long video understanding
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d969256772e8bb967c084d933943fb219d1359f7.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e4286990af860ccb3f16a8fce15b1da8635ef0b5.zip
InfiniBench: A Comprehensive Benchmark for Large Multimodal Models in Very Long Video Understanding
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission