Neptune: The Long Orbit to Benchmarking Long Video Understanding
video understanding;dataset;metric;long video understanding;benchmark
Yes
No ethics review needed.
1. As a benchmark for long videos, single-frame assessment fails to reflect the significance of temporal modeling. I believe that the sensitivity to temporal order should be evaluated by adopting methods such as reversing or shuffling the input frames.

2. I have observed that the Gemma Equivalence Metric shows similar or even inferior performance compared to direct assessment with Gemini 1.5 Pro, with an accuracy rate of only around 70%. In contrast, previous works such as Lingo-Judge[1] seem to demonstrate that fine-tuned model-based evaluation methods can achieve assessment accuracy close to human judgment. Is this due to differences in dataset complexity?

3. I have noticed that the distribution of generated QA types exhibits a clear long-tail distribution, with the largest proportion being Temporal Ordering tasks. Does this not suggest that the complexity and richness of the generated questions are insufficient? In my view, Ordering tasks belong to very simple temporal reasoning and do not meet the authors' claim of being complex questions that are difficult for humans to propose.

[1] LingoQA: Visual Question Answering for Autonomous Driving
1. The semi-automated annotation method proposed in this paper overcomes some of thechallenges associated with manual annotation, such as the creation of complex temporal reasoning questions, which can be laborious for humans. The utilization of GPT to generate these questions reduces this burden.

2. The introduction of the GEM metric addresses the need for a static, open-source evaluation metric for open-ended VideoQA, which has been a limitation in prior work.

3. The paper clearly articulates the motivation behind the Neptune dataset and the GEM metric, making it easy for readers to understand the importance of the work.
The paper introduces a new benchmark dataset called Neptune, designed to assess multimodal, high-level understanding of long videos. The key contributions of the paper are as follows:

1. Semi-Automatic Pipeline: The authors propose a scalable, semi-automatic pipeline for generating challenging question-answer-decoy sets for long videos, leveraging large video language models (VLMs) and large language models (LLMs) to automatically generate dense, time-aligned video captions and tough QAD sets for video segments up to 15 minutes long.
2. Neptune Dataset: The dataset, Neptune, covers a broad range of long video reasoning abilities and includes a subset that emphasizes multimodal reasoning. It consists of 3,268 QAD annotations for 2,405 videos.
3. Evaluation Metrics: Given the limitations of existing metrics for open-ended question answering, the authors introduce a new model-based metric called GEM (Gemma Equivalence Metric) to score open-ended responses on Neptune, offering a static and reproducible evaluation method.
4. Benchmark Evaluations: The paper presents benchmark evaluations that reveal current open-source long video models perform poorly on Neptune, especially on questions of testing temporal ordering, counting, and state changes, indicating a significant gap between open-source and proprietary models like Gemini-1.5 and GPT-4.

In summary, the paper contributes a new benchmark dataset and metric aimed at advancing the field of long video understanding through multimodal reasoning, with the goal of spurring the development of more advanced models in this area.
1. The key contributions that distinguish it from previous benchmarks are not clearly highlighted. Although the paper has made a simple comparison with EgoSchema on the impact of the number of input frames on performance, I feel that it is too simplistic. It would be constructive to see more comparisons with other existing datasets (e.g.,VideoMME) to understand how NEPTUNE's complexity and diversity align or differ from them, which could highlight the unique challenges it presents.

2. In addition to the Gemini bias discussed in the paper, I noticed that the frame captions are generated by PaLI-3. I am concerned that the hallucinations and preferences of the VLM itself (such as a preference for describing static images) may affect the quality of the
final generated QA.

3. As a benchmark for long videos, the discussion on the impact of the number of input frames on model performance is too simplistic. For example, Table 3 only discusses the performance of Gemini with 1 frame, 150 frames, and All frames. In my view, the performance with 150 frames has already reached saturation, but it is uncertain how the performance with 100 frames and 50 frames would be, that is, where the performance saturation point is.
Yes
No ethics review needed.
1. In Tables 3 and 4, it is evident that a language-only Gemini model (without ASR) even outperforms the baseline image model (BLIP2) and a short-context MLLM (Video-LLaVA). Could you provide possible reasons for this? Is it because the language models in BLIP2 and Video-LLaVA are not sufficiently robust to describe visual information?
2. In Table 3, the authors present the performance with and without ASR. The blind model (0 frame) achieves a comparable (Gemini) or even higher (VideoLLaMA2) GEM score than the model evaluated with full-frame input. Are there possible ways to mitigate the impact of ASR during dataset construction to encourage the design of vision-centric MLLMs?
1. Neptune addresses several essential question types related to temporal-aware video understanding, including the challenging temporal ordering and counting.
2.	Two subsets are introduced to comprehensively assess current multi-modal large language models.
This paper introduces a scalable, semi-automatic pipeline for constructing the Neptune benchmark dataset, aimed at evaluating long video understanding capabilities. A new evaluation metric, GEM, is proposed, demonstrating its advantages over traditional rule-based evaluation metrics.
1. In Figure 4, the authors show that EgoSchema reaches saturation at approximately 16 frames, while performance continues to increase with Neptune. This conclusion is drawn based on the powerful Gemini model; it would be beneficial to additionally include results from some open-source models (e.g., short or long context MLLMs) to better promote the development of open-source MLLMs.
2. Model names should be consistently formatted (e.g., VideoLLaMA2 vs. VideoLlaMA2).
Yes
None
No ethics review needed.
Why not compare with other benchmarks and test the latest open-source models?
How to ensure the reliability of GEM's evaluation results?
1. The paper introduces an innovative semi-automatic pipeline designed to generate question-answer-decoy (QAD) sets. This method effectively reduces the annotation costs associated with manual video captioning and question generation. By leveraging large models such as VLMs and LLMs, the pipeline automates the creation of dense, time-aligned video captions, which are then used to derive challenging QAD sets for segments of video content. This approach not only scales well but also maintains a high standard of quality in the dataset.

2. A significant contribution of this paper is the training and proposal of a model for the automatic evaluation of open-ended QA responses. The authors address the limitations of existing metrics, which are either rule-based or rely on proprietary models, by finetuning an open-source model on a generic answer equivalence dataset. This results in the GEM (Gemma Equivalence Metric), a new metric that provides a more accurate assessment of open-ended responses on the NEPTUNE dataset, thus facilitating more reliable benchmarking and comparison of different models.

3. The NEPTUNE benchmark covers a wide range of video lengths and types, ensuring diversity in the dataset. This comprehensive approach guarantees that the models evaluated on this benchmark are tested on their ability to reason over a variety of video content, going beyond the capabilities required for understanding short video clips. The inclusion of different video domains and the emphasis on longer-form content make the NEPTUNE dataset a robust platform for assessing multimodal video understanding systems across a broad spectrum of real-world scenarios.
The paper introduces NEPTUNE, a new benchmark for evaluating multimodal understanding of long videos. It addresses the limitations of current video datasets and models, which often focus on short clips and can be solved by image models applied per frame. The authors propose a semi-automatic pipeline that leverages large Video Language Models (VLMs) and Large Language Models (LLMs) to automatically generate dense, time-aligned video captions and challenging question-answer-decoy sets for video segments up to 15 minutes long. The paper also discusses the related works in video question answering, the motivation behind the NEPTUNE dataset, and the detailed pipeline for dataset creation. It concludes with the limitations of the dataset and potential areas for future improvement, such as reducing biases introduced by the models used in the creation process and enhancing the dataset with additional annotations like temporal grounding or entity labels.
1. The paper does not include an evaluation and comparison with the latest open source models such as InternVL, LLaVA-OneVision, and MiniCPM. These models are part of the current research landscape and offer a different perspective on video understanding capabilities 

2. The paper primarily focuses on the analysis of benchmarks like NextQA and EgoSchema but does not provide a thorough comparison with more recent benchmarks such as MLVU, Video-MME, and LongVideoBench, which are designed to evaluate long-form video understanding 

3. Although the semi-automatic pipeline proposed in the paper effectively reduces the workload of manual annotation, it lacks novelty in terms of a detailed analysis and comparison with other pipelines. The paper could benefit from a more in-depth exploration of the unique aspects of its pipeline and how it compares to existing methods 

4. While the proposed GEM metric may have lower evaluation costs compared to assessments using models like GPT or Gemini, the paper lacks a comparison with the consistency of human evaluations. Introducing human annotations to quantify and analyze the quality of assessments would strengthen the paper's findings
Automatic pipeline to generate data for long video understanding
@inproceedings{
anonymous2024neptune,
title={Neptune: The Long Orbit to Benchmarking Long Video Understanding},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=5ddsALwqkf},
note={under review}
}
This paper describes a semi-automatic pipeline to generate challenging question-answer-decoy sets for understanding long videos.  Many existing video datasets and models are focused on short clips (10s-30s). While some long video datasets do exist, they can often be solved by powerful image models applied per frame (and often to very few frames) in a video, and are usually manually annotated at high cost. In order to mitigate both these problems, we propose a scalable dataset creation pipeline which leverages large models (VLMs and LLMs), to automatically generate dense, time-aligned video captions, as well as tough question answer decoy sets for video segments (up to 15 minutes in length). Our dataset Neptune covers a broad range of long video reasoning abilities and consists of a subset tha temphasizes multimodal reasoning. Since existing metrics for open-ended question answering are either rule-based or may rely on proprietary models, we provide a new open source model-based metric (GEM) to score open-ended responses on Neptune. Benchmark evaluations reveal that current open-source long video models perform poorly on Neptune, particularly on questions testing temporal ordering, counting and state changes. Through Neptune, we aim to spur the development of more advanced models capable of understanding long videos.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
video understanding dataset metric long video understanding benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7eb14dc2be4d3225bf479a2953ea6255985b9e22.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/7fe0840e410ed636069ff02af38a500efb1397f1.pdf
Neptune: The Long Orbit to Benchmarking Long Video Understanding
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission