Unhackable Temporal Reward for Scalable Video MLLMs
Video MLLMs; Temporal hacking; Temporal Perplexity
Yes
No ethics review needed.
1. How is equation 6 used in loss function?
2. For equation 6, why using T:T instead of T-t:T for the second term?
3. Is that possible to get the TPL of your baselines comparing with Video-UTO?
1.  The paper introduces temporal hacking as a new perspective on reward hacking in RL and effectively adapts this concept to video MLLMs.
2. TPL score is a notable contribution, providing a quantitative way to assess temporal misalignment, a challenge often overlooked in video modeling.
3. The approach shows promise for scalability with additional data, making it versatile for future applications and datasets.
4. The results are good comparing to various larger models.
This paper addresses a critical problem in MLLMs for video processing: "temporal hacking," where models simplify the task by focusing on select frames, missing broader context. The authors propose a framework called UTR that mitigates temporal hacking by emphasizing consistent frame activation across the entire video, leading to more accurate video comprehension. This framework incorporates TPL to quantify misalignment in temporal modeling and guides the model through two key principles: high frame information density and inter-frame information dynamics.
1. UTR relies on high-density, high-dynamic frame descriptions. Several datasets is lack of this, I am wondering how are the other methods performance could be improved after simply data modeling.
2. The ablation study is confusing, does table 3 "-Data Modeling" means no data modeling and no task modeling or just no data modeling? 
The comparison data scale is different which is unfair to compare.
3. For ablation result, the Video-UTO is not the best without Data Modeling on MMVet and also UTR-Data size of 180K results are sometimes worse than 0K is also confusing.
Yes
No ethics review needed.
1. There is an inconsistency between Table 1 and the text in Line 376 regarding the data scale used for Video-UTR. Table 1 lists the data scale as 1.1M, whereas Line 376 claims the performance is achieved with approximately 700K video data points. Which number is correct?
1. The proposed $\mathcal{T}\_{tpl}$ is interesting and correlates well with the model performance.

2. This paper systematically analyzes reward hacking from the RL perspective and proposes two guiding principles for designing unhackable rewards.

3. The proposed method Video-UTR outperforms several baseline methods on both video and image understanding tasks.
This paper thoroughly analyzes the phenomenon of temporal hacking, where video MLLM learns to spuriously correlate its answer to query prompts to a subset of selected frames without considering the whole video context. The author introduces temporal perplexity (TPL) to assess this misalignment, showing a good correlation with model performance. Moreover, the author proposes the Unhackable Temporal Rewarding (UTR) frameworks and empirically demonstrates their effectiveness by comparing their methods with several open-sourced MLLM.
- **Over-complicated Presentation**: I feel the presentation of the paper is unnecessarily complicated and does not explain the details sufficiently. The paper spends too much space introducing the concept of **temporal hacking**, which can be efficiently summarized as *existing training data and objectives make the learned MLLM find shortcuts when answering the query prompt by spuriously correlating the answers with only a subset of video frames, ignoring the whole video frame context.* In contrast, the author is supposed to provide more details of their proposed methods in Section 3.2 instead of putting them in the Appendix.

- **Lack of detailed description of the proposed TPL $\mathcal{T}\_{tpl}$**: Equation 6 defines $\mathcal{T}\_{tpl}$, which requires query the function $\mathcal{R}\_{ppl}$. However, the paper does not provide details about $\mathcal{R}\_{ppl}$ other than citing the original paper developed the $\mathcal{R}\_{ppl}$. For example, does calculating the $\mathcal{R}\_{ppl}$ require access to a specific MLLM?

- **Empirical results**: comparison between the proposed Video-UTR and LLaVA-OneVision is insufficient. The paper only reports LLaVA-OneVision’s results on 3 out of 7 benchmarks, where it outperforms Video-UTR on 2 out of those 3. Besides, the authors should underline PLLaVA's accuracy for ANet-QA rather than underlining their Video-UTR. Additionally, the authors should include results for MLLMs that are also finetuned from QWen2-7B in Table 2. Without these results, it is difficult to assess the source of the performance improvements.

- **Regarding the RL perspective**: Reward hacking is not exclusive to reinforcement learning, as RL is just one approach to maximizing a reward function. In the field of aligning pretrained visual generation models, several methods optimize toward a reward model by directly backpropagating gradients from a differentiable reward model. These approaches are also susceptible to reward hacking, as discussed in [1, 2]. Reward hacking generally occurs when the learning objective fails to fully capture the intended model behavior. I recommend revising this statement and including a citation for [1,2] to provide a broader context.

Minor:

- **Lack of detailed figure captions**: The authors should thoroughly explain each figure within the main text rather than leaving readers to infer their meanings.

- **Missing citations**: Various benchmarks have recently been proposed to evaluate MLLM's video understanding capabilities, e.g., [3, 4]. The authors should consider evaluating their methods on these benchmarks and cite them in the related works sections.

Overall, the presentation of the paper can be significantly improved.

[1] Li et al., Reward Guided Latent Consistency Distillation. TMLR 2024

[2] Zhang et al., Large-scale Reinforcement Learning for Diffusion Models.

[3] Fang et al., MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding. NeurIPS 2024

[4] He et al., MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos.
Yes
No ethics review needed.
1. Could the authors provide more detailed explanations of the TPL score calculation and its computational overhead? How does it scale with longer video sequences?
2. How does the computational complexity of UTR compare to existing video MLLM approaches? Is the additional overhead from attribute extraction justified by the performance gains?
3. Have the authors explored how the quality and computational cost of expert models affect the final performance? Is there a minimum accuracy threshold needed for these models, and how does their use impact the overall training time?
4. Are there any limitations or potential drawbacks to the UTR approach in scenarios with highly redundant or static video content?
5. Could the authors provide more details about how TPL correlates with human judgment of video understanding quality?
6. Can the framework be applied to other domains involving sequential data, such as speech or time-series analysis?
- **Originality**: The paper introduces the novel concept of temporal hacking in video MLLMs and frames it within a reinforcement learning context, offering a fresh perspective on the anti-scaling issue.
- **Theoretical foundation**: The reinforcement learning perspective provides clear insights into the problem and guides the development of the solution.
- **Practical metric**: The introduction of the Temporal Perplexity (TPL) score provides a quantifiable measure of temporal misalignment, which correlates with model performance and can guide future research.
- **Proposed solution**: The Unhackable Temporal Rewarding (UTR) framework is a constructive approach to mitigate temporal hacking, showing significant improvements in video comprehension tasks.
- **Comprehensive evaluation**: The paper presents extensive experiments across multiple benchmarks and includes detailed ablation studies, demonstrating the effectiveness of the proposed methods.
- **Practical impact**: The proposed solution is implementable and shows strong empirical results, indicating its potential applicability in real-world video MLLMs.
The paper addresses a paradox observed in training video-processing Multimodal Large Language Models (MLLMs), termed the "anti-scaling law," where increasing data and model size leads to decreased performance. The authors identify "temporal hacking" as the culprit—a phenomenon where models focus on select frames instead of capturing the full video narrative, effectively shortcutting temporal understanding. I believe this behaviour is analogous to posterior collapse in VAEs, where models find ways to ignore part of the input while still optimizing the objective, suggesting a fundamental challenge in enforcing comprehensive information utilization in deep learning models. The authors present a theoretical framework from a reinforcement learning perspective to explain temporal hacking and introduce the Temporal Perplexity (TPL) score as a metric to assess misalignment in temporal modelling. To mitigate temporal hacking, they propose the Unhackable Temporal Rewarding (UTR) framework, which enhances video comprehension by aligning proxy rewards with true objectives through spatiotemporal attribute extraction and bidirectional querying. This solution addresses the root cause: the misalignment between training objectives and desired behaviour in video understanding. Empirical results show that TPL correlates strongly with temporal modelling quality and that UTR significantly improves video MLLMs' performance across multiple benchmarks, while also demonstrating scalability with increased data and frame lengths.
- **Clarity and organization**: My main issue is that the paper's presentation suffers from unclear explanations and a lack of cohesion in some sections. Technical terms are sometimes introduced without sufficient context, making it hard to follow, at least to me. I even found Figure 1, which aims to provide a clear illustration of the phenomenon, relatively unclear. 
- **Limited discussion of computational overhead**: While the supplementary material provides some training details, the paper lacks a thorough analysis of the computational cost introduced by the UTR framework, particularly regarding the extraction of spatiotemporal attributes using expert models. This makes it difficult to assess the practicality and scalability of the approach compared to existing methods.
- **Dependency on expert models**: The method relies heavily on the accuracy of expert models for attribute extraction, which may limit its applicability if these models are not sufficiently accurate or if they introduce significant computational overhead.
- **Theoretical depth**: The reinforcement learning framework and the concept of temporal hacking could be more rigorously developed. Definitions and theoretical justifications need strengthening.
- **Related work**: There is a lack of thorough comparison with existing methods addressing similar issues, which limits understanding of the paper's novelty and significance in the broader context.
- **Limited exploration of failure cases**: The paper could benefit from more analysis of scenarios where UTR might not perform well, such as with highly redundant or static video content.
- **Reproducibility**: Details on experimental settings, such as hyperparameters, data preprocessing, and model configurations, are insufficient, hindering the ability to replicate results.
Yes
No ethics review needed.
Please see weaknesses above.
The paper is well-written, with the key contributions listed clearly in the introduction. The authors also identified relevant literature in this line of research and highlighted their contributions in the paper. The authors introduce very fundamental principles that should be used to design a video captioning metric, which are high frame information density and high inter-frame information dynamics. They construct a metric by extracting the location, appearance, and action information of subjects in a frame using pre-trained models. They construct datasets based on 3 open-source video datasets, from which they extract subject attributions from each frame in the video.
The paper introduces Unhackable Temporal Rewarding (UTR), a framework to improve multimodal large language models (MLLMs) by addressing a phenomenon called "temporal hacking". This phenomenon occurs when MLLMs, which process video frames sequentially to understand narratives, exploit shortcuts by focusing on only a few frames (sometimes just the first and the last), thereby missing the entire context. This issue has led to an anti-scaling paradox where increasing model size or data volume degrades performance. The authors propose UTR to counteract this problem by enforcing comprehensive temporal processing, ensuring the model engages with the full video content.
1. Since the proposed metric relies on subject attributions extracted from pre-trained models, the errors from those models might propagate into the extracted metadata and subsequent make the metric unreliable. Can the authors comment on whether this is a viable concern?
2. Have the authors measured the reliability of extracted metrics in the presence of multiple subjects in one frame?
Unhackable video-langauge modeling for existing video MLLM.
@inproceedings{
anonymous2024unhackable,
title={Unhackable Temporal Reward for Scalable Video {MLLM}s},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Gf1uBeuUJW},
note={under review}
}
In the pursuit of superior video-processing MLLMs, we have encountered a perplexing paradox: the “anti-scaling law”, where more data and larger models lead to worse performance. This study unmasks the culprit: “temporal hacking”, a phenomenon where models shortcut by fixating on select frames, missing the full video narrative. In this work, we systematically establish a comprehensive theory of temporal hacking, defining it from a reinforcement learning perspective, introducing the Temporal Perplexity (TPL) score to assess this misalignment, and proposing the Unhackable Temporal Rewarding (UTR) framework to mitigate the temporal hacking. Both theoretically and empirically, TPL proves to be a reliable indicator of temporal modeling quality, correlating strongly with frame activation patterns. Extensive experiments reveal that UTR not only counters temporal hacking but significantly elevates video comprehension capabilities. This work not only advances video-AI systems but also illuminates the critical importance of aligning proxy rewards with true objectives in MLLM development.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Video MLLMs; Temporal hacking; Temporal Perplexity
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4a9c7bf084713ca1d5381b352b824fdac2bd481e.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Unhackable Temporal Reward for Scalable Video MLLMs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission