Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge
Video MLLM; Streaming Video Understanding
Yes
No ethics review needed.
Please refer to the part of weaknesses.
Nice paper. This work is comprehensive, provides significant contributions, and is well-written.

The biggest contribution is StreamBench. Previously, there was no suitable benchmark for streaming video understanding. StreamBench is a high-quality addition.

Furthermore, StreamChat is also very interesting, demonstrating that an agent-like system can also achieve state-of-the-art results in streaming video understanding.
This paper introduces StreamChat, a training-free framework for streaming video reasoning and conversational interaction, and StreamBench, a comprehensive benchmark for evaluating streaming video understanding across various media and interactive scenarios. Extensive evaluations on StreamBench and other public benchmarks show that StreamChat significantly outperforms existing state-of-the-art models in accuracy and response times, highlighting its effectiveness for streaming video understanding.
Overall, the paper is satisfactory, but I have some concerns regarding StreamChat:

- What motivated the selection of LongVA as a foundation? What are the essential capabilities required for other Video-LLMs to serve as the base model?

- Would fine-tuning LongVA with some streaming vision-language data be better than the current system?

- Is it feasible to integrate the proposed hierarchical memory storage into a trainable framework?
Yes
No ethics review needed.
- Do you plan to make this benchmark publicly available?
- Do you plan to open source your method so that others may benefit from the engineering effort you've put into your method? (I am of course talking about academic researchers)
- What the process of identifying the various categories STREAMBENCH encompasses? I am of course talking about the domains and classes of videos. 
- In the appendix, you very briefly mention that you're using a LLM to determine accuracy. The formula you present is unclear. (What's N, T?). What are you getting out of the LLaMA model? Also, do you use any particular prompt for the LLaMA model?
-
Strong empirical results. This is a systems pape which is tied to a particular benchmark which was designed for it, so it's unsurprising that the proposed method works best. With that said, given the engineering effort put into it, it still is impressive, particularly because of the fact that this system could be used in a realtime scenario. 

I am personally excited by the new benchmark that is proposed in this paper perhaps even more so than the method, because it captures a whole class of interactions which were under-represented before.
This paper introduces a method for online interactions with videos through an LLM, and a benchmark suitable for evaluating streaming-style scenarios where the dialogue happens in real time with the video.

Contributions:
- Video-based LLM which processes high framerate video (typically LLMs need to subsample severely the video due to computational constraints). This method uses optical flow magnitude to assess whether to process any particular frame (w.r.t. previous), therefore saving considerable amount of computation needed. The LLM also has a "short/long" term memory system.
- New benchmark that should be useful since it's one of a kind, simulating multi-round interactions centered on the video content. 25h of content was used in the creation of this benchmarks, making it one of the larger ones available.
I am kind of baffled by the language of "our method can further improve our foundation model LongVA [20] by 5.1% in accuracy". Does this imply that you're the authors of this method? (Please don't answer this question, but please be mindful of the language which might accidentally de-anonimize you).

The hierarchical design seems a bit ad-hoc. It would have been nice to spend a bit more time to motivate the various design choices. Currently, it sounds like the paper is in the "this is what we did" style, without spending too much into the motivation which led to the various choices made. I could easily imagine many other ways in which the hierarchical memory could have been implemented.

The abstract claims "compression" of video features, but reading the paper reveals that this is actually not at all the case. I find the abstract to be misleading and should be updated to reflect the reality of the methods involved.

Minor comments:
- I am having a hard time reading figures. For example, in Figure 1 I had to use a color picker to figure out which method was which line. In the interest of color blind people, (or partially color blind people like me), please use symbols (*, triangle, diamond, etc) or line styles in addition to colors to make the figures more accessible (either to color-challenged people like me, or when printed out in grayscale).
- There's a typo "Gine-grained" in the appendix.
Yes
No ethics review needed.
See weaknesses. I am just at the borderline due to 1) the contribution of StreamBench and StreamChat, 2) the over-complicated design of StreamChat (being too engineering and RAG-like).
1. The proposed benchmark, StreamBench, is a comprehensive benchmark testing various capabilities of real-time video LLMs. As far as I know, this is the first benchmark designed for online streaming video task evaluation.
2. The proposed method, StreamChat, integrates frame selection, memory building and retrieval, and several optimizations into a video LLM in a training-free manner. Though designed complicatedly, the latency of StreamChat is still under control (under one second). Several benchmarks, including online StreamBench and offline video understanding tasks, show the effectiveness of the proposed StreamChat. The authors also put a detailed inference example, every components illustrated, in Fig 6. Also, the detailed ablations on threshold and memory parameters should be appreciated.
This paper presents a benchmark, named StreamBench, and as well a model, named StreamChat, for real-time streaming video LLM answering. The StreamBench comprehensively evaluates assorted capabilities of video LLMs, including spatial search, long & short memory recall, common knowledge reasoning, etc. The size of StreamBench is also larger than previously-proposed benchmarks. The authors also paid a lot of efforts in building the training-free StreamChat model, which bases on current video LLM LongVA and consists of novel techniques including hierarchical memory storage, optimized system scheduling. Evaluations on various benchmarks show that StreamChat enjoys better performance with lower latency compared to previous streaming models.
My major concerns comes from two sides:
1. **The presentation**. The presentation of StreamBench is good but the part of StreamChat seems to need to be further improved. It is hard to tell what StreamBench is designed to do when understanding Fig 4 and Fig 5. Though there are both captions and texts written in the method section, it is still hard to relate text descriptions to components in the figures (especially for ones less familar with memory mechanism and RAG).
2. **The complexity**. StreamChat seems too complicated: there are selective frame stacking based on motion vectors, vision-based short-term memory, caption-based long-term memory (also a clustering mechanism in it), dialogue memory (whose language encoder is different from video LLM), and FAISS indexing library. Although being tuning-free, all these components make this 'system' super heavy. The authors should justify every components used in building the memory system in StreamChat. Moreover, this looks more like a RAG system specifically designed for online videos. The authors should compare with other basic RAG methods.
Yes
No ethics review needed.
Q1: Will the example in the prompt impact the models' prediction? For example, your example in the prompt is "{'pred': 'A'}". Does it increase the probability of predicting "A"?

Q2: The StreamBench benchmark supports only QA tasks now. Are there any plans to extend it to include additional streaming video understanding tasks, such as dense streaming video captioning or temporal/spatiotemporal grounding?
S1: The paper introduces a novel hierarchical memory mechanism that compresses video representations over long sequences, enabling efficient video feature retrieval in real-time multi-turn dialogue contexts.

S2: I believe the introduction of StreamBench as a benchmark for streaming video understanding is a significant contribution that can fill a critical gap in the field. By offering a standardized evaluation framework, StreamBench enables more rigorous comparisons across models, driving advancements in streaming video understanding.

S3: The proposed StreamChat achieves the best performance on StreamBench, surpassing other state-of-the-art methods, which is impressive.
In this work, the authors propose a training-free framework, StreamChat, for streaming video understanding. StreamChat uses a hierarchical memory system to efficiently process long video sequences for real-time, multi-turn dialogue. With parallel scheduling, it improves speed and reduces latency. 

The authors also introduce a new benchmark, StreamBench, which tests video understanding across diverse scenarios, and results show StreamChat outperforms current models in accuracy and response time.
W1: StreamChat was compared only with several open-source models. Including proprietary models like GPT-4o and Gemini-1.5 in the StreamBench evaluation would provide a more comprehensive comparison. 

W2: Human performance is absent in the comparison, which would help illustrate the gap between current models and human capabilities if included.

W3: The authors compare their new benchmark only with older datasets or benchmarks, such as MSVD, MSRVTT, and ActivityNet. It would strengthen the evaluation to include comparisons with more recent benchmarks for video understanding, such as Seed-Bench [1], Video-Bench [2], MVBench [3], and LVBench [4].

W4: The paper lacks an analysis of factors that impact performance on StreamBench, such as the input frame sequence length or the language model size used by the models.

[1] Li, Bohao, et al. "Seed-bench: Benchmarking multimodal llms with generative comprehension." arXiv preprint arXiv:2307.16125 (2023).

[2] Ning, Munan, et al. "Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models." arXiv preprint arXiv:2311.16103 (2023).

[3] Li, Kunchang, et al. "Mvbench: A comprehensive multi-modal video understanding benchmark." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Wang, Weihan, et al. "LVBench: An Extreme Long Video Understanding Benchmark." arXiv preprint arXiv:2406.08035 (2024).
Yes
No ethics review needed.
Please respond to Weaknesses.
1. The proposed StreamBench may be the first benchmark for streaming video understanding.
2. The proposed StreamChat outperforms the selected baselines on StreamBench.
3. The processing speed of StreamChat significantly outperforms those of baselines.
This paper studies streaming video understanding. The authors propose a new benchmark named StreamBench to evaluate streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks. They also propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction, with a complex memory mechanism. Their method enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Extensive evaluations on StreamBench and some public benchmarks demonstrate that StreamChat outperforms the selected
baselines.
1. The proposed dataset is too small, only 306 videos and 1.8K  question-answer pairs are collected. The current video benchmark typically has at least thousands of videos and tens of thousands of QA pairs. This version of the benchmark is not ready for release.
2. State-of-the-art video LLMs are not included in the benchmark, such as MiniCPM-V 2.6 [1], InternLM-XComposer2.5 [2], VILA [3], and InternVL2 [4]. The effectiveness of the proposed method is unclear.
3. The Selective Frame Stacking seems may ignore small objects in the video and only focus on the global frame feature.
4. The proposed memory mechanism is very complicated and discards a large amount of information in the video.
5. The authors test the processing speed of models on two NVIDIA Tesla A800 80GB GPUs, which is not a typical scenario of model development in real-world applications.


[1] Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., ... & Sun, M. (2024). Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.
[2] Zhang, P., Dong, X., Zang, Y., Cao, Y., Qian, R., Chen, L., ... & Wang, J. (2024). Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320.
[3] Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., & Han, S. (2024). Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 26689-26699).
[4] Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., ... & Qiao, Y. (2024). How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.
A Vertasile Approach and Benchmark for Streaming Video Understanding and Multi-round Interaction
@inproceedings{
anonymous2024streaming,
title={Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=JbPb6RieNC},
note={under review}
}
Recent advances in Large Language Models (LLMs) have enabled the development of Video-LLMs, advancing multimodal learning by bridging video data with language tasks. However, current video understanding models struggle with processing long video sequences, supporting multi-turn dialogues, and adapting to real-world dynamic scenarios. To address these issues, we propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction. StreamChat leverages a novel hierarchical memory system to efficiently process and compress video features over extended sequences, enabling real-time, multi-turn dialogue. Our framework incorporates a parallel system scheduling strategy that enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Furthermore, we introduce StreamBench, a versatile benchmark that evaluates streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks.  Extensive evaluations on StreamBench and other public benchmarks demonstrate that StreamChat significantly outperforms existing state-of-the-art models in terms of accuracy and response times, confirming its effectiveness for streaming video understanding.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Video MLLM; Streaming Video Understanding
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2955158372fc9afb76eccf9446dcd3a91ae326ec.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission