BLIP-3-Video: You Only Need 32 Tokens to Represent a Video Even in VLMs
video representation;video foundation model;vlm;multimodal language model
Yes
No ethics review needed.
1. What novel designs does this method introduce compared to TTM? Are there ablation studies for these designs?

2. The model utilizes the VideoChatGPT instruction set. Why hasn't it been evaluated on that benchmark?
1. An impressive tradeoff between efficiency and accuracy on the MSVD-QA benchmark.
2. Extensive explorations on temporal encoders to reduce visual tokens.
3. The paper is well written and easy to follow.
This paper presents BLIP-3-Video, which introduces a "temporal encoder" alongside a conventional visual tokenizer, allowing it to significantly reduce visual tokens (32 tokens compared to thousands in other models). The study explored various temporal encoders, including learnable spatio-temporal pooling and sequential models like token turning machines (TTM). Detailed experiments showed that different encoder types had a noticeable impact on performance, particularly in handling complex video scenarios. Experimental results show that BLIP-3-Video achieved video question-answering accuracies comparable to much larger state-of-the-art models while being smaller and more efficient.
1. Novelty: The primary weakness is the insufficient novelty. As detailed in Section 2.2, the only improvements to TTM include (1) time-stamped positional encodings and (2) a 'grouped' TTM temporal encoder. These minor changes do not substantiate a significant contribution.

2. Evaluation Benchmarks: The evaluated benchmarks are unconvincing for assessing Video LMMs. The model was only evaluated on MSVD-QA, MSRVTT-QA, ActivityNet-QA, TGIF-QA, and NExT-QA, which are not so ideal for testing LMMs. The authors may consider newer benchmarks like VideoMME and MVBench, which are proposed for assessing Video LMMs.
Yes
No ethics review needed.
- The (video part of the) training of this model is on video captioning data and video question-answering datasets. If the downstream task were to change to a more complex task, like temporal reasoning, would the model require more tokens or would 16~32 still be sufficient? i.e. is there enough visual information encoded in the 16~32 tokens?
- In addition, if the downstream task requires remembering multiple details and nuanced events over a long diverse scenario, how would this approach perform? Is there a built-in mechanism that prevents information loss during token pooling?
- The temporal encoder which aggregates visual tokens across frames in a highly efficient manner results in computational efficiency of training the model.
- Extensive ablation study in the temporal encoder design validates the design choice while also demonstrating the flexibility in the design of temporal encoder. 
- Competitive performance on various video question-answering benchmarks, despite its smaller size.
The paper introduces BLIP-3-Video, a multimodal vision language model which demonstrates strong performance on video understanding with high token accuracy. BLIP-3-Video uses as few as 16~32 tokens to encode an entire video sequence, which is highly efficient to other video VLMs, enabled by the incorporation of a temporal encoder. BLIP-3-Video achieves competitive accuracy on various video question-answering benchmarks while having smaller model parameters.
- The model proposed in the paper utilizes 8 frames per video which are uniformly sampled. This approach might not work for tasks that inherently require more than 8 frames to understand the video. If this method could scale up, an explanation of why that might be would be helpful.
- The experiments of the paper focuses on video question-answering benchmarks only, and this limited experimentation may not capture the model's ability in other video-based tasks. Further evaluation on other video tasks, such as temporal understanding would demonstrate the applicability of this approach to more general and diverse video-related tasks.
Yes
No ethics review needed.
1. Can one BLIP-3-Video model produce 32 and 128 tokens on demand as a hyper parameter? Or they are different models trained individually?

2. What does text encoder in Fig 2 mean? The text tokenizer?
This paper investigates an important topic, how to efficiently & effectively understand videos by LLMs, which is underexplored so far. This paper proposes the BLIP-3-Video to understand videos with just 32 tokens (in LLMs) based on Phi-3, and it outperforms both parameter-heavy or visual-token-heavy models (as shown in Fig 1) on QA and captioning benchmarks. Additionally, this paper presents a compelling finding (somewhat): a video can be effectively represented by just 32 tokens in LLMs for QA and captioning tasks. I think this research line is promising and can benefit several downstreaming tasks, e.g., captioning for text-to-video generation.
This paper presents an efficient Video LLM, coined as BLIP-3-Video, by incorporating extra modules to transform dense video tokens into sparse tokens (e.g., # can be 32). Specifically, the authors use a sequential transformer (so called token turing machine) on top of frame-wise image tokens and perceiver-resampler to produce limited 32 tokens. Diverse video QA benchmarks show the competitive performance of BLIP-3-Video on video question answering tasks. The authors also test the captioning ability of BLIP-3-Video.
Although this is overall a good paper, several concerns are here:
1. The presentation of the main method (Sec 2.2) somwhat presents confusion: Does BLIP-3-Video both use spatio-temporal attentional pooling and TTM? Is there a perceive resampler before temporal encoder in BLIP-3-Video (cannot be infered from Figure 2)?

2. Compressing a video into 32 tokens is a compelling and exciting idea. However, I am worried that spatial-temporal details will be missing through compression, which is crucial for some detailed reasoning in LLMs. More evaluation of BLIP-3-Video on diverse tasks beyond captioning and MCQ are encouraged.
(also, as the compression is not text query guided, the compression is solely dominated by the visual information itself. That is to say, 32 tokens per a video are fixed under different text query, which might not be appropriate in general)

3. Relating with the weakness 1, extra modules introduced besides the visual encoder (SigLIP) and LLM sound too complicated. If I understand correctly, there are a perceiver-resampler and a temporal encoder (attention pooling or TTM). My idea is naive and simple, can we just finetune a perceiver-resampler in BLIP-3 into a temporal encoder, rather than just compressing tokens per frame? Given the strong performance of cross attention layers in the perceiver resampler, this seems to be a missing but promising ablation study in this paper.
Yes
No ethics review needed.
1. **Temporal Encoder Generalization**: A smaller number of visual tokens is particularly important for understanding longer videos. However, this paper only tested on a few simple short video benchmarks. Please provide test results on video benchmarks of different lengths and scenarios, such as VideoMME, MVBench, etc.

2. **Scalability Concerns**: How does the model's performance and efficiency scale with longer videos, and have you observed any limitations in terms of the number of frames the model can effectively process?

3. **Model Interpretability**: The paper mentions the use of different types of temporal encoders. Are there any plans to provide insights into how these encoders make decisions?

4. **Comparison with Other Efficient Models**: How does BLIP-3-Video compare with other recent models that also focus on efficiency, such as those employing knowledge distillation or sparse attention mechanisms? Could the authors provide some insights into the trade-offs involved? Please provide persuasive evidence from experimental studies.

5. **Novelty**: Compared to LLaMA-VID, where is the core novelty of this paper? Although the experimental results show that 32 tokens achieved better performance on four short video benchmarks, this standard will change with different video lengths, video scenarios, and the complexity of question answering. The scalability and generalizability of this method are questionable. Perhaps a more effective mechanism for accommodating more frames and selecting key information for video question answering from a large number of visual tokens is worth exploring, rather than determining the specific numerical value of a visually overfitted token on a few benchmarks. Similar architectures have been explored enough in a series of works such as Video-LLaVA, LLaVA-VID, LLaVA-NEXT, and so on.
- **Originality**: It introduces an innovative temporal encoder that significantly reduces the number of visual tokens needed to represent videos, offering a new approach to efficiency in video understanding models.

- **Quality**: The model is thoroughly evaluated against state-of-the-art benchmarks, demonstrating competitive performance. The ablation studies are providing insightful analyses into the model's components.

- **Clarity**: The paper is well-organized, with clear explanations and visual aids that effectively convey complex information, making the technical content accessible to readers.

- **Significance**: BLIP-3-Video's efficiency in handling video data with fewer resources.
The paper introduces BLIP-3-Video, a novel multimodal language model designed for video understanding that efficiently captures temporal information across frames. A key innovation is the integration of a 'temporal encoder' that maps a sequence of tokens from multiple frames into a compact set of visual tokens, allowing BLIP-3-Video to operate with significantly fewer visual tokens compared to its competitors. The model explores various temporal encoders, including learnable spatio-temporal pooling and sequential models like Token Turing Machines. Experiments demonstrate that BLIP-3-Video achieves comparable video question-answering accuracies to much larger models, while being more efficient and smaller in size due to its reduced token usage. The paper also details the model's architecture, training recipe, and experimental results, highlighting the effectiveness of the temporal encoder in representing videos for question-answering tasks with a small number of tokens.
- **Diversity of Datasets**: The experiments primarily rely on a limited set of public benchmarks for evaluation. Expanding the evaluation to include a more diverse range of benchmarks, particularly those with varying lengths and complexities of videos, could provide a more comprehensive assessment of the model's generalizability and robustness.

- **Scalability Analysis**: While the paper demonstrates the model's efficiency, there is a lack of analysis on how the model scales with increasing video length and complexity. Future work could benefit from exploring the model's performance as it processes longer videos, which is crucial for real-world applications.

- **Comparison with State-of-the-Art**: Although comparisons are made with other models, the paper could benefit from a more detailed analysis comparing the trade-offs between BLIP-3-Video and the state-of-the-art models in terms of accuracy, computational resources, and inference time.

- **Implementation Details**: Some aspects of the model's implementation, such as the specific choices made in the architecture of the temporal encoder, could be elaborated upon with more technical depth. This additional detail would aid other researchers in understanding the design decisions and potentially replicating or improving upon them.
We present BLIP-3-Video, which is a compact video-based VLM using 20x less visual tokens compared to the standard models.
@inproceedings{
anonymous2024blipvideo,
title={{BLIP}-3-Video: You Only Need 32 Tokens to Represent a Video Even in {VLM}s},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=CKYsXi0dOV},
note={under review}
}
We present BLIP-3-Video, a multimodal language model for videos, particularly designed to efficiently capture temporal information over multiple frames. BLIP-3-Video takes advantage of the `temporal encoder' in addition to the conventional visual tokenizer, which maps a sequence of tokens over multiple frames into a compact set of visual tokens. This enables BLIP-3-Video to use much fewer visual tokens than its competing models (e.g., 32 vs. 4608 tokens). We explore different types of temporal encoders, including learnable spatio-temporal pooling as well as sequential models like Token Turing Machines. We experimentally confirm that BLIP-3-Video obtains video question-answering accuracies comparable to much larger state-of-the-art models (e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using fewer visual tokens.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
video representation video foundation model vlm multimodal language model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/51b0026137447689d180516b521f8c18a76edb70.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
BLIP-3-Video: You Only Need 32 Tokens to Represent a Video Even in VLMs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission