Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification
Efficient Multimodal Large Language Model
> **Baselines and benchmarks.**

We primarily use LLaVA-1.5 [14] as our main baseline, employing the same 12 benchmarks utilized in LLaVA-1.5. This choice is motivated by the fact that most works in the field [1-7] validate their results on these benchmarks for LLaVA-1.5, allowing for a fair comparison by adopting the same settings. To evaluate the generation ability of MLLMs, we use the LVIS-VQA single- and multi-round benchmarks. Our goal is to assess whether Dynamic-LLaVA, while reducing output text tokens, significantly impacts the generation ability of MLLMs. The results indicate that our method enhances inference efficiency with minimal impact on generation performance. In the rebuttal, we will supplement our results with additional baselines and benchmarks.

> **Writing.**

We recognize that excessive notation may hinder broader reader comprehension. In the revised paper, we are incorporating more intuitive visualizations and adding simplified descriptions to improve the readability.

[1] IVTP: Instruction-guided Visual Token Pruning for Large Vision-Language Models. ECCV2024.

[2] Llava-prumerge: Adaptive token reduction for efficient large multimodal models. Arxiv2403.

[3] VoCo-LLaMA: Towards Vision Compression with Large Language Models. Arxiv2406.

[4] Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs. Arxiv2409.

[5] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV2024.

[6] HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models in Resource-Constrained Environments. Arxiv2408.

[7] SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference. Arxiv2410.

[8] Language models are unsupervised multitask learners. OpenAI blog.

[9] Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America.

[10] Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. ACL Workshop 2005.

[11] Fast inference from transformers via speculative decoding. ICML2023.

[12] Online speculative decoding. Arxiv2310.

[13] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. NeurIPS2024.

[14] Improved Baselines with Visual Instruction Tuning. CVPR2024.
The Preliminary Rebuttal (3/3)
> **Distinctions from LLMs' KV Cache Compression Methods.**

The proposed Dynamic-LLaVA framework differs fundamentally from LLMs' KV cache compression methods in the following ways:

1. Dynamic-LLaVA introduces a tailored sparsification inference scheme specific to various inference modes, i.e., prefill, decoding with KV cache, and decoding without KV cache. Only in the case of decoding with KV cache can it be regarded as a form of "online KV cache compression". For prefill and decoding without KV cache, where KV cache activations are not involved, Dynamic-LLaVA can still significantly accelerate MLLM's computation, a benefit that KV cache compression methods cannot achieve. We have noted the reviewers' focus on the decoding with KV cache scenario and the classification of our method as a KV cache compression approach. In the revised paper, we will include a more detailed comparison with LLM KV cache compression methods and discuss the computational benefits in the prefill and decoding without KV cache scenarios. This includes the importance of prefill-stage inference acceleration for MLLMs [5] and potential applications for decoding without KV cache, such as the widely used speculative decoding strategy [11, 12].
2.   Considering the complete generation process of MLLMs with KV cache, Dynamic-LLaVA achieves computation and GPU memory savings in the prefill stage, as well as GPU memory savings during decoding (disregarding computational acceleration from KV cache reduction for extremely long sequences [13]). In contrast, typical KV cache compression methods generally offer only GPU memory savings during decoding. This distinction arises because Dynamic-LLaVA evaluates each current token's features to determine whether its activations through $W_K$ and $W_V$ should be added to the KV cache, rather than removing entries from the past cache (detailed in Eq. 6 of the main paper). KV cache compression methods, however, often rely on past generated KV cache. E.g., H2O [13] uses the current Q with the past KV cache in decoding to compute attention scores and decide which cache activations to discard, rather than directly deciding to retain or discard activations for the current token. We will expand upon this discussion in the upcoming revision of the paper.
3. Dynamic-LLaVA is an MLLM inference acceleration framework that considers the distinct properties of different modalities and incorporates tailored sparsification strategies accordingly. As noted in [5], “While these methods have boosted the inference efficiency of LLMs, they are designed for text-only language models, and whether their effectiveness can be transferred to LVLMs remain under-explored.”. Thus, the direct applicability of LLM KV cache compression methods to MLLMs is yet to be verified. In our detailed rebuttal, we will provide results on the application of LLM KV cache compression methods in MLLMs.

> **Predictor mechanism.**

The predictor used in Dynamic-LLaVA is applied only once, after the l-th decoder layer of the MLLM. Specifically, once the predictor makes a decision on tokens, this decision is shared across all subsequent layers. For instance, in both the prefill and decoding without KV cache stages, after the predictor reduces the token set $S_l$ to $S_l^*$, all subsequent layers use this reduced token set $S_l^*$ for computation. In decoding with KV cache, the decision of whether to add the current token to the KV cache is likewise propagated to subsequent layers. Following [5], we set l=2, meaning that token reduction occurs after the 2nd layer. Ablation studies on the position of l are provided in Tab. 6 of the main paper.

As for the additional inference overhead introduced by the predictor, we first present theoretical FLOP values. For LLaVA-1.5-13B, the predictor's FLOPs in the prefill stage are under 0.01T, compared to the MLLM’s 19.6T FLOPs. In Table 4 of the main paper, we also report latency measurements on a real GPU for prefill and decoding without KV cache stages. During the decoding with KV cache phase, the predictor's computational overhead is similarly minimal and can be considered negligible. We will provide further details in our comprehensive rebuttal and the revised paper.
The Preliminary Rebuttal (2/3)
Dear Reviewers,

Thank you very much for your effort on our paper. 

We are preparing a detailed rebuttal to address the valuable feedback provided by the reviewers. Currently, we have observed that some reviewers have already adjusted their scores and comments before our complete rebuttal. We have decided to release a preliminary rebuttal for several key and commonly raised concerns.  *Please note that this is not the final rebuttal.* A point-by-point response to all reviewers' comments, along with a revised version of the paper, will be submitted subsequently.

Sincerely Yours,

The Authors


> **Comparison with token reduction methods.**

We present the performance of current SoTA and recent token reduction methods. The results in the table are based on evaluations conducted with LLaVA-1.5-7B, using the performance of Dynamic-LLaVA$_{I/T}$ as the base. We report the performance differences of other methods relative to ours. The 'Free' column indicates whether a method is training-free (i.e., can be applied directly to MLLMs without training).

|                             | Free           | Token$\downarrow$ | TFLOPs$\downarrow$ | VQA$^{v2}$  | GQA         | SQA          | VQA$^{T}$    | POPE        | MME            | MMB         | SEED        |
| --------------------------- | -------------- | ----------------- | ------------------ | ----------- | ----------- | ------------ | ------------ | ----------- | -------------- | ----------- | ----------- |
| ***Dynamic-LLaVA$_{I/T}$*** | ***&#10005;*** | ***20%***         | ***25%***          | ***77.9***  | ***61.3***  | ***68.6***   | ***56.5***   | ***85.9***  | ***1501.0***   | ***64.1***  | ***65.0***  |
| (ECCV24) IVTP [1]           | ✕              | -                 | 53%                | 77.8 (-0.1) | 60.4 (-0.9) | 67.8 (-0.8)  | 58.2 (+1.7)  | 85.7 (-0.2) | -              | 66.1 (+2.0) | 56.4 (-8.6) |
| PruMerge+ [2]               | ✕              | 25%               | 25%                | 76.8 (-1.1) | -           | 68.3 (-0.3)  | 57.1 (+0.6)  | 84.0 (-1.9) | 1462.4 (-38.6) | 64.9 (+0.8) | -           |
| VoCo-LLaMA [3]              | ✕              | 22%               | 22%                | 76.9 (-1.0) | 59.8 (-1.5) | -            | -            | -           | -              | 61.0 (-3.1) | 59.1 (-5.9) |
| TRIM [4]                    | &#10005;       | 21%               | -                  | 76.4 (-1.5) | 61.4 (+0.1) | 48.1 (-20.5) | 53.7 (-2.8)  | 85.3 (-0.6) | 1461.3 (-39.7) | 67.4 (+3.3) | -           |
| (ECCV24) FastV [5]          | ✓              | 25%               | 32%                | 75.1 (-2.8) | 57.5 (-3.8) | 68.7 (+0.1)  | 56.2 (-0.3)  | 81.0 (-4.9) | 1458.9 (-42.1) | 63.5 (-0.6) | 62.8 (-2.2) |
| HiRED [6]                   | ✓              | 20%               | 20%                | 74.7 (-3.2) | -           | 66.4 (-2.2)  | 44.2 (-12.3) | -           | -              | -           | -           |
| SparseVLM [7]               | ✓              | 22%               | -                  | 73.8 (-4.1) | 56.0 (-5.3) | 67.1 (-1.5)  | 54.9 (-1.6)  | 80.5 (-5.4) | 1696 (+195)    | 60.0 (-4.1) | -           |

Our proposed Dynamic-LLaVA_I/T achieves the best performance in 34 out of 42 valid results across eight commonly used benchmarks. It is worth noting that, similar to our method, some other approaches, such as PruMerge+ [2], also require additional post-training.  For LLaVA-1.5, both Dynamic-LLaVA_I/T and PruMerge+[2] undergo one epoch of post-training. However, Dynamic-LLaVA_I/T outperforms PruMerge+ in 4 out of 6 results when LLaVA-1.5-7B is used as the baseline. Moreover, for LLaVA-1.5-13B, Dynamic-LLaVA_I/T consistently surpasses PruMerge+ across all benchmarks (see Tab.1 in the main paper). Compared to other token reduction methods, Dynamic-LLaVA_I/T not only reduces 80% of image tokens in the prefill stage but also decreases 50% of output text tokens during decoding, consistently outperforming both training-free and training-required methods in most cases.

For the generation ability benchmarks, it is important to note that Dynamic-LLaVA_I/T achieves comparable PPL and METEOR scores to FastV [5], indicating similar levels of fluency and quality in generated text [8, 9, 10]. This suggests that, while some generated tokens are discarded, the impact on MLLM's ability to generate coherent text is minimal. However, this similarity in scores does not imply equivalent overall performance (see Tab. 1 in the main paper, where Dynamic-LLaVA_I/T outperforms FastV in 14 out of 16 results on the LLaVA-1.5 experiments).  Additionally, Dynamic-LLaVA_I/T further reduces 50% decoding computation (without KV cache) and GPU memory overhead (with KV cache) compared to FastV [5].
The Preliminary Rebuttal (1/3)
Yes
No ethics review needed.
See weaknesses.
1. This work pioneers the reduction of KV cache length during long text generation for MLLMs, whereas previous research has focused solely on reducing prefilled vision tokens.
2. The end-to-end sparsification training can be viewed as a novel way to enhance the token sparsity of MLLMs.
3. The paper proposed extensive experiments to show the effectiveness of the proposed method.
In this work, the authors propose Dynamic-LLaVA, a novel approach to enhance both prefilling and generation efficiency in Multimodal Large Language Models (MLLMs). The modified model employs two predictors: an image predictor that dynamically identifies and drops less significant image tokens during prefilling, and an output predictor that filters out non-essential output text tokens. When training these two predictors, the authors propose an end-to-end sparsification training pipeline, which added a mask to the original softmax attention to keep a fixed number of image tokens and generated text tokens. Additionally, the argmax is replaced by Gumbel-Softmax during backward to avoid the gradient flow problem. Extensive experiments show the effectiveness and efficiency of the proposed method.
1. While Dynamic-LLaVA demonstrates improvements over previous approaches like FastV and PruMerge+, it requires additional post-training, which introduces extra computational overhead. In table 3, it seems that the performance of Dynmaic-LLaVA and FastV are even comparable. Therefore, for real-world deployments where computational efficiency is crucial, implementing existing methods may be more practical. However, this raises an important question: the work would benefit from a clearer justification for why a trainable approach is necessary and what specific advantages it offers over more lightweight solutions. 

2. For the output reduction, as we only need to determine whether preserving the current generated token, this pipeline is the same as some work for LLM, such as StreamingLLM [1], H2O [2]. However, the author only compare the Random and Structure strategies. Therefore, I think this is a well-defined problem for LLMs, and the current experimental results cannot convince me. 

3. The methodology section of this work requires improvement to enhance clarity and comprehension. It is currently challenging for readers to grasp the entire training and inference pipeline quickly. Additionally, several design choices related to the training process lack sufficient explanation, which may leave readers puzzled about the rationale behind them.

4. Furthermore, the name "Dynamic-LLaVA" may lead to misconceptions about the method. It implies that the system dynamically drops the KV cache at each layer; however, it should be clarified that the cache is dropped permanently and cannot be recovered in subsequent layers.

5. The training process is not aligned with the inference process for the output predictor. The former one can see all generated tokens and decide the most important ones, while the latter one can only decide whether to drop the current one based on previous context.

[1] Xiao, Guangxuan, et al. "Efficient streaming language models with attention sinks." arXiv preprint arXiv:2309.17453 (2023).
[2] Zhang, Zhenyu, et al. "H2o: Heavy-hitter oracle for efficient generative inference of large language models." Advances in Neural Information Processing Systems 36 (2024).
Yes
No ethics review needed.
1. Previous studies have shown that deeper layers exhibit higher attention sparsity. How is sparsification strength adjusted for different decoder layers?
2. Running predictors introduces additional latency. However, the proportion of this latency relative to overall prefill and decoding time is not disclosed. Additionally, in standard LLaVA decoding, previous KV cache is fetched, and byproduct attention scores are accumulated to assess token importance, which is quick and efficient. However, for "Online KV Cache Compression," running predictor incurs additional latency. And only the current token is used to predict all previous tokens, do you think solely performing prediction on a current token provide enough information? Have you analyzed the wrongly evicted tokens?
3. Can this method be applied to language models that utilize Group Query Attention?
4. Have you considered scenarios where long-text context is present during the prefill stage?
This paper introduces a novel design for two token predictors, formulating visual and textual context compression as an end-to-end optimization problem. Techniques such as Gumbel-Softmax and the Straight-Through Gradient Estimator are employed to enhance training stability. In addition, parallel sparsification inference optimization would provide an effective solution for model serving.
This paper proposes a novel method, Dynamic-LLaVA, which dynamically sparsifies visual context during the prefill stage and vision-text context during the decoding stage of Visual Language Model inference. Specifically, two learnable predictors are designed to identify important visual and language tokens. Additionally, masked softmax is applied during training, and Gumbel-softmax with a Straight-Through Gradient Estimator is used to address gradient flow issues. Furthermore, a batch-parallel strategy is introduced to enable efficient sparsification during inference in the Dynamic-LLaVA framework. Empirical experiments demonstrate that this method can compress approximately 75% of visual context and save around 50% of GPU memory for the KV cache.
1. It is unclear whether the two learnable predictors are applied to each decoder layer. If they are, do they share the same parameters or are they independently parameterized?  If not all layers, please clarify which specific decoder layers the predictors are applied to.
2. Compressing generated text is primarily beneficial for long-generation tasks. Would you please provide specific statistics on the average output length for each task they evaluated? And discuss how the benefits of their method scale with increasing output length.
3. Please quantify the additional training time and computational resources required for this method compared to direct KV cache compression approaches, as well as additional prediction overhead compared normal decoding. This would help readers better understand the tradeoffs.
4. More recent related work about KV Cache compression method should be included, such as TidalDecode (https://arxiv.org/abs/2410.05076) focusing on decoding compression. 
5. A few mathematical representations are not appropriate. For example, in section 3.1, $\mathcal{I}^I = $\{ $1, 2, \cdots, N_{l}^I $}, layer index $l$ should be set for $\mathcal{I}^I $; And in section 3.3.1, all layers share the same binary masks $\mathcal{M}^I and \mathcal{M}^{OT}$? since there are no layer index $l$. 
6. The writing could be improved to make the narrative more concise and clear.
Yes
No ethics review needed.
There aren’t many issues; the method in the paper is logically clear, the experiments are comprehensive, and the performance is excellent.
1. The article has a rigorous logical structure, with precise formulations for each method and complete notation. Readers who read carefully can clearly understand the methods explained in the text.

2. The experimental results are excellent; it appears that inference speed and computational cost are significantly reduced with minimal loss in performance.

3. The method design is ingenious; selective approaches are often challenging to train, but the paper provides an effective end-to-end training method that successfully alleviates the gradient flow problem.
Dynamic-LLaVA is a framework for efficient inference in multimodal large language models, dynamically reducing vision context redundancy during the prefill and decoding stages, cutting computation by ~75% in prefill, and saving ~50% computation and memory during decoding, without significant loss in performance.  It also designs the specific training and inference strategy for parallel sparsification, making their approach more efficient.
Although the writing is logically clear and the formulas are complete, this level of detail makes it somewhat challenging for readers.  There are too many formulas here. Perhaps reduce them a bit and add more diagrams to aid understanding. Some high-level schematic diagrams could help; Figure 2 aids understanding to some extent, but it’s still not sufficient.
Yes
No ethics review needed.
1. I am curious why it is not tested on the latest versions of llava such as llava-one-vision, llava-next, etc.
2. I don’t quite understand why Table 5 uses GQA and VQAv2 as vision evaluations. I think a more vision-centric dataset such as MMVP is needed.
3. Paper mentions that different sparsification strategies are used in the pre-filling stage and the decoding stage, but the specific differences between the two and the motivation for this design are not clear. In particular, why both visual and language context are sparsified in the decoding stage, while only visual tokens are sparsified in the pre-filling stage, what is the design consideration behind this?
4. In order to avoid unstable training when there are few samples, paper only sparses the samples whose text length exceeds a certain threshold. However, does this strategy ensure the stability of the model when facing inputs of different lengths? In actual use, will the model cause significant performance fluctuations due to changes in input length? If a model behaves unstable for certain input lengths, this may significantly limit its applicability in real-world applications.
1. As an article focusing on efficiency, it grasps the key point very well: solving efficiency problems from the perspective of efficiency. From the numerical results, Dynamic-LLaVA performs well in reducing computation and memory consumption, especially in the pre-filling and decoding stages, reducing the computation by about 75% and the GPU memory overhead by about 50%.

2. The **SPARSIFICATION INFERENCE** part achieves dynamic sparsification of visual and language context by introducing learned predictors, effectively reducing the computational and memory burden in different reasoning modes, making the entire generation process more efficient. I think this is a highlight. Sparsification of tokens during test time is a very useful method, and I personally believe that less is more. These sparse parts should actually improve the representation ability.

3. MaskedSoftmax is a good point to note. From my previous fine-tuning experience, since the generated $logp(y|x)$ is calculated by logits under the corresponding label $y$, the appearance of 0 value in the whole chain process is absolutely harmful. This soft labeling method is very thoughtful.
The paper introduces Dynamic-LLaVA, a dynamic vision-language context sparsification framework aimed at enhancing the inference efficiency of mllms. The main challenge addressed is the increasing computational and memory overhead during inference, particularly as output tokens are generated, which diminishes efficiency benefits of prior vision context reduction methods. Dynamic-LLaVA reduces redundancy in both the vision and language contexts during the prefill and decoding stages, with a specialized sparsification strategy tailored for different inference modes, including decoding with and without KV cache. Experimental results demonstrate that Dynamic-LLaVA significantly reduces computational consumption—by about 75% during the prefill stage and 50% during decoding without KV cache—and saves approximately 50% of GPU memory with KV cache, all while maintaining or even improving model performance. This framework is the first to attempt simultaneous sparsification of both vision and language contexts, achieving efficient inference with negligible performance degradation.
1. In the method section, the description of sparse reasoning and end-to-end training relies too much on symbolic expressions and lacks intuitive explanations. It could be aided by adding some simplified explanations or visual flowcharts to help a wider readership understand the complex technical details.
2. You can consider adding some evaluations. I personally feel that the current evaluation benchmarks are not comprehensive enough to explain the performance of MLLMs. And you should analyze as thoroughly as possible. You can refer to the Cambrian-1 [1] standard or try to apply the method to the Cambrian model.
3. In the results of the experimental section, the impact of sparsification on complex tasks (such as generating long texts or multi-round conversations) is not sufficiently verified.

[1] Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., ... & Xie, S. (2024). Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860.
@inproceedings{
anonymous2024dynamicllava,
title={Dynamic-{LL}a{VA}: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=hzVpZDrW73},
note={under review}
}
Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision understanding, reasoning, and interaction. However, the inference computation and memory increase progressively with the generation of output tokens during decoding, directly affecting the efficacy of MLLMs. Existing methods attempt to reduce the vision context redundancy to achieve efficient MLLMs. Unfortunately, the efficiency benefits of the vision context reduction in the prefill stage gradually diminish during the decoding stage. To address this problem, we proposed a dynamic vision-language context sparsification framework Dynamic-LLaVA, which dynamically reduces the redundancy of vision context in the prefill stage and decreases the memory and computation overhead of the generated language context during decoding. Dynamic-LLaVA designs a tailored sparsification inference scheme for different inference modes, i.e., prefill, decoding with and without KV cache, to achieve efficient inference of MLLMs. In practice, Dynamic-LLaVA can reduce computation consumption by $\sim$75\% in the prefill stage. Meanwhile, throughout the entire generation process of MLLMs, Dynamic-LLaVA reduces the $\sim$50\% computation consumption under decoding without KV cache, while saving $\sim$50\% GPU memory overhead when decoding with KV cache, due to the vision-language context sparsification. Extensive experiments also demonstrate that Dynamic-LLaVA achieves efficient inference for MLLMs with negligible understanding and generation ability degradation or even performance gains compared to the full-context inference baselines.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Efficient Multimodal Large Language Model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d8856e8bb3f21994b3ef6ddcfe21f9d81f928a9c.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/d345998fca0c4510cdce9578ff0013e82c380754.zip
Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission