From Bulk to Budget: Best Practices To Compress Multimodal Large Language Models
Multimodal large language models;model pruning;knowledge distillation;model compression
Yes
No ethics review needed.
1. What are the limitations of the chosen pruning strategies for different types of tasks within MLLMs?
2. Can the proposed data-efficient recovery techniques be generalized to other model architectures or require specific adjustments?
1. The paper presents a unique combination of widthwise and layerwise pruning for MLLMs, complemented by targeted recovery strategies like finetuning and knowledge distillation. This distinguishes the work from typical compression techniques that rely solely on one pruning or finetuning method, offering a more adaptable framework for diverse deployment needs.
2. This ppaer emphasizes data-efficient model recovery, highlighting scenarios where only 5% of the original data suffices to restore a substantial portion of the model’s performance, making the proposed method practical for environments where labeled data is scarce or costly, potentially expanding its applicability in low-data or real-time contexts.
3. Extensive experimentation across two MLLMs with varying compression ratios, recovery techniques, and data requirements provides a detailed view of how different configurations affect model performance. The ablation studies also offer a deeper understanding of the benefits and limitations of each pruning and recovery method.
This paper addresses the challenge of compressing multimodal large language models in a data-efficient way, introducing structured pruning and recovery techniques for resource-constrained deployments. The paper explores layerwise and widthwise pruning strategies, with a focus on identifying effective recovery techniques, such as finetuning and knowledge distillation. Key findings include the effectiveness of widthwise pruning in low-resource scenarios and the combination of projector finetuning with hidden state knowledge distillation for optimal performance recovery. The study provides insights into best practices for compressing MLLMs while maintaining high performance across a variety of tasks.
1. The proposed techniques are evaluated within the context of MLLMs; however, they lack comparisons with other prevalent compression methods. This absence makes it challenging to assess their effectiveness against existing solutions, especially as structured pruning and knowledge distillation are already well-established in the field.
2. While the framework demonstrates potential at lower compression ratios, its performance gains are limited at higher compression ratios. This limitation may reduce the practical appeal of the method for applications that demand more aggressive compression while still preserving task performance.
3. Although the proposed combination of multiple pruning and recovery strategies is thorough, it increases implementation complexity. The reliance on complex data structures and recovery steps may hinder deployment. An author should clarify this.
Yes
No ethics review needed.
see weaknesses
The paper provides extensive empirical evaluations across different compression ratios, recovery strategies, and data requirements.

The experiments are well-designed and cover multiple dimensions: pruning methods, and data efficiency.
This paper presents a comprehensive study on compressing Multimodal Large Language Models (MLLMs) while maintaining performance. The authors investigate two pruning strategies (widthwise and layerwise) and various recovery methods (supervised finetuning and knowledge distillation) across different compression ratios. They conduct experiments on two MLLMs (LLaVA-v1.5-7B and Bunny-v1.0-3B) and provide best practices for MLLM compression based on their findings.
1. The authors' claim of being the first to investigate general-purpose MLLM compression overlooks existing work, particularly the survey paper on efficient MLLMs [1], which contradicts the authors’ claim. The literature review could be more comprehensive, especially in acknowledging related work in MLLM efficiency. 

2. The technical contributions largely adapt existing LLM compression techniques to MLLMs without introducing significant novel methods. 

3. The findings mostly confirm expected behaviors from LLM compression research. The paper primarily combines existing techniques rather than introducing new methodological advances. The exploration could better highlight MLLM-specific challenges and solutions that differentiate it from general LLM compression


[1] Efficient Multimodal Large Language Models: A Survey
Yes
No ethics review needed.
See Weakness
- This paper is easy to understand and follow. 
- This paper provides concrete best practices for practitioners
This paper provides a recipe to compress MLLM by evaluating the width-wise pruning and layerwise pruning, finding that widthwise pruning is better. Then, by recovery strategies like SFT and KD to restore the performance.
- The experiments are not comprehensive with just limited model selection. Please include newer MLLM architectures like InternVL, CogVLM, MiniCPM-v; 
- The title of this paper claimed "Best Practices To Compress MLLM". However, this paper only focuses on pruning and then knowledge distillation. Additionally, the pruning only focuses on LLM and there is no experiment on pruning Vision encoder. It is a little bit overclaimed when you do not involve other compression methods like quantization or low-rank factorization. This paper should dive into compression and provide more detailed insight for readers. 
- Limited novelty: pruning+knowledge distillation are the common techniques on LLM, as presented in Sheared LLaMA and Minitron. This paper has limited novelty as there is no difference. 
- Limited comparison with other compression methods: this paper proposed several techniques like width/depth pruning and KD. This paper did not involve any compression method with other methods.
Best practices for MLLM compression
@inproceedings{
anonymous2024from,
title={From Bulk to Budget: Best Practices To Compress Multimodal Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=774F8gF0UO},
note={under review}
}
Multimodal large language models (MLLMs) are increasingly developed to meet diverse deployment needs, varying in scale and computational demand. While recent research has focused on building MLLMs from Small Language Models (SLMs), these efforts remain limited in flexibility and are still data- and compute-intensive. In this paper, we present the first comprehensive study on flexibly compressing and recovering existing MLLMs in a data-efficient manner. Hence, we address a critical gap in the literature by empirically analyzing best practices for adapting to specific hardware or resource limitations. Our study investigates pruning and knowledge distillation techniques, examining their impact on downstream performance across various model compression strategies, including pruning paradigms, recovery training schemes, and data requirements. Key findings reveal that widthwise pruning is particularly effective in resource-constrained scenarios. For smaller compression ratios, finetuning the multimodal projector alone can restore most performance, while combining finetuning with hidden state knowledge distillation proves most effective across all compression levels. Notably, we demonstrate efficient model downsizing using as little as 5% of the original dataset for moderate compression. Our analysis suggests best practices for compressing MLLMs for resource-efficient deployment. With our best practices, Bunny-v1.0-3B retains over 95% of its original performance, while LLaVA-v1.5-7B maintains more than 97%, with compression ratios below 30%.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multimodal large language models model pruning knowledge distillation model compression
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4a9c251327e90252a42300853b247a5246dede48.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
From Bulk to Budget: Best Practices To Compress Multimodal Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission