MoDeGPT: Modular Decomposition for Large Language Model Compression
LLM;model compression;matrix decomposition
Yes
No ethics review needed.
1. Why does Table 3 include only 50% compression results for models like SparseGPT but lack results for 40% compression? Why is a 40% compression result of MoDeGPT compared to a 50% compression result of SparseGPT?
2. I am curious why magnitude-based and SVD-based compression methods seem to cause model collapse in Table 1, performing worse than random compression (Uniform).
3. The authors applied different compression rates to different layers, but are the compression rates for the three types of computations identical? Based on the analysis in Figure 4, it might be better to allocate a higher compression rate for Attention computations.
4. Why is MoDeGPT more efficient than the baseline at the same compression rate (Figure 3)?
1. The authors propose the interesting idea of using three different matrix decomposition algorithms to compress computations in both MLP and Attention.
2. Experimental results demonstrate that the proposed method offers advantages in terms of both performance and efficiency compared to prior pruning and matrix decomposition algorithms.
3. The Appendix includes additional methods and experiments related to group-query attention.
This paper proposes a novel model compression method by applying three different matrix decomposition algorithms to three distinct types of computations within Transformers. Compared to previous model compression algorithms, this approach achieves a significant improvement in performance.
1. The authors suggest using three different types of matrix decompositions for three different types of computations within Transformers, but they do not provide motivation for this choice. For example, why is CR decomposition more suitable for Type-2 computation?
Yes
No ethics review needed.
1. In Table 3, is the main claim that although semi-structured methods may outperform MoDeGPT, they are held back by custom GPU support which hinders research velocity?
2. It would be nice to see a throughput versus perplexity graph as well, as opposed to just sparsity vs ppl/throughput, e.g. merge tables 2 and 3.
To the best of my knowledge, the method of structured approximations across multiple matrices is novel and the results are strong. For the most part, the paper is well-written.
This paper proposes MoDeGPT, which compresses transformers by applying structure decompositions on operations that span *two* weight matrices. The parameter subgroups targeted are the MLP weights, key and query projections, and value and attention output projections. Experimental results show that MoDeGPT is the best no-gradient structured method, and also comparable to the best structured and gradient-based method.
One weakness is the lack of justification for the approximation methods for each weight group. Could you give more intuition behind why each method was chosen? For example, the sentence "Since $W_U$ is inside a nonlinear function $\sigma_s$, we constrain the search space for its approximation to a matrix multiplication $W_U S_k$ for tractability, where $S_k$ is the $k$-column selection matrix" (line 244) only describes the approximation, whereas a justification would explain why Nystrom is a better fit for this problem than other methods.

Another weakness is the relative lack of analysis on the global sparsity allocation. However, this is orthogonal to the main contribution of structured multi-weight approximations.
Yes
No ethics review needed.
1. Can MoDeGPT outperform "efficient" competitors, such as SliceGPT [2], SLEB, if the competitors perform fine-tuning on the sample dataset to have the same pruning cost as MoDeGPT?

2. Could you elaborate on the detailed explanation of the proof for Theorem 4? Is it permissible to assume that epsilon is large enough to simplify the problem?

3. Does the proposed Global Sparsity Allocation outperform OWL [3]'s strategy?

4.  Does MoDeGPT outperform competitors when pruning gigantic models, e.g., Llama-3 70B?

5. What are the characteristics of Nystrom approximation, CR decomposition, and SVD, and why do we have to use them as proposed in this paper?

### References

[1] Song, Jiwon, et al. "SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks." arXiv preprint arXiv:2402.09025 (2024).

[2] Ashkboos, Saleh, et al. "Slicegpt: Compress large language models by deleting rows and columns." arXiv preprint arXiv:2401.15024 (2024).

[3] Yin, Lu, et al. "Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity." arXiv preprint arXiv:2310.05175 (2023).
This paper has diverse strengths and I summarize them as follows:

### Method
1. The authors introduce Nystrom approximation, CR decomposition, and SVD to pruning row-column pairs in LLMs. To the best of my knowledge, this is the first work to use Nystrom approximation and CR decomposition to prune LLMs. The authors carefully use them to prune different types of modules.

2. The authors propose a novel global sparsity allocation algorithm with entropic regularization. If this algorithm contributes a lot to improving the accuracy of the pruned models, then this algorithm can be broadly used in pruning.

### Experiments
3. The authors conduct exhaustive experiments to show the superiority of MoDeGPT. Their experiments not only covers accuracies, but also inference speed and pruning cost.

4. The authors analyze the effect of MoDeGPT in a detailed way. They also analyze the sparsity patterns.

### Writing

5. The contents are well-organized and easy to read. Specifically, the authors assign unique colors for each module type and consistently use them. This was very helpful to understand this paper.
This paper proposes MoDeGPT, an accurate structured pruning algorithm for LLMs. 
The main idea of MoDeGPT is to define "modules", a novel pruning structure, and apply tailored decomposition algorithms for three different types of modules.
The main strengths of this paper are (1) introducing decomposition algorithms that are not previously used in this domain, (2) proposing a new global sparsity allocation algorithm, and (3) exhaustive experiments and theoretical analysis in Appendix.
However, I have concerns regarding the following: (1) overclaiming regarding the efficiency of MoDeGPT,  (2) lack of experiments regarding large models, e.g., Llama 3 70B, and (3) too simplified proof of Theorem 4.
Therefore, I summarized my concerns in "Weaknesses" and "Questions" and I need to discuss them with the authors.
The score can be increased according to the author's response.
### Method

1. In the caption of Figure 1, the authors insist that their new pruning structure avoids the need for extra adapters. However, SliceGPT's adapters are introduced to improve accuracy and can be removed for inferencing without (dimensional) errors. Therefore, that statement should be modified.

2. The main contribution of this paper is introducing diverse decomposition algorithms and applying them to the proper modules. However, there are lack of explanations of the characteristics of these decomposition algorithms and justification for using them for each type of module.

3. The proof of Theorem 4 is too simplified and hard to understand. There are lack of explanations to get Equation 33. The authors impose a strong assumption that epsilon becomes infinity which indicates the uniformness of phis.

### Experiment

4. The authors emphasize that MoDeGPT is an efficient pruning algorithm, for example, in Lines 475-477, but MoDeGPT requires expensive pruning costs more than 8 hours for pruning Llama-2 13B models. According to SELB [1], most of pruning algorithms requires less than 16 minutes for pruning Llama-2 13B models. Therefore, it is an overclaiming to insist that MoDeGPT is an efficient algorithm. 

5. There are lack of competitors. The authors should compare their results with state-of-the-art pruning algorithms, especially layer (or block) pruning algorithms, such as SLEB [1]. Layer pruning algorithms provide significant inference speedup and should be included in Figure 3.

### Writing

6. The second paragraph of the Introduction is too detailed and hard to find the main point. It is hard to capture "these challenges" in the third paragraph after reading.

7. The criteria of Table 1 are ambiguous. (1) "No backward propagation" seems like an indirect criteria of pruning efficiency, but MoDeGPT is slow without requiring backpropagation. (2) What is the threshold of maintaining accuracy? (3) SparseGPT supports 2:4 pruning which is treated as a (semi-)structured pruning algorithm.
Yes
No ethics review needed.
The specific questions and suggestions are listed below:

(1)Do you consider evaluating on more diverse tasks to verify the method's generalizability?

(2)In the specific experiments, could you provide the chosen rank size for the matrix decomposition or an analysis of the related experiments?
The paper has the following strengths：

(1) The paper presents a novel training-free compression method called MoDeGPT, applies matrix decomposition at the module level for the first time, and extends the theoretical foundation for weight decomposition in language models.

(2) The paper offers a comprehensive literature review and theoretical analysis, demonstrates significant performance improvements through experimental results, and provides error guarantees along with a theoretical framework.

(3) The method outperforms previous approaches in compression performance, achieves a 46% increase in inference throughput, and enhances the practical value of large language models.
This paper introduces MoDeGPT, a novel training-free compression method for large language models.  It presents a systematic framework for categorizing approximation challenges in Transformer compression, complete with error guarantees. MoDeGPT demonstrates significant performance gains. This method outperforms prior approaches in compression, and achieves a 46% increase in inference throughput.
The Weaknesses of the paper are listed as follows：
(1) MoDeGPT shows intrinsic bias, performing well on some zero-shot tasks but poorly on others, and currently lacks a solution for bias removal.
(2) Overfitting of the model to calibration data prevents the compression method from generalizing across most tasks.
A framework that expands matrix decomposition for LLM compression beyond SVD
@inproceedings{
anonymous2024modegpt,
title={MoDe{GPT}: Modular Decomposition for Large Language Model Compression},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=8EfxjTCg2k},
note={under review}
}
Large Language Models (LLMs) have significantly advanced AI with their exceptional performance across a wide range of tasks. However, their extensive computational requirements restrict their use on devices with limited resources.
While recent compression methods based on low-rank matrices show potential
solutions, they often suffer from significant loss of accuracy or introduce substantial
overhead in parameters and inference time. In this paper, we introduce Modular De-
composition (MoDeGPT), a new, efficient, and structured compression framework
that overcomes these limitations. MoDeGPT jointly decomposes pairs of consecu-
tive subcomponents within Transformer blocks, reduces hidden dimensions through
output reconstruction on a larger structural scale than conventional low-rank meth-
ods, and repurposes three classical matrix decomposition algorithms—Nyström
approximation, CR decomposition, and SVD—to ensure bounded errors in our
novel decomposition approach. Our experiments show that MoDeGPT, without
relying on backward propagation, consistently matches or surpasses the performance of prior techniques that depend on gradient information, while achieving a
98% reduction in compute costs when compressing a 13B-parameter model. On
LLaMA-2/3 and OPT models, MoDeGPT retains 90-95% of zero-shot performance
with compression rates of 25-30%. The compression process can be completed on
a single GPU in a few hours, boosting inference throughput by up to 46%.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM model compression matrix decomposition
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b698bdb71bf0e62b4650b364692baf1b6c022da6.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
MoDeGPT: Modular Decomposition for Large Language Model Compression
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission