ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
LLM;model compression;Low-rank decomposition;efficient AI
Yes
No ethics review needed.
See above
- The paper is well written, with extensive empirical results support the arguments. 

- The combination with quantization approaches is desirable.
This paper proposes Activation-aware Singular Value Decomposition (ASVD) to prune the weights of LLMs in a training-free manner. In particular, the technical contributions include a data whitening trick with activation magnitude or Cholesky decomposition as well as a layer-wise rank search method. The empirical studies are extensive.
- One major issue is the close relationship and similar results with SVD-LLM. In particular, the paper offers a choice of using activation magnitude to construct the whitening matrix while SVD-LLM advocates the Cholesky decomposition of the activation covariance. As shown in Table 2, the latter is consistently superior (corresponding to ASVD+). In this sense, the original contribution of this paper is minor. The searching method of layer-wise rank with a calibration set is relatively novel compared to SVD-LLM, but it is still widely used in training-free LLM compression (see [1, 2]).

- The rationale of KV cache compression is not clear enough. In particular, the positional embedding is not considered when developing the low-rank projections. In my practice, this cannot decrease peak running memory for the whole system, while slightly reducing the storing memory. Besides, comparing your compressed states with those in EigenAttn [3], which one is better? In my opinion, EigenAttn does not consider the weight matrix when constructing the low-rank projections but your method considers that weight. What will this lead to?

- When the Param ratio is relatively low (e.g., less than 0.7), the compression is more promising, but the method is worse than SVD-LLM. Besides, metrics like MMLU should be reported in each scenario because it is of more concern in practice. And other task-specific metrics should also be included like accuracies on gsm, wino, etc. 


[1] MODEL TELLS YOU WHAT TO DISCARD: ADAPTIVE KV CACHE COMPRESSION FOR LLMS. 

[2] MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention.

[3] Eigen Attention: Attention in Low-Rank Space for KV Cache Compression.
Yes
No ethics review needed.
### Questions:
- Several other unstructured pruning methods use non-uniform layerwise sparsity, such as OWL [3]. Could the authors compare sensitivity-based truncation rank search with these non-uniform sparsity approaches?
- Could you test the actual memory footprint and computation speedup of ASVD?

---

[3] Yin, Lu, et al. "Outlier Weighted Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity." *Forty-first International Conference on Machine Learning.*
### Strengths:
- ASVD introduces a pioneering approach in SVD pruning by accounting for the influence of activation distribution, inspiring subsequent works like SVD-LLM and Palu.
- ASVD innovates with a sensitivity-based truncation rank search, enabling dynamic allocation of non-uniform ranks across layers.
This article introduces a novel, training-free, low-rank pruning method for reducing the memory and computational demands of large language models (LLMs). ASVD addresses the challenges of distribution variance in activations and sensitivity differences across layers.
### Weaknesses:
- This type of activation-based pruning is not novel in the pruning field, particularly within LLM pruning, as demonstrated by methods like Wanda [1], RIA [2], and others.
- The memory compression rate of ASVD is lower than that of other methods, such as quantization and unstructured/semi-structured pruning, which may limit its applicability. However, exploring memory compression techniques beyond quantization and pruning is also valuable. ASVD’s contribution to SVD pruning may have the potential for future advancements in-memory compression.
- The sensitivity-based truncation rank search involves calculating PPL for each layer at varying truncation ratios, which may be time-consuming for large models.
- The paper lacks experiments on actual memory footprint or computation speedup.

[1] Sun, Mingjie, et al. "A simple and effective pruning approach for large language models." arXiv preprint arXiv:2306.11695 (2023).

[2] Zhang, Yingtao, et al. "Plug-and-play: An efficient post-training pruning method for large language models." In The Twelfth International Conference on Learning Representations. 2024.
Yes
In the main part of this paper, the authors cite follow-up papers, which in turn cite the preprint of the ASVD paper. This creates a loop in the citation graph. Additionally, the authors' names are not properly anonymized. I'm unsure how to address this issue.
Yes, Other reasons (please specify below)
* Am I correct in assuming that the matrix $S^{-1}$ is also counted in the total number of parameters? Does it make sense to set $S = L$, where $L$ is the Cholesky factor, in terms of the parameters stored?
* Would it be possible to include plots of the normalized singular values for different layers, as seen in "Reduced-Order Modeling of Deep Neural Networks" by Gusak et al.?
The main idea is clear, and the paper is well-written. There are enough experiments to convincingly demonstrate that the algorithm performs better than pure SVD. The possibility of quantization and KV-cache compression is also highlighted.
ASVD is a method for the compression of LLMs. SVD is applied not to the original weight matrix, but to the transformed one. The transformation is computed using samples from the dataset.
The only solid dataset is MMLU, and the only LLM family is LLAMA-2.
Yes
No ethics review needed.
Please refer to weakness.
1. The method proposes a train-free method for model compression of LLMs, and it uses activation values to rescale the weight matrix so that weight vectors with larger activation norms are more important.
2. It further reduces the KV cache due to SVD, but one may need to recompute the KV using the low-rank KV at the next prediction step.
3.  Its performance is better than SVD and FWSVD.
The paper introduces the Activation-aware Singular Value Decomposition (ASVD), a training-free method for compressing large language models (LLMs). It targets compression challenges like activation distribution variances and layer sensitivity. ASVD achieves compression by transforming weight matrices based on activation distribution and adjusting for layer sensitivity through an iterative calibration.
1. The methodology framework of this paper is mostly based on weighted SVD, where you use a diagonal matrix to reweight the weight vector. The difference between this paper and FWSVD is that this paper puts the activation magnitude in the diagonal, and FWSVD puts the fisher information into the diagonal matrix.
2. One of the problems of SVD is that the full-rank model actually has twice the size of the original model. As a result, if you prune the model to 90% parameters of the original model, you remove 55% of ranks on average. A better way for SVD is to look at layers where they are not connected by activation functions (non-linearity), where you can get 50% parameter reduction for 50% rank reduction, examples are the value matrix and output transformation matrix like in [1].
3. Missing results regarding zero-shot performance beyond MMLU. In SVD-LLM, they tested their results on other tasks, like MathQA, PIQA, etc.
4. Missing related works on compressing language models with SVD [2]. In [2], the method can find the number of ranks per weight matrices in a learning process which is very relevant to section 3.4 of this paper.

[1] Lin, Chi-Heng, et al. "MoDeGPT: Modular Decomposition for Large Language Model Compression." arXiv preprint arXiv:2408.09632 (2024).

[2] Gao, Shangqian, et al. "Adaptive Rank Selections for Low-Rank Approximation of Language Models." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.
Activation-aware Singular Value Decomposition (ASVD), a training-free method for compressing Large Language Models. It addresses activation variance and layer sensitivity, achieving 10%-30% model size compression and reducing KV cache memory by 50%.
@inproceedings{
anonymous2024asvd,
title={{ASVD}: Activation-aware Singular Value Decomposition for Compressing Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=HyPofygOCT},
note={under review}
}
In this paper, we introduce a new post-training compression paradigm for Large Language Models (LLMs) to facilitate their wider adoption. We delve into LLM weight low-rank decomposition, and find that the challenges of this task stem from ❶ the distribution variance in the LLM activations and ❷ the sensitivity difference among various kinds of layers. To address these issues, we propose a training-free approach called Activation-aware Singular Value Decomposition (ASVD). Specif- ically, ❶ ASVD manages activation outliers by transforming the weight matrix based on the activation distribution. This transformation allows the outliers in the activation matrix to be absorbed into the transformed weight matrix, thereby enhancing decomposition accuracy. ❷ Additionally, we propose an efficient iter- ative calibration process to optimize layer-specific decomposition by addressing the varying sensitivity of different LLM layers. In this way, ASVD can compress a network by 10%-30%. Based on the success of the low-rank decomposition of projection matrices in the self-attention module, we further introduce ASVD to compress the KV cache. By reducing the channel dimension of KV activations, memory requirements for KV cache can be largely reduced. ASVD can further achieve 50% KV cache reductions without performance drop in a training-free manner.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM model compression Low-rank decomposition efficient AI
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/75fc45b56ac45606a154ed0131e50122cbfa83d9.pdf
infrastructure, software libraries, hardware, systems, etc.
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/dd3ba4835d9560a8fbfdf7e6ead509ffad23f2fc.zip
ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission