OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition
network pruning;low-rank;compression;sparsification;large language models;outlier features
I hope this message finds you well.

I have a question regarding the CPU throughput tests presented in Section 3.4 of your paper. Specifically, I was wondering if any hardware acceleration techniques, such as compression or other methods, were used during these tests. If not, in scenarios where the total data volume (SUV matrix > Weight matrix) is larger, what strategies would you suggest for improving throughput?

Thank you for your time, and I look forward to your insights.

Best regards
Question about CPU throughput tests presented in Section 3.4
Yes
No ethics review needed.
One potential extension of this work could involve incorporating quantization into the proposed framework. Although this addition may be a long shot, integrating quantization could make the model a unified approach to transformer compression. Could the authors provide insights on whether quantization can be integrated with their current framework, or if not, what are the main challenges?
- The concept of compressing a model as a sum of a sparse and low-rank matrix is very promising. Unlike most prior methods, which focus on one approach, OATS leverages both to potentially enhance performance.

- OATS is retraining-free, which is crucial for practical applications where even a single backpropagation pass can be computationally prohibitive.

- The framework has been tested on state-of-the-art models like Llama and ViT, demonstrating competitive performance.

- The Alternating Thresholding technique in OATS heuristically finds an effective combination of low-rank and sparse components and accommodates different sparsity patterns.

- By scaling the weight matrix $W$ with a diagonal rescaling matrix $D$, OATS emphasizes outliers, enabling outlier-aware compression that avoids performance degradation.
The paper introduces OATS (Outlier-Aware Pruning Through Sparse and Low-Rank Decomposition), a method designed to compress large transformer models without the need for retraining. The central concept involves representing weight matrices as the sum of a sparse and low-rank matrix.

The authors propose an iterative alternating thresholding technique to compute the joint sparse and low-rank decomposition of a matrix. They also focus on preserving outliers by scaling weights according to input embeddings prior to decomposition. The results indicate that OATS performs effectively on both language and vision tasks, outperforming other pruning methods proposed for transformers across various compression levels and tasks. Additionally, OATS offers speed improvements on the CPU.
One concern is that the method relies on multiple calls to truncated SVD, which can be computationally intensive. Specifically, finding the top-$r$ singular values of an $m \times n$ matrix has a time complexity of $O(mnr)$. Given that compression speed is a significant factor for practical applications, it would be helpful if the authors could clarify the time complexity and wall-clock time spent on the compression process of the overall algorithm. This would offer a more concrete understanding of its practicality.
Yes
N/A
No ethics review needed.
See the Weakness. Additionally, why use inverse transformation to reach the compressed weight? What is the actual speedup when using N:M sparsity on GPUs.
+ Low-rankness plus sparsity is a good fit for compressing LLMs without retraining.
+ By separating the model into a sparse and a low-rank part, the approximation error can be theoretically reduced as the two parts can compensate for each other. 
+ This paper provides measurements for practical speedups on CPUs with existing sparse computation frameworks.
This paper introduces OATS, a novel compression technique for large-scale transformers that combines sparse and low-rank approximations to reduce memory and compute costs without requiring costly retraining. OATS scales model weights by the second moment of input embeddings to preserve essential outlier features in transformers, ensuring model performance is maintained during compression. This approach addresses the typical performance degradation seen with increasing compression in existing pruning methods.
- The novelty is limited. The combination of low-rankness and sparsity is an old topic that has been explored for many years [R1, R2]. Applying the well-established approximation techniques to decompose/compress the large matrices in LLMs has little technical contribution. Besides, compressing DNN models using low-rank and sparse decomposition has already been well explored in [R3]. This paper just scales it to larger models and matrices. Authors are encouraged to specify the unique difference from existing approaches and why this difference is also unique for LLMs.
- The proposed Truncated SVD and Threshold strategies to achieve low-rankness and sparsity are too trivial. It is unknown how to decide the rank and number of zeroes. Besides, the order of applying SVD and thresholding has a significant impact on the approximation errors. Authors are encouraged to clearly explain why using such decomposition strategies.
- This paper claims "outlier information" in this paper. However, I have not seen any analysis or explanation for the "outlier information," and the proposed solution is not related to the "outlier information." Instead, this paper seems to directly apply the pruning approaches proposed in Wanda. The authors are encouraged to provide explanations of why the diagonal matrix is related to "outlier information" and why it is good for compression.
- Many works have been proposed to compress LLMs with low-rankness and sparsity [R4-R6]. The authors have not presented the main differences among them and the unique contributions that stand out from those works.
- Even though theoretical analysis may not have a practical guarantee of accuracy, the authors are encouraged to provide.
- The paper presentation could be improved, especially the math equations. 

[R1] Sparse and Low-Rank Matrix Decompositions, Forty-Seventh Annual Allerton Conference, 2009.

[R2] Godec: Randomized low-rank & sparse matrix decomposition in noisy case. ICML 2011.

[R3] On compressing deep models by low rank and sparse decomposition, CVPR 2017.

[R4] Slope: Double-pruned sparse plus lazy low-rank adapter pretraining of llms

[R5] LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation

[R6] SLiM - One-shot Quantized Sparse Plus Low-rank Approximation of LLMs
Yes
No ethics review needed.
It would be nice to see a bigger hyperparameter selection section with more architerctures considered.
The method proposed by this paper is well-explained and well-justified. The actual practical algorithm is easy to follow. 

Real-time speedup is shown in the CPU setting.

The experiments cover a range of model sizes (3.8-14B parameters). I especially liked seeing results on fairly small models, since those may be harder to compress. 

Section 5 is an interesting way to look at the problem, which I think can lead to interesting further work in the direction of interpretability.
This paper proposes a relatively novel model compression technique aimed at Transformer architectures, in which weight matrices are approximated as a sum of a sparse and a low-rank matrix. To control for the (previously documented) outlier feature problem, weights are also scaled by the second moment of their input.
The choice of the rank ratio parameter could have been better explored (in particular, looking at multiple architectures/tasks). 

Typos:
Line 18: “approximating each weight” -> “approximating each weight matrix”
Line 142: “the activations are calculated through a calibration set that is propagated through the compressed layers” - should be uncompressed?
Yes
No ethics review needed.
1. The author has clarified the differences between Wanda and OATS. However, an ablation study would further illustrate the impact of the low-rank term on performance. How much does the low-rank term contribute to the performance boost compared to Wanda alone?
2. The hardware speedup on GPU with N:M sparsity patterns can be shown and discussed in Section3.4.
3. The paper lacks details on how the sparsity pattern is defined for a matrix composed of a sparse matrix S plus a dense matrix LLL, especially within the context of N:M sparsity and structured pruning. Providing more detailed explanations would enhance clarity. 
4. The baseline for models are all originally designed for LLMs. It would be valuable to compare against pruning metrics specifically designed for ViTs. This would present a stronger baseline to effectively showcase the proposed method’s advantages.
5. The motivation behind the choice of outlier information for the sparse term S is not entirely clear. Given that there are various methods for selecting the sparse components, such as using magnitude-based or gradient-based criteria, it would be helpful to know if the authors experimented with these or other selection methods.
6. Given that most pruning studies report results on 70B-parameter models, has the author tested their method on larger language models, such as Llama2-70B or the newer Llama3-70B?
1.This paper propose a novel method for large transformers compression that utilizes the second moment of the input embeddings to approximate the model’s weight matrices as a sum of a sparse matrix and a low-rank matrix.

2.Extensive experiments on recent large language models demonstrate the effectiveness of the proposed method, which also generalizes well to vision Transformers.
In this paper, the author presents a novel approach to compressing large transformers, coined OATS, that utilizes the second moment information in the input embeddings to decompose the model weights into a sum of sparse and low-rank matrices. The author also conducts a lot of experiments showing that OATS is able to consistently outperform prior state-of-the-art across multiple benchmarks and compression rates while also improving on speed-up.
Please see the questions.
@inproceedings{
anonymous2024oats,
title={{OATS}: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=DLDuVbxORA},
note={under review}
}
The recent paradigm shift to large-scale foundation models has brought about a new era for deep learning that, while has found great success in practice, has also been plagued by prohibitively expensive costs in terms of high memory consumption and compute. To mitigate these issues, there has been a concerted effort in post-hoc neural network pruning techniques that do not require costly retraining. Despite the considerable progress being made, existing methods often exhibit a steady drop in model performance as the compression increases. In this paper, we present a novel approach to compressing large transformers, coined OATS, that compresses the model weights by approximating each weight as the sum of a sparse matrix and a low-rank matrix. Prior to the decomposition, the weights are first scaled by the second moment of their input embeddings, so as to ensure the preservation of  outlier features recently observed in large transformer models. Without retraining, OATS achieves state-of-the-art performance when compressing large language models, such as Llama-3 and Phi-3, and vision transformers, such as Google's ViT and DINOv2, by up to $60\\%$, all while speeding up the model's inference on a CPU by up to $1.37\times$ compared to prior pruning methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
network pruning low-rank compression sparsification large language models outlier features
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7792f3ad24e3cdbbebf56b24502295fe69d0f4e2.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/11c515e19114f0e02aeb5051500c9c639ab994ff.zip
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission