FISTAPruner: Layer-wise Post-training Pruning for Large Language Models
large language models;post-training pruning
**Response to Weakness 1:**

Thank you for your comment. We would like to clarify that FISTAPruner does not involve retraining during pruning; instead, it utilizes a layer-wise pruning approach, similar to methods like SparseGPT and Wanda. To clarify, retraining typically refers to the process of re-optimizing the entire model, usually by fine-tuning it on a specific loss function with respect to a training dataset after pruning. In contrast, FISTAPruner prunes the model in a layer-wise manner without such re-optimization, making it comparable to SparseGPT and Wanda, which also perform layer-wise pruning without retraining.

We understand the concern about computational costs, particularly in comparison with heuristic-based pruning methods like SparseGPT, Wanda, and DSnoT. While FISTAPruner may require slightly more pruning time due to its optimization-based approach, its key contribution is the introduction of an optimization model that systematically addresses the post-training pruning problem, leading to significantly better performance. This performance gain comes with an associated increase in pruning time, but the trade-off is justified by the improved outcomes.

Furthermore, while PERP (Parameter-Efficient Retraining after Pruning) is used for retraining pruned models, we have included comparisons between FISTAPruner and SparseGPT+PERP, as well as Wanda+PERP, in Table 4. The results show that FISTAPruner, which does not involve any retraining, outperforms SparseGPT and Wanda with retraining via PERP. Additionally, our method is compatible with retraining approaches and can provide a better initialization point for the retraining process, should that be required.

In conclusion, although FISTAPruner may take more time for pruning compared to heuristic methods, the substantial performance improvements it provides justify this additional time cost. Moreover, since our method does not involve retraining, the overall time cost remains competitive. We believe that, given the enhanced performance and the potential for further optimization, the time cost should not be viewed as a major concern.



**Response to Weakness 2:**

Thank you for your comment. As mentioned in Lines 188--190, our model incorporates $l_1$-norm regularization to encourage sparsity. Mathematically, the $l_1$-norm is non-smooth, which prevents the direct application of traditional gradient descent methods for optimization. Furthermore, although the sub-gradient descent method could be used, its convergence rate is $\mathcal{O}(1/\sqrt{k})$, which is slower compared to FISTA, which achieves a faster convergence rate of $\mathcal{O}(1/k^2)$, as mentioned in Lines 229--231.



**Response to Question 1:**

Thank you for your question. The key difference between retraining and our approach lies in the optimization objective. Retraining typically involves optimizing a model using a loss function that depends on the model's predictions, which is highly non-convex and requires computationally expensive backpropagation to compute gradients. In contrast, our approach uses a layer-wise error measured by the Frobenius norm in the objective function. This error is convex, and its gradient has closed-form expressions, making the optimization both convex and computationally more efficient. Therefore, while both methods aim to adjust the model, our approach does not involve retraining the model in the traditional sense of fine-tuning using a loss function.


**Response to Question 2:**

Thank you for your question. We indeed conduct sequential pruning within each decoder layer, specifically among the linear operators (K, Q, V, Out projections, and MLPs), to mitigate error accumulation through our intra-layer error correction mechanism. However, to enhance pruning efficiency, we treat each decoder layer as an independent pruning unit, enabling parallel pruning across multiple decoder layers on different devices. In other words, while pruning is performed sequentially within each individual decoder layer, the pruning process can occur in parallel across different layers, which allows us to strike a balance between performance and efficiency. We hope this clarifies how parallel pruning is achieved alongside the sequential pruning within layers.


**We appreciate your thoughtful comments and will ensure that all the concerns and questions raised, including the clarification of the pruning process, are addressed more clearly in the revised version of the paper.**
Yes
No ethics review needed.
1. In line 89, the paper states, "Our results confirm that FISTAPruner can efficiently create sparse networks from pretrained LLMs without retraining." However, the process of finding the pruned weights W* seems to function similarly to retraining. Could you clarify this point, as it may cause confusion for readers?
2. In line 306, the paper states, "We treat each decoder layer as an independent pruning unit, enabling parallel pruning across multiple decoder layers on different devices." However, the proposed method conducts pruning sequentially. Can you explain how parallel pruning is achieved alongside sequential pruning? A more detailed explanation or revision would be helpful.
(+) The method effectively incorporates FISTA for efficient pruning during the post-training process, leading to faster optimization and enhanced performance
(+) By effectively employing LASSO to identify pruned weights with targeted sparsity, the approach minimizes reliance on heuristic-based methods, thereby improving overall effectiveness in the pruning process.
(+) The authors enhance the proposed method by developing an algorithm that enables semi-structured pruning, allowing for practical acceleration on real-world hardware.
The paper proposes FISTAPruner, a layer-wise pruning method designed for both unstructured and semi-structured pruning, targeting efficient sparsification of large language models (LLMs). This approach utilizes the FISTA method (Fast Iterative Shrinkage-Thresholding Algorithm) to facilitate efficient convergence. Additionally, it employs a LASSO-like convex optimization model to effectively enhance sparsity in LLMs. To address the cumulative output error between the full and pruned models due to the sequential output error transfer across transformer decoder layers, the authors utilize layer-wise pruning with an intra-layer error correction mechanism.

Experiments conducted on various model sizes, ranging from 125M to 70B parameters—including OPT, LLaMA, LLaMA-2, and LLaMA-3—across datasets such as WikiText-2-raw, PTB, and C4, demonstrate that FISTAPruner outperforms existing baseline methods (e.g., SparseGPT, Wanda, Wanda+DSnoT, SparseGPT+PERP, and Wanda+PERP) in terms of model performance after pruning.
(-) The paper requires experiments to compare FISTAPruner with other methods that have similar computational costs. Existing baseline methods, such as SparseGPT and Wanda, do not involve retraining during pruning. In contrast, FISTAPruner conducts retraining in the process of finding W∗. Although DSnoT and PERP are used instead of retraining, their computational costs are lower than the layer-by-layer approach employed in FISTAPruner. So, it is necessary to compare their performance under similar computational cost conditions (e.g., training on SparseGPT is performed layer by layer).
(-) The benefits of using FISTA over traditional gradient descent methods are not sufficiently explained, which may leave readers unclear about its specific advantages in this context.
Yes
No ethics review needed.
**1. Major Question:** In Wanda [1], they prune model weights by choosing the smallest $|W| ||X||_2$. While in your methods, you prune weights by minimizing the discrepancy of $||W^\*X^\*-WX||_2$. What's the difference between your sparsity objective with that of Wanda?   

**2. Minor Question:** In the paper, the error correction mechanism is applied solely within individual layers. Why is the error correction confined to intra-layer applications rather than being implemented across the entire model? In my understanding, extending the error correction mechanism globally could further mitigate the phenomenon of error accumulation throughout the network. 


[1] A SIMPLE AND EFFECTIVE PRUNING APPROACH FOR LARGE LANGUAGE MODELS, ICLR 2024
1. The paper presents a unique approach by integrating FISTA with a LASSO-like model, which is innovative in the context of LLM pruning.
2. The results demonstrate that FISTAPruner can prune up to 50% of model parameters while retaining high accuracy, outperforming existing methods like SparseGPT and Wanda.
3.  The integration of an intra-layer error correction mechanism is novel, which may avoid error cumulation.
The paper introduces FISTAPruner, a novel method for pruning large language models (LLMs) post-training to achieve significant sparsity, thereby reducing memory footprint and computational demands without compromising model performance. The authors introduce a LASSO-like convex optimization model tailored for layer-wise pruning, utilizing the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) to induce sparsity. A another innovation is the integration of an intra-layer error correction mechanism that mitigates cumulative errors across decoder layers during the pruning process. Additionally, FISTAPruner is extended to support 2:4 semi-structured pruning, aligning with hardware acceleration capabilities. Comprehensive experiments on various models (OPT, LLaMA) demonstrate that FISTAPruner outperforms state-of-the-art methods such as SparseGPT, Wanda, DSnoT, and PERP across multiple benchmarks, including perplexity and zero-shot task performance.
**1. Major Weakness:**   The intra-layer error correction mechanism is briefly mentioned but could benefit from a more detailed explanation and analysis. It raises the question of whether other methods (e.g., SparseGPT and Wanda) could achieve better performance if integrated with this mechanism.
Yes
No ethics review needed.
Question:
- Can you discuss difference between this work and sparseGPT?
- Can you perform ablation studies on 2:4 structured sparsity
- This work is theoretically grounded, and provide some guarantees on convergence time.
- This work shows strong results in structured 2:4 pruning setup.
- This paper is overall well-written and easy to understand.
The author proposed an LLM pruning algorithm that uses “FISTA” (Fast Iterative Shrinkage-Thresholding Algorithm) to identify optimal pruning masks. The author demonstrates the utility of the proposed technique across many state-of-the-art LLMs and with both structured and unstructured sparsity. The improvement over prior art is, however, small.
- It’s unclear to me what’s new in this work relative to, say, SparseGPT, which also sets up pruning as an optimization problem and generally yields similar results as this work in unstructured pruning setup. It occurs to me that the fundamental difference appears to be that this work uses a different optimizer to solve essentially the same problem.
- While in structured 2:4 pruning setup this work yields substantial improvement, it is unclear why this is the case.
- Neither “Amount of Calibration Data” nor “Warm Start” is actually ablation study. Please do proper ablation studies by removing specific features of your algorithm design.

I am willing to raise my score if the authors can deliver real ablation studies that pinpoints why this proposed algorithm achieved superior performance in 2:4 structured sparsity setup.
Yes
No ethics review needed.
1. What's the difference between FISTA, and L1-regularized training using SGD w/ momentum?

2. Is there any reason you use outdated models such as OPT and Llama-1? How about using the latest models such as Phi, Gemma, and Mistral, if you want to use diverse models?

3. Could you compare the performance of your "Adaptive hyperparameter tuning" algorithm with existing hyperparameter search algorithms, e.g. BOHB [3]?

4. Are DSNoT and PERP (1) competitors or (2) compatible algorithms? If (1) competitors, then how about integrating Tables 2 to 4 as a single table? If (2) compatible algorithm, then how about integrating Tables 3 and 4? In this case, it would be better to compare the performance of "FISTAPruner" with "FISTAPruner + DSnoT" and "FISTAPruner + PERP" to show the compatibility.

5. What is the main point of the Section "Warm Start"? Could you clarify the takeaway of this section?

### References
[1] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Llm-pruner: On the structural pruning of large language models." Advances in neural information processing systems 36 (2023): 21702-21720.

[2] Song, Jiwon, et al. "SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks." arXiv preprint arXiv:2402.09025 (2024).

[3] Falkner, Stefan, Aaron Klein, and Frank Hutter. "BOHB: Robust and efficient hyperparameter optimization at scale." International conference on machine learning. PMLR, 2018.
The main strengths of this paper are as follows:

1. The authors achieve meaningful accuracy improvement in diverse settings. For example, FISTAPruner shows almost 5% higher accuracy than the second-best algorithm, i.e., SparseGPT, when pruning Llama-3 70B models.

2. This paper conducts extensive experiments covering diverse models from OPT to Llama-3 to show the robustness of FISTAPruner. FISTAPruner consistently shows comparable or the highest accuracy (or the lowest perplexity) in all cases.

3. The figures in this paper are straightforward to understand.
This paper proposes FISTAPruner, an accurate pruning algorithm for large language models (LLMs).
The main ideas of FISTAPruner are (1) intra-layer error correction, (2) FISTA-based optimization algorithm, and (3) adaptive hyperparameter tuning algorithm.
The authors conduct exhaustive experiments to verify the effectiveness of FISTAPruner, and find that FISTAPruner is more accurate than existing algorithms; specifically, it shows almost 5% higher average accuracy on zero-shot tasks when pruning Llama-3 70B.
The main strength of this paper lies in its high accuracy and exhaustive amounts of experiments.
However, the novelty and writing quality of this paper are insufficient.
I summarize the weakness of this paper below. I use the symbols [M] and [m] for each numbering to distinguish between major and minor weaknesses.

### Method
The main weakness of this paper is the lack of originality (or novelty). We summarize the weaknesses of the proposed method as follows.
1. [M]  Error correction, the first idea, is just using the output of the pruned previous linear operators, and this idea is already used in previous works. Furthermore, the authors ignore the "inter-layer errors" induced by the pruning of previous layers when they correct errors.

2. [M] The authors make use of the existing optimization algorithm, FISTA, without any modification. Introducing L1 regularization for pruning is a prevalent idea and there is no novelty.

3. [M] The authors propose a new hyperparameter tuning algorithm, which has no specific name, but there is no explanation of the strength or novelty of this algorithm. There are no experiments that compare the performance of this algorithm with previous hyperparameter tuning algorithms.


### Writing
The followings are the weaknesses in writing.

4. [M] The main contribution of this paper is to use FISTA algorithm to prune LLMs. However, explanation about FISTA is too insufficient. It would better introduce the basics of FISTA in Section 2 (Background) and explain the modification to use FISTA for pruning LLMs in Section 3.2.

5. [M] According to "1.", it is hard to agree with the statement "Instead of pruning each operator in isolation like existing works" in line 148.

6. [m] Minor issues in writing:

  6.1  (line 193) "The proposed optimization model 3" -> "The proposed optimization model in Equation 3"

  6.2 (line 262) "Theorem 3.3" -> "Theorem 1"

  6.3 (All equations) Use bold texts for representing matrices and vectors following the guideline of ICLR. It would be better to use blackboard bold S for representing a set of permissible sparsity patterns in Equation 1.

  6.4 (All tables) Move captions of tables above the tables following the guideline of ICLR.

  6.5  This paper does not contain the "Reproducibility Statement" which is encouraged by ICLR.

  6.6 (Table 6) There are too many bold texts in Table 6.


### Experiments

7. [m] The authors compare the performance of FISTAPruner with limited competitors without justification. The authors should compare the performance of FISTAPruner with structured pruning algorithms [1,2] or justify their selection of competitors.

* References are at the end of this review
@inproceedings{
anonymous2024fistapruner,
title={{FISTAP}runer: Layer-wise Post-training Pruning for Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=BINwUtUGuq},
note={under review}
}
Pruning is a critical strategy for compressing trained large language models (LLMs), aiming at substantial memory conservation and computational acceleration without compromising performance. However, existing pruning methods typically necessitate inefficient retraining for billion-scale LLMs or rely on heuristically designed metrics to determine pruning masks, leading to performance degradation. This paper presents, for the first time, a LASSO-like convex optimization model crafted to induce sparsity in LLMs. By leveraging the FISTA, we introduce FISTAPruner, a novel method that includes a cumulative error elimination mechanism within decoder layers and supports parallel pruning for unstructured pruning. Additionally, we extend this method to 2:4 semi-structured pruning. We comprehensively evaluate FISTAPruner on models such as OPT and LLaMA variants with 125M to 70B parameters under unstructured and 2:4 semi-structured sparsity, showcasing superior performance over existing methods across various language benchmarks. Notably, it can remove 50% of the model parameters for LLaMA-3-70B while retaining 98.6% and 95.6% of the zero-shot task performance under these two sparsity patterns, respectively.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language models post-training pruning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/18e06b3bd42a40c9942cadee8eaea46bb35db379.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
FISTAPruner: Layer-wise Post-training Pruning for Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission