Streamlining Redundant Layers to Compress Large Language Models
large language models;model compression;structured pruning
Thank you for your thorough review and the constructive feedback on our manuscript. We have carefully reviewed and considered each of the questions and suggestions you have raised. Below, we provide detailed responses to your comments. In the content below, we use **W** to represent Weakness and **Q** to represent Question.
 

**W1: Comparison of the acceleration effect of layer replacement with pure layer pruning.**

We are sorry for the misunderstanding. In our experiments, under the same sparsity constraint, when using a Transformer layer as the lightweight network, the pruned model's structure remains identical to that of pure layer pruning. As a result, the acceleration effect of layer replacement is equivalent to that of pure layer pruning. While when using an MLP as the lightweight network, our pruned model includes a layer that omits the attention architecture, resulting in a slightly better acceleration effect compared to pure layer pruning. 


We also report the inference speed (in output tokens per second) in Table 11 of the appendix in the revised paper (We have also put this table at the bottom of comments.) . The results indicate that our approach (layer) — replacing with Transformer layer— achieves the same speed (29.45 tokens/s) as our approach (None) — pure layer pruning. Additionally, our approach (FFN) — replacing with MLP — is slightly faster, at 29.87 tokens/s, compared to pure layer pruning.

**W2: Regarding how many layers are removed at once in Equation 3.**

Thank you for your question. The number of layers removed is determined based on the specified sparsity. For instance, in our experiments, we set the sparsity to about 25%, meaning we prune layers equivalent to 25% of the total parameters at once. For example, Llama2-7B has 32 layers, so pruning 25% of the model's parameters is roughly equivalent to removing 8 layers.

**W3: one-shot pruning or iterative pruning**

We are sorry for the misunderstanding. All of our experiments use the one-shot pruning approach that you mentioned. Iterative pruning—removing one layer at a time and replacing it with a lightweight network—would not significantly reduce the overall number of parameters. This is because a lightweight network, such as an MLP, contains roughly two-thirds the parameters of a single Transformer layer. Therefore, replacing each layer individually with a separate lightweight network, rather than using one lightweight network to replace multiple layers, would not lead to a significant reduction in parameters. On Llama2-7B, to achieve approximately 25% sparsity by using different lightweight networks to replace each layer, we would need to train about 24 lightweight networks to replace 24 Transformer layers. This results in a significantly high training cost.

**W4: Comparison between MSE loss and the original language model loss**

Thank you for raising the comment! In terms of computational resource usage, training with MSE loss only requires storing the lightweight network in memory, without needing to load the entire LLM. In contrast, training with the original language model loss requires loading the full LLM into memory. By using MSE loss, we minimize the memory requirements during training, making LLM-Streamline feasible in memory-constrained environments. 

**W5: Experiments on higher sparsity level**

Thank you for your suggestion. We will conduct experiments at higher sparsity levels, and we will strive to update the paper with these results before the end of the rebuttal period.



|          Llama2-7B         | Dense | LLM-Pruner | SliceGPT | Ours(None) | Ours(FFN) | Ours(Layer) |
|:--------------------------:|:-----:|:----------:|:--------:|:----------:|:---------:|:-----------:|
|      Pruning Ratio (%)     |  0.00 |    24.8    |   25.4   |    24.0    |    25.0   |     24.0    |
| Inference Speed (tokens/s) | 20.96 |    30.30   |   31.30  |    29.45   |   29.87   |    29.45    |

Table 1: The inference speed of models pruned using different methods.
Thank you very much for your thorough review and valuable feedback on our manuscript. We have carefully reviewed and considered each of the questions and suggestions you have raised. Below, we provide detailed responses to your comments. In the content below, we use **W** to represent Weakness and **Q** to represent Question.

**W1 and W2: Experiments on higher sparsity level, more diverse architectures, and larger models**

Thank you for your suggestion. We will conduct experiments at higher sparsity levels, with more diverse architectures, and on larger models, and we will strive to update the paper with these results before the end of the rebuttal period.

**W3: Result of real hardware performance**

Thank you for your feedback. We have included the results for inference speed in the Appendix of the revised paper(See **Table 11**). We have also put this table at the bottom of comments.



|          Llama2-7B         | Dense | LLM-Pruner | SliceGPT | Ours(None) | Ours(FFN) | Ours(Layer) |
|:--------------------------:|:-----:|:----------:|:--------:|:----------:|:---------:|:-----------:|
|      Pruning Ratio (%)     |  0.00 |    24.8    |   25.4   |    24.0    |    25.0   |     24.0    |
| Inference Speed (tokens/s) | 20.96 |    30.30   |   31.30  |    29.45   |   29.87   |    29.45    |

Table 1: The inference speed of models pruned using different methods.
Thank you very much for your thorough review and valuable feedback on our manuscript. We have carefully reviewed and considered each of the questions and suggestions you have raised. Below, we provide detailed responses to your comments. In the content below, we use **W** to represent Weakness and **Q** to represent Question.
 

**W1 and Q1: Comparison of Cosine Similarity with Other Metrics**

Thanks for your question. In **lines 159-176** of our paper, we detail our rationale for selecting cosine similarity over other metrics. Since most modern LLMs utilize a post-norm architecture, this results in the norm of the hidden states increasing with the depth of the layers. This leads to a bias where deeper layers tend to exhibit higher dot product similarity, while earlier layers show smaller Euclidean distances.
 

Regarding perplexity, we find it to be a highly sensitive metric. As shown in our experiments (see **lines 702-730** in the appendix), using perplexity as a pruning metric leads to the selection of different pruning layers depending on the pre-training dataset. Moreover, we compare the performance of models pruned using perplexity with those pruned using cosine similarity. The results suggest that models pruned using cosine similarity outperform those pruned with perplexity.
 
 

**W2 and Q3: Comparison of LLM-Streamline and LoRA**

Thanks for your suggestion. In **lines 199-215** of our paper, we highlight the advantages of LLM-Streamline over LoRA. Specifically, LLM-Streamline offers lower memory consumption and faster training speeds. Moreover, the lightweight network used in LLM-Streamline is trained through distillation, resulting in enhanced performance compared to LoRA. Furthermore, in **lines 473-485**, we present a detailed comparison between LLM-Streamline and LoRA, with experimental results showing that LLM-Streamline consistently outperforms LoRA in both performance and resource efficiency.
 

**W3: A simplified explanation of stability**

Thank you for your suggestion. We can easily explain our motivation for introducing the concept of stability. After pruning, a model may correctly answer questions that it previously answered incorrectly in classification tasks based on perplexity (PPL), which can even lead to an increase in accuracy for certain tasks. However, this does not necessarily indicate that the model’s overall capability has improved after pruning. Therefore, we propose stability as a metric, which focuses on the consistency of the model’s responses before and after pruning. We think it can provides a more accurate reflection of the pruned model’s performance than accuracy alone.
 

**Q2: Examples that illustrate the advantage of stability over accuracy**

Thank you for your valuable suggestion. For example, in PIQA, there is a question like this: 

The following makes sense: 

A.
Question: How to make tissue paper window decorations? Answer: Find tissue paper and fold it in half. Take scissors and cut out pieces of the paper in the middle. When you are done, tape it to your window. 

B.
Question: How to make tissue paper window decorations? Answer: Find tissue paper and fold it in half. Take scissors and tear out pieces of the paper in the middle. When you are done, tape it to your window. 


Llama2-7B has a very similar PPL for both options, indicating great uncertainty in this question and choosing the wrong answer B. However, the pruned Llama2-7B still has a very similar PPL for both options but manages to guess the correct answer A. This situation reflects the limitations of accuracy, whereas stability can well identify such cases, thus more accurately reflecting the performance of the pruned model.
Thank you very much for your thorough review and valuable feedback on our manuscript.
 

In response to your question, “Why didn’t you compare with some knowledge distillation methods for LLM?”, we appreciate your insight. While both distillation and pruning are model compression techniques, they serve different purposes. Distillation is primarily used to improve the training of smaller models, whereas pruning focuses on directly reducing the size of a large model. Although some pruning methods, such as the LLM-Streamline approach we proposed, involve the training process and can incorporate distillation loss during this phase, the two methods can complement each other. Therefore, papers introducing distillation methods, like DistiLLM, typically do not compare with pruning techniques, and vice versa.
Yes
No ethics review needed.
I'm not an expert of LLM pruning/compression field. I'm curious why you didn't compare with some knowledge distillation method for LLM, for example 
https://proceedings.mlr.press/v235/ko24c.html
Basically, this submission has two major innovations:
1. layer replacement
2. new metric named stability.

The first one is a very good contribution that mitigates the loss of pruning only.
This submission presented LLM-Streamline, a new model pruning method for LLMs. Besides traditional pruning, this method also proposed layer replacement, a novel module that trains a lightweight network to replace the pruned layers to mitigate performance loss. In addition, the authors also proposed stability as a new metric. Experimental results show the superiority of this method.
I didn't find any weakness of this submission.
Yes
No ethics review needed.
No ethics review needed.
1. Can you provide more detail on why cosine similarity was chosen as the primary measure for layer redundancy? How does it compare to using perplexity or other metrics?
2. For the stability metric, is there a specific scenario or task where it offers clear advantages over traditional accuracy measures? An illustrative example would help clarify this.
3. Have you considered a detailed comparison with other model compression methods, like LoRA? This would help highlight what sets LLM-Streamline apart.
1. Originality: This paper combines layer pruning with lightweight network replacement in a novel approach for compressing LLMs. This method effectively maintains model performance even after significant pruning.
2. Significance: The proposed stability metric enhances LLM compression evaluation by addressing limitations in standard accuracy metrics, providing a potentially more reliable measure of retained model performance.
3. Technical Quality: The experiments are comprehensive, covering various benchmarks that showcase the model’s effectiveness in classification and content generation. Each step of the pruning and replacement process is clearly explained, highlighting the technical soundness of the approach.
4. Practical Implications: LLM-Streamline improves model efficiency while retaining performance, making it valuable for real-world applications where computational resources are limited.
In this work, the authors propose LLM-Streamline, a new approach for compressing large language models (LLMs) by pruning layers based on cosine similarity and replacing the pruned layers with a lightweight network to preserve or even enhance model performance. This two-step method is shown to effectively reduce model size with minimal loss in accuracy, as demonstrated by experiments across various classification and generation benchmarks. Additionally, the work introduces a stability metric to address limitations of traditional accuracy metrics in evaluating pruned models. Experimental results highlight the superiority of LLM-Streamline over other state-of-the-art pruning methods in terms of accuracy retention and computational efficiency, especially under hardware constraints. Overall, this work offers a novel technique with practical implications for large-scale LLM deployments.
1. Metric Justification: While cosine similarity is chosen as the primary metric for layer redundancy, additional justification for this choice over other metrics (e.g., perplexity, Euclidean distance) would be beneficial.
2. Comparison with Other Methods: While the authors mention alternative approaches, such as LoRA, they do not provide a detailed comparison. A more thorough discussion of how LLM-Streamline performs relative to other popular fine-tuning and compression techniques would give readers a clearer perspective.
3. Explanation of Stability Metric: The paper could simplify its explanation of the stability metric, making it more accessible to readers new to model compression. Clarifying why stability offers practical advantages over accuracy would also be beneficial.
Yes
No ethics review needed.
Please refer to weaknesses section for questions.
* This paper uses a lightweight network and training based on hidden states before and after compression to compensate for the loss caused by pruning, reducing the need for computing resources and leading to better precision recovery.

* The continuous layer pruning used in this paper reduces the complexity of the compressed model and is easier to accelerate on hardware than unstructured pruning and other methods.

* The paper provides an in-depth analysis of the limitations within traditional accuracy metrics, offering a well-thought-out solution to address it. Additionally, the ablation studies are exceptionally comprehensive, with thorough and insightful analysis throughout.
The paper presents LLM-Streamline, a novel layer-wise pruning and replacement framework aimed at compressing large language models (LLMs) by identifying and removing less significant layers. The process is divided into two stages: (1) Layer Pruning to identify and prune layers with minimal impact based on cosine similarity, and (2) Layer Replacement, which trains a lightweight network to compensate for the removed layers, mitigating performance loss. Additionally, a new stability metric is introduced to more accurately reflect performance post-pruning, which is compared against existing methods across multiple benchmarks.
* The sparsity levels explored in the paper do not exceed 25%, leaving higher sparsity scenarios untested. In contrast, methods like LLMPruner evaluate and compare performance at a 50% sparsity level. This omission raises concerns about whether the proposed contiguous layer pruning approach would remain effective at higher sparsity levels.

* The models selected for evaluation are all based on the LLaMA architecture, which limits the assessment of the proposed method’s generalizability. Testing on more diverse architectures (e.g., MoE, GQA) and larger models with greater sparsity requirements (such as LLaMA-30B, LLaMA-70B) would provide a more comprehensive validation of LLM-Streamline’s effectiveness across various model types and scales.

* The study can add comparisons of real hardware performance (e.g., inference speed, FLOPs) with other structured pruning methods, where actual hardware constraints play a critical role in model selection and pruning effectiveness.
Yes
n/a
No ethics review needed.
Please see the above weakness section.
* The concept of layer replacement for performance recovery is intriguing.
* I appreciate the detailed comparison between perplexity and cosine similarity as pruning metrics, effectively revealing the data-sensitive drawbacks of perplexity.
* The paper comprehensively covers various types of LLMs and evaluation benchmarks.
* I like the exploration of multiple designs for lightweight networks.
* Overall, the paper is well-written and easy to follow.
This work addresses the compression of LLMs through layer pruning and layer replacement. A lightweight module is trained to substitute for the pruned layers. Additionally, a new evaluation metric is proposed to mitigate the limitations of accuracy-based metrics.
* I think that replacing layers with FFN or Transformer layers may not provide the full inference speed benefits typically gained through layer pruning, even if parameter counts remain similar. Could you analyze the speedup of layer replacement and compare it with pure layer pruning?
* It’s unclear how to set the number of contiguous layers (n) in Equation (3). How many layers are typically removed at once during pruning? Could you elaborate on this aspect?
* Is this method intended as one-shot pruning (pruning-replacement once) or iterative pruning (pruning-replacement, then pruning-replacement again)? Would it be possible to compare the two approaches?
* Why is MSE chosen as the loss function for the replacement module in Equation (3)? What would happen if it were trained using the original language model loss?
* A 25% compression rate seems relatively modest. Could you provide a comparison at higher pruning ratios (e.g., 50% layer removal) against baseline methods?
@inproceedings{
anonymous2024streamlining,
title={Streamlining Redundant Layers to Compress Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=IC5RJvRoMp},
note={under review}
}
This paper introduces LLM-Streamline, a pioneer work on layer pruning for large language models (LLMs). It is based on the observation that different layers have varying impacts on hidden states, enabling the identification of less important layers to be pruned. LLM-Streamline comprises two parts: layer pruning, which removes consecutive layers with the lowest importance based on target sparsity, and layer replacement, a novel module that trains a lightweight network to replace the pruned layers to mitigate performance loss. Additionally, a new metric called stability is proposed to address the limitations of the widely used accuracy metric in evaluating model compression. Experiments show that LLM-Streamline outperforms both previous and concurrent state-of-the-art pruning methods in terms of both performance and training efficiency.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language models model compression structured pruning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3d9897782f2cbb1786c6a42380ebd851be3aead2.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Streamlining Redundant Layers to Compress Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission