NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer
Pruning;Compression;Entropy;Deep Learning
Yes
No ethics review needed.
Questions
- Check weaknesses
- Did the authors observe any particular trends in the layer types that are typically dropped or have larger number of neurons set to "off" state. Are the observations different across model families?
- How does the compute time for NEPENTHE compare to different baselines?
- Given that perplexity often doesn't translate to donwstream task performance (in-context learning properties) for decoder-only LLMs [1]. How do the authors see NEPENTHE being extended to large language models? Given that NEPENTHE requires iterative training, could the authors comment on the scalability of the approach?

[1] Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z. and Yang, Y., 2023. Compressing llms: The truth is rarely pure and never simple. arXiv preprint arXiv:2310.01382.
Quality and Clarity
- The paper is clearly written, with sufficient background and motivation for the problem

Originality and Significance
- The proposed iterative entropy guided pruning scheme is novel 

Experiments
- The paper studies the generalizability of the method across wide variety of models and applications ranging from a Resnet for classification to a RoBERTa for language tasks
- The paper provides thorough details on the experimental setup (in the appendix)
- The analyses provided on neuron states per layer, allowing to drop entire layers (hence allowing for some structured sparsity and latency gains) is interesting and valuable contribution
- The ablation study provided in Table 4 depict the value of each component of NEPENTHE
This paper presents NEPENTHE, a pruning method designed to reduce the depth of over-parameterized deep neural networks, addressing their computational inefficiency in real-time applications. Unlike traditional pruning that focuses on reducing model width, NEPENTHE uses an entropy-based unstructured pruning technique to target and remove entire low-entropy layers, effectively shrinking the network's depth without compromising performance. NEPENTHE is validated on models like MobileNet, Swin-T, and RoBERTa and demonstrates that certain layers can be linearized when networks are over-parameterized, leading to reduced computational demands while retaining accuracy.
- I find the model types studied to to limited. Since overparameterization is most prelevant in modern decoder-only language models, I think studying this approach eg: on a Llama-3.1-8B and larger scales would greatly improve the applicability and impact of the work
- Baselines: The paper mainly compares against Iterative magnitude pruning and comparison with more recent pruning methods is missing [1,2,3,4] and [5,6,7] for large language models. I encourage the authors to also compare to different baselines in terms of compute time taken for pruning and also compare against zero-shot pruning methods (after recovery fine-tuning).
- Given that the method relies on expensive training after every iteration of weight pruning, this makes adopting the method to much larger scales inefficient. Could the authors comment on the scalability of the method?
- Latency gains in practice: Given that NEPENTHE does not guarantee completely sparse prunable layers, hence structured pruning. I find the practical latency gains of the method quite limited, if they cannot be enforced/specified by the user. 

[1] Kurtic, E., Campos, D., Nguyen, T., Frantar, E., Kurtz, M., Fineran, B., Goin, M. and Alistarh, D., 2022. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.0725

[2] Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W. and Cui, L., 2022, June. Width & depth pruning for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 3, pp. 3143-3151)

[3] Ilhan, F., Su, G., Tekin, S.F., Huang, T., Hu, S. and Liu, L., 2024. Resource-Efficient Transformer Pruning for Finetuning of Large Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16206-16215).

[4] Klein, A., Golebiowski, J., Ma, X., Perrone, V. and Archambeau, C., 2024. Structural Pruning of Pre-trained Language Models via Neural Architecture Search. arXiv preprint arXiv:2405.02267.

[5] Sun, M., Liu, Z., Bair, A. and Kolter, J.Z., 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695.

[6] Ashkboos, S., Croci, M.L., Nascimento, M.G.D., Hoefler, T. and Hensman, J., 2024. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024.

[7] Ma, X., Fang, G. and Wang, X., 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36, pp.21702-21720.
Yes
No ethics review needed.
Are there any results on ImageNet-1K with Resnet50 and a ViT model like Vit-B/16?
1. Not so many pruning papers focuses on reducing depth of deep models. Depth reduction can bring significant speedup. 

2. The idea of using entropy for unstructured pruning sounds interesting, esp. the study of the effect of pruning on layer entropy (Sec. 4.2).
The paper presents an iterative pruning method to reduce network depth, based on layer entropy. The layer entropy is based on neuron entropy, which is defined based on the sign of the output activation. They further have an entropy aware pruning score and put it into an iterative pruning algorithm. Experiments on some small-scale datasets on image classification and NLP show the potential merits of the method.
1. Methodologically, the method may have clear flaws.
- The entropy of a neuron is defined based on its sign. I am not sure if this is grounded. For ReLU networks, the neuron's output is nearly always positive, then the entropy is 0, and thus can be removed according to this paper. This is clearly not correct.
- L186: "The output of the i-th neuron is always the same as its input, this neuron can in principle be absorbed by the following layer as there is no non-linearity between them anymore." -- this statement is also baseless. It only applies to ReLu networks, while for networks with other activation functions (such as sigmoid), even when the output of a neuron is always positive, the nonlinearity cannot be omitted.
- L198 - 205, the method derivation is based on weight Gaussian and input Gaussian assumptions. I am not sure they really hold in today's practical models.


2. As pointed in the paper, the method needs iterative pruning which is very costly. And this invites the comparison fairness problem. For many pruning methods, they do not need so many training epochs. And we know the training epochs will impact the performance significantly. The question is, if the other methods, equipped with the same long training epochs, will they just compete the proposed method?


3. Experiments
- Some of the results look strange. Tab. 1, why resnet18 on cifar10 only has 91.66% top1 accuracy? ResNet56 can easily get to 93.5% and ResNet18 is designed for ImageNet, which has many more params than ResNet56. 
- Most of the compared methods are baseline approaches, probably implemented by the authors, lacking comparison with more recent papers.


4. (presentation, writing) issues and questions
- Eq 2, missing punctuation.
- One closely related work is missing - https://proceedings.neurips.cc/paper/2016/hash/41bfd20a38bb1b0bec75acf0845530a7-Abstract.html
- the function Weights to prune -> The function
- this neuron can in principle be absorbed by -> This
Yes
No ethics review needed.
Could you provide detailed statistics on the percentage of parameters pruned by the NEPENTHE method compared to other pruning approaches? This information would be valuable for a comprehensive assessment of the method's efficiency and its impact on the overall computational resources required for both training and inference.

How does the removal of different types of layers (e.g., convolutional vs. fully connected) affect the overall network architecture? Are certain layers more "entropy-critical" than others?

How do you choose which samples to use to compute the probability of being on or off of each neuron?

Does the reduced depth network generalize to different dataset as in the lottery ticket hypothesis?

References:
Frankle, Jonathan, and Michael Carbin. "The lottery ticket hypothesis: Finding sparse, trainable neural networks." arXiv preprint arXiv:1803.03635 (2018).

ElAraby, Mostafa, Guy Wolf, and Margarida Carvalho. "OAMIP: optimizing ANN architectures using mixed-integer programming." International Conference on Integration of Constraint Programming, Artificial Intelligence, and Operations Research. Cham: Springer Nature Switzerland, 2023.
Innovation in Pruning: The approach is novel in its use of entropy as a criterion for pruning, shifting the focus from width reduction to depth reduction, which is less explored.
Comprehensive Evaluation: The method is thoroughly tested across several datasets and architectures, providing a robust validation of its effectiveness.
Preservation of Performance: NEPENTHE successfully demonstrates the preservation of model performance even with significant reductions in network depth, addressing a common challenge in neural network pruning.
The paper presents NEPENTHE, an approach aimed at reducing the depth of over-parametrized deep neural networks (DNNs) to decrease computational demands. This method uses an entropy-based strategy to prune layers that exhibit low entropy, allowing for the removal of entire layers from the network with minimal performance loss. The approach is validated on architectures like MobileNet, Swin-T, and RoBERTa.

 NEPENTHE utilizes unstructured pruning that is guided by the entropy of neuron activation within each layer. By re-weighting the pruning process based on entropy levels, the method prioritizes the removal of layers with the lowest entropy. The technique is supported by both theoretical findings and empirical results, showcasing effective reduction of network depth without significant degradation in model performance.
Complexity in Implementation: The entropy-based pruning requires careful calibration and may introduce complexity in tuning the parameters for optimal performance across different architectures.
Limited Discussion on Scalability: The method's scalability to extremely large networks.
The function weights_to_prune which includes the whole logic behind the pruning idea is not explained in the main paper.
Yes
No ethics review needed.
For Table 2, did authors reimplemented other techniques for comparison, or took numbers from external source?
- Layer pruning has its benefits such as speeding up the model on GPUs. Other techniques like unstructured pruning does not.
- The approach is relatively simple according to equations, and requires measuring "frequency" of activations, with later normalization.
The paper proposes a method to perform pruning of trained neural network. The approach is based on per layer entropy and removes entire layers to target a faster model. The paper connects entropy based pruning and weight magnitude pruning, and arrives into a single criteria per layer. The method requires training of the model.
- The approach is tested on out-dated models and datasets and as a result is compared with non so recent models. 
- Depth pruning is recently studied in LLMs. Most approaches don't require fine-tuning. The proposed approach should be compared to those to demonstrate that the metric is better. 

Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853, 2024.

Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834, 2024.

Siddiqui, Shoaib Ahmed, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, and Pavlo Molchanov. "A deeper look at depth pruning of LLMs." arXiv preprint arXiv:2407.16286 (2024).
@inproceedings{
anonymous2024nepenthe,
title={{NEPENTHE}: Entropy-Based Pruning as a Neural Network Depth's Reducer},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=fk5ePN7YCS},
note={under review}
}
While deep neural networks are highly effective at solving complex tasks, their computational demands can hinder their usefulness in real-time applications and with limited-resources systems. Besides, it is a known fact that, for many downstream tasks, off-the-shelf models are over-parametrized. While classical structured pruning can reduce the network's width, the computation's critical path, namely the maximum number of layers encountered at forward propagation, apparently can not be reduced.

In this paper, we aim to reduce the depth of over-parametrized deep neural networks: we propose an e**N**tropy-bas**E**d **P**runing as a n**E**ural **N**etwork dep**TH**'s r**E**ducer (NEPENTHE) to alleviate deep neural networks' computational burden.
Based on our theoretical finding, NEPENTHE leverages "unstructured'' pruning to bias sparsity enhancement in layers with low entropy to remove them entirely. We validate our approach on popular architectures such as MobileNet, Swin-T and RoBERTa, showing that, when in the overparametrization regime, some layers are linearizable (hence reducing the model's depth) with little to no performance loss. The code will be publicly available upon acceptance of the article.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Pruning Compression Entropy Deep Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e3d7f604311b99a4c31499ac31407aa1b67de1ba.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/97ca6277b03387d58a4655a64c914a1a92ca20e9.zip
NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission