HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning
Structured Pruning;Training;AutoML
Dear reviewers and ACs,

We greatly thank all the reviewers for the careful review and for providing insightful and constructive feedbacks. We believe that incorporating these valuable suggestions could significantly improve the quality of our paper.  We are also happy to see that the significance and practical impacts of the work, writing quality, structural logics, experiment broadness, and theoretical boundness have been positively recognized by the reviewers. 

At present, we are working on revising the manuscript and will tackle all raised concerns and questions within this discussion period, such as including more recent baselines and more comparisons. 

Look forward to our further discussion.

Thanks all again!

Best,

Authors of Paper 3237
Thank all reviewers for their careful reviews and insightful feedbacks.
Yes
No ethics review needed.
1. What is the computational cost of employing HESSO? I am wondering whether it will significantly increase the computational memory and flow training speed.
2. Seems CIRC can be used as a generic module to identify the indispensable structures. Can it be applied to other pruning algorithms and what is the performance?
1. This paper is well written and organized.
2. This paper provides thorough experiments, by evaluating the performance of HESSO on various tasks and comparing with many other pruning methods.
3. It is demonstrated that HESSO is able to achieve strong and impressive results with a considerable pruning ratio.
This paper proposes Hybrid Efficient Structured Sparse Optimizer (HESSO), a structured pruning algorithm. In addition, for reliably identifying the indispensable structure, the authors introduce Corrective Redundant Identification Cycle (CRIC). The experimental results show that HESSO with CRIC plugged in can perform well on various types of models, e.g., image classification, object detection and large language models.
See questions.
Yes
No ethics review needed.
Please see the weakness part.
1.By simplifying hyper-parameter tuning and explicitly controlling sparsity exploration, HESSO enhances usability, making it accessible to practitioners without deep optimization expertise.

2. The optimizer is designed to be applicable to any DNN architecture and task, increasing its potential impact across various domains and applications.
The paper addresses the challenge of deploying large deep neural networks by proposing **HESSO**, a new optimizer that enables automatic one-shot joint training and structured pruning for various DNN architectures and tasks. HESSO simplifies the hyper-parameter setup and employs a progressive pruning strategy to explicitly control sparsity exploration, enhancing usability. It optionally integrates a Corrective Redundancy Identification Cycle mechanism to accurately identify redundant groups, minimizing the risk of irreversible performance degradation caused by pruning indispensable structures.  The authors have conducted experiments on various type of tasks and models.
1. As a paper on structured pruning, it does not compare with state-of-the-art methods on ImageNet, such as OFA, EagleEye ResRep, etc. The comparisons are too simplistic, making it impossible to see the advantages of the proposed method. Furthermore, in experiments on large models, it only compares with SliceGPT, which is not tailored for structured pruning. More mainstream baseline methods like LLMPruner need to be included for comparison. Overall, the experimental section significantly dampens my enthusiasm for this paper.

2. The authors' discussion of related work in the field is rather lacking and not sufficiently grounded. The entire introduction, methodology, and experimental discussions focus only on OTO-based papers[4,5,6]. However, as far as I know, there are many papers on automatic structure search for structured pruning, including OFA[1], EagleEye[2], etc. I suggest that the authors provide a more comprehensive coverage of the field, which would effectively enhance the quality of this paper.

[1] Cai, H., Gan, C., Wang, T., Zhang, Z. and Han, S., 2019. Once-for-all: Train one network and specialize it for efficient deployment. In ICLR, 2020.

[2] Li, B., Wu, B., Su, J. and Wang, G., 2020. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In ECCV, 2020.

[3] Ding, X., Hao, T., Tan, J., Liu, J., Han, J., Guo, Y. and Ding, G., 2021. Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In ICCV, 2021.

[4] Chen, T., Ji, B., Ding, T., Fang, B., Wang, G., Zhu, Z., Liang, L., Shi, Y., Yi, S. and Tu, X., 2021. Only train once: A one-shot neural network training and pruning framework. In NeurIPs, 2021.

[5] Chen, T., Liang, L., Ding, T., Zhu, Z. and Zharkov, I., 2023. Otov2: Automatic, generic, user-friendly. In Arxiv, 2022.

[6] Chen, T., Ding, T., Zhu, Z., Chen, Z., Wu, H., Zharkov, I. and Liang, L., 2023. OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators. In Arxiv, 2023.
Yes
No ethics review needed.
1. How does HESSO manage the trade-off between sparsity and performance? Is there a recommended range for optimal sparsity levels?
2. Is there a risk that the method could lead to overfitting, especially during the pruning phase? Maybe evaluation in held-out datasets are needed.
1. This paper is well-motivated, and the topic is highly relevant in the current AI landscape, where deploying efficient models in resource-constrained environments is critical. 
2. The manuscript provides a well-structured theoretical framework for HESSO, which helps justify the design choices made in the method.
3. The authors conduct extensive experiments in an array of tasks.
The manuscript presents a novel approach to deep neural network (DNN) training and structured pruning through Hybrid Efficient Structured Sparse Optimize (HESSO). The topic is highly relevant in the current AI landscape, where deploying efficient models in resource-constrained environments is critical. HESSO stands out by simplifying hyper-parameter setup and offering a progressive pruning strategy that enhances user-friendliness. In addition, the authors also discuss the limitations of the proposed method and the approximation errors of salience scores.
1. While HESSO aims to simplify hyper-parameter tuning, the manuscript does not provide sufficient evidence to support the claim of reduced sensitivity. A comparative analysis of tuning efforts required for HESSO versus traditional methods is needed.
2. The scalability of HESSO to larger models is not well demonstrated. A table or figure listing the computational cost of applying HESSO to models across different model sizes should be provided.
Yes
No ethics review needed.
1. In line 215, the authors state, “The selection of the saliency score in HESSO is flexible and can be tailored to different purposes.” I would like to know how the authors select the saliency score for different tasks in Sections 4.2-4.6. 
2. In lines 503-507, the authors state, “Since both HESSO and HESSO-CRIC … without requiring tensor parallelism.” Does this mean that the HESSO method does not support models larger than 3 billion parameters? In addition, since both HESSO and HESSO-CRIC utilize full gradient information of LLMs, I would like to know the training cost of HESSO and HESSO-CRIC on LLMs.
[1] Structured Pruning Learns Compact and Accurate Models. ACL 2022 Main.
[2] Auto-Train-Once: Controller Network Guided Automatic Network Pruning
from Scratch. CVPR 2024.
[3] LLM-Pruner: On the Structural Pruning of Large Language Models. NeurIPS 2023.
[4] LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning. ACL 2024 Findings.
[5] LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery. ArXiv.
[6] OTOv3: Towards Automatic Sub-Network Search Within General Super Deep Neural Networks. ArXiv.
1. The paper is well-written, with clear organization and rigorous logic, making it understandable to readers.
2. HESSO can automatically and efficiently train and prune DNNs with any architecture, providing competitive performance, which holds broad significance in the field of neural network pruning.
3. The authors conducted a comprehensive analysis of how approximation errors of saliency score can lead to misidentification of indispensable structures in pruned DNNs, resulting in performance collapse. This part is well-organized, highly interesting, and very insightful.
This paper proposes HESSO, a novel optimizer designed for structured pruning of DNNs. HESSO aims to automatically and efficiently train any DNN to produce high-performing sub-networks with minimal need for hyper-parameter tuning, enhancing user-friendliness. Additionally, this paper introduces an extension of HESSO called HESSO-CRIC to address issues related to the irreversible performance collapse that sometimes occurs in pruned models. HESSO and HESSO-CRIC are evaluated across various applications, including computer vision and natural language processing, where it demonstrates competitive performance compared to existing baselines.
1. I believe the main issue with this paper is the lack of many necessary comparison baselines, or the existing comparison baselines are outdated.
1) Section 4.3: In the CoFi [1] method, BERT-Base achieves a lossless F1 score on the SQuAD dataset with a 70% reduction in parameters (Figure 2 in CoFi). However, in the HESSO method, BERT-Base shows a drop of about 4% in F1 score with approximately a 50% reduction in parameters.
2) Section 4.4: HESSO seems to be inferior to the ATO [2] method, as ATO achieved a TOP-1 accuracy of 76.07% with a 61.7% reduction in FLOPs, only a -0.06% drop compared to the baseline. However, under the same FLOPs, the accuracy of HESSO is below 76%.
3) Section 4.6: The authors only compare with the SliceGPT method, lacking comparisons with some important structured pruning baselines for LLMs, such as LLM-Pruner (with LoRA fine-tuning) [3], LoRAPrune [4] and LoRAShear [5].
Due to the lack of the above baselines, I have concerns about the effectiveness of the HESSO method.
2. It seems that HESSO is simply improved version of OTO v3. As far as I know, OTO v3 also proposes a hybrid training scheme that applies different update mechanisms to important and redundant variable groups. HESSO does not seem to introduce any significant innovations but rather represents a minor technical improvement over OTO v3.
3. Different saliency scores likely result in varying approximation errors. The authors should demonstrate whether the CRIC mechanism effectively corrects each type of approximation error. Therefore, the authors should show the impact of different saliency scores in HESSO-CRIC on the final accuracy.
4. The phrase "can not be recovered given user resources" in Definition 3.1 is confusing, as it is evident that models with different parameters require significantly different training resources. The authors should clarify the definition of "user resources" more explicitly.
@inproceedings{
anonymous2024hesso,
title={{HESSO}: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=LXlTdn9hY9},
note={under review}
}
Structured pruning is one of the most popular approaches to effectively compress the heavy deep neural networks (DNNs) into compact sub-networks while retaining the original network performance. The existing methods suffer from multi-stage procedures along with significant engineering efforts and human expertise. The Only-Train-Once series (OTOv1-v3) has been recently proposed to resolve the many pain points by streamlining the workflow. However, the built-in sparse optimizers in the OTO series, i.e., the Half-Space Projected Gradient (HSPG) family, have limitations that require hyper-parameter tuning and the implicit controls of the sparsity exploration, consequently requires intervening by human expertise. To further address such limitations, we propose a novel Hybrid Efficient Structured Sparse Optimizer (HESSO). HESSO could automatically and efficiently train a DNN within a single run to produce a high-performing sub-network. Meanwhile, it is almost tuning-free and enjoys user-friendly integration for generic training applications. To address another common issue of irreversible pruning performance collapse observed in some DNNs, we further propose a novel Corrective Redundant Identification Cycle (CRIC) to plug into HESSO for reliably identifying indispensable structures. We numerically demonstrate the efficacy of HESSO and its enhanced version HESSO-CRIC on a variety of applications ranging from computer vision to natural language processing, including large language model. The numerical results showcase that HESSO can achieve competitive performance to varying state-of-the-art benchmarks and support most DNN architectures. Meanwhile, CRIC can effectively prevent the irreversible performance collapse and further enhance the performance of HESSO on certain applications.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Structured Pruning Training AutoML
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/90c564634082f8c8c8876b9d9290b56459a64e92.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/63b6fcbcbca4a1f519dd5b8b20c1e570e1d287e7.zip
HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission