CoNNect: A Swiss-Army-Knife Regularizer for Pruning of Neural Networks
Connectivity;Regularization;Pruning
Yes
No ethics review needed.
1. How are the theoretical results reflected in the experimental results?
A more through analysis perhaps can give more insight about the impact of 
the regularizer.
1. A proof that the approach approximated the L0 regularizer  and guarantees maximally connected
structures as a stationary points.
The focus of the paper is to develop a pruning strategy for a sparse network. This has been
achieved by the differentiable regularizer. The approach maintains connectivity and prevents
layer collapse. This is an important and topical area of research. The results from the approach 
in the paper, named CoNNect has been compared with L1 norm regularizer and used for one-shot
pruning in LLMs.
1. Several research work on pruning has aimed at and succeeded in accomplishing 
even more successfully what has been achieved here.
2. A comparison with the existing body of literature is needed as maximal connectivity, 
avoiding layer collapse and zero shot pruning have also been addressed previously. 
So the contribution of this work in the correct context is not emphasized.
3. The CoNNect is only compared with magnitude pruning and Synflow. Other pruning
approaches have not been considered for comparison.
4. The experimental results in Table 2 doesn't show any significant difference
with CoNNect as compared to others. The performance also falls significantly 
with pruning.
Yes
No ethics review needed.
My main concerns and suggestions are already mentioned in the weaknesses section. I would like to mention some minor points here.

1. Time/space complexity analysis could be provided.

2. The presentation could be improved. I think Figure 2 could serve as a perfect illustration of intuition, thus I suggest drawing a similar one in the introduction part.
1. It is a novel use of Katz centrality to enhance pruning by maintaining connectivity, which is an overlooked factor in neural network pruning. This approach ensures that information can effectively flow from input to output layers, addressing the common issue of layer collapse in sparse networks.

2. The authors develop a strong theoretical foundation, proving that CoNNect approximates L0 regularization while preventing collapse, which strengthens the validity of their approach.

3. Through comprehensive experiments, the authors provide evidence of CoNNect’s superiority over standard L1- and L2-based regularization techniques. The approach's effectiveness in CV and NLP benchmarks demonstrates the practical impact of this research.

4. The writing is clear and well-structured, making it easy to follow the authors’ arguments and contributions.
This paper studies the problem of neural network pruning. The authors propose two axioms for pruning: reducing the NNs' size by removing unnecessary weights and preserving their connectivity to ensure stable information flow. To address these two axioms, the authors propose a novel regularizer called CoNNect, which computes a connectivity matrix based on Katz centrality, providing a measure that prefers direct paths over parallel ones. The method demonstrates higher performance in various pruning scenarios, including VGG-11 for CIFAR-10 and large language models (LLMs) like LLaMA-7B for various NLP tasks.
1. The authors should discuss more about the second axiom (Preserve Neural Network Connectivity). Why is preserving connectivity important for pruning? How could keeping the flow of information benefit the performance of the pruned model? More insights on this point would make the paper more convincing.

2. The convergence analysis for regularizer loss in lines 243-280 seems impractical and unnecessary, since the parameters not only receive gradients from the regularizer but also from the main task loss. Then these two gradients will added together. With this different gradient, will it still converge to minimal point of regularizer loss?

3. This work does not apply to some important architectures like resnet with residual connections. The authors could provide some discussions on possible solutions for these architectures.

4. The experiments are not extensive. If the authors could provide more results on some modern CV models and datasets, it would be more convincing.
Yes
No ethics review needed.
Please refer to weaknesses. Because the reviewer is not an expert, the novelty and the correctness of the proofs cannot be evaluated.
1. The paper is overall well-written and logically well-organized.
2. The CoNNect is proven to approximate the L0-norm regularization, and the pruned neural network architecture guarantees the maximal connected network structure, which seems solid.
3. The CoNNect is differentiable and can be optimized by gradient descent.
The paper introduces a new neural network pruner CoNNect that focuses on maintaining connectivity between network layers. CoNNect achieves this by leveraging Katz centrality to ensure sparse yet fully connected networks. The CoNNect regularizer is a differentiable and effective surrogate for the L0 norm, promoting sparse architectures while preserving crucial connections. CoNNect avoids issues like layer collapse by maintaining a stable structure, beneficial for both unstructured and structured pruning. Experiments demonstrate that Connect performs well on the LLMs.
1. In the introduction, the author highlights that neural network connectivity is important for good running. However, the benefits of maintaining such connectivity for pruning are unclear to readers. It would be better if the authors could illustrate this further.

2. In Section 3.3.1, the motivation to use Katz centrality for neural network pruning can be illustrated better. Specifically, while there are many centrality measures in network analysis, it is unclear why Katz centrality stands out.

3. In Section 4.1, is it necessary to prune a small MLP model for evaluation?

4. Some typos, e.g. caries -> carries in Line 348.
Yes
No ethics review needed.
1. Can the authors provide the detailed implementation of the calculation of the total conectivity regularizer Eq. (3) for mainstream architectures (e.g. VGG, ResNet, Transformer, UNet)? Do we need to parse the network into an adjacency matrix representation, or there is a more efficient and practicable implementaion?

2. Can the authors compare the FLOPs and inference acceleration of the pruned models against the baseline methods? Can the authors provide a complexity analysis of the proposed algorithm?
1. The authors introduce a principled algorithm (CoNNect regularizer) to preserve the output-input layer connectivity during pruning.
2. The authors provide theoretical guarantee on the sparsity of CoNNect minizer, and characterize its stationary points.
3. The authors validate the effectiveness of CoNNect on MLP, VGG, and LLaMA-transformer.
This work proposed CoNNect, a regularization-based neuron connectiviy-preseving neural network pruning algorithm. Specifically, the authors consider the computation graph of neural network as a directed weighted graph, where the each node represents a neuron, each weight parameter represents a weighted-edge. The CoNNect regularizer penalizes the the negative logarithm of the connectivity between the input layer and the output layer. The authors proved that 1) Theorem 1: CoNNect does incur sparsity, 2) Theorem 2: the stationary points of CoNNect regularizer are its global minimizers. The authors validate the effectiveness of CoNNect on MLP on synthesis regression data, VGG-11 on CIFAR-10, and LLaMA-7B on multiple datasets.
1. The connectivity analysis did not explicitly consider the residual connection structure, which is ubiquitous in mainstream machine learning models, e.g. ResNet, UNet, and Transformers. When the residual connection is involved, the input and output features are ensured to be connected. In this case, it seems that existing pruning methods can still enjoy a satisfactory connectivity between the input and output layer. I wonder to what extend does the residual connection structure overshadow the necessity and benefit of the proposed CoNNect regularizer (which may hinder the training process).

2. Beside the standard gradient-based $L_1$ regularization, the authors are recommended to compare their method with improved $L_1$ optimization methods, e.g. Lasso shrinkage operator-based optimization [1] and Spred-$L_1$ [2].

3. According to Figure 4, it seems that the CoNNect regularizer hinders the train from scratch process of the model, as the `No Reg. w/ Tun` consistently outperforms the `CoNNect Reg. w/ Tun.` until a pruning ration below $0.8$.

4. Minor Typo: 

   a) In Page 4, lines 207-210, should $V_k$ be captalized as $V_K$?

   b) Page 6, lines 311-312, magnetude-based.



[1]. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288.

[2]. Liu Ziyin and Zihao Wang. 2023. Spred: solving L1 penalty with SGD. (ICML'23), Vol. 202.
Yes
No ethics review needed.
I would like to see more comprehensive comparisons with other recent pruning techniques for large language models (LLMs). For instance, the following two papers report higher accuracies at more aggressive levels of pruning, as evidenced by Table 23 in the *Wanda* paper (the second in the list):

1. **SparseGPT: Massive language models can be accurately pruned in one-shot** (Elias Frantar and Dan Alistarh, ICML 2023)
2. **A simple and effective pruning approach for large language models** (Sun et al., ICLR 2024)

Incorporating these methods into the experimental results would provide a clearer picture of how **CoNNect** performs relative to state-of-the-art pruning techniques for LLMs, particularly when it comes to accuracy and the extent of pruning.

Additionally, the experiments in the current paper are not sufficiently comprehensive. The results are reported for only one LLM architecture and one model size. However, in empirical studies of this nature, it is common practice to evaluate multiple model sizes across different architectures to ensure generalizability. Expanding the range of experiments would help substantiate the claims made about CoNNect’s effectiveness and scalability.
1. **Originality**: 
   The paper introduces an original approach by leveraging Katz centrality, a concept from graph theory, to measure and preserve neural network connectivity during pruning. 

2. **Quality**: 
The authors establish the stability and effectiveness of CoNNect in approximating L0 regularization while avoiding layer collapse. 

3. **Clarity**: 
   The paper is fairly well-written and structured.

4. **Significance**: 
The CoNNect regularizer directly tackles the limitations of L1 regularization and other sparsity-inducing methods, which often lead to disconnected or underperforming models. By maintaining network connectivity, CoNNect ensures that pruned models retain their predictive power and functional integrity.
The paper introduces a novel differentiable regularizer, CoNNect, for training sparse neural networks. Inspired by Katz centrality from graph theory, CoNNect is designed to enhance neural network pruning techniques while maintaining key connectivity between the input and output layers throughout training. Traditional pruning methods often rely on L1-regularization as a surrogate for L0-norm, but these methods may lead to issues like layer collapse and loss of network structure. CoNNect addresses these limitations by ensuring the network retains essential paths, thereby preserving the model’s performance during pruning.

The paper demonstrates CoNNect's ability to approximate L0-regularization more effectively than L1-regularization, leading to improved sparsity without sacrificing connectivity. It is suitable for both unstructured and structured pruning techniques. The authors validate CoNNect's performance across several experiments, including channel-level pruning in CNNs and one-shot pruning for large language models, demonstrating improved performance compared to existing pruning techniques.
1. **Complexity of Implementation**:
   The complexity of implementing the CoNNect regularizer in practice. The reliance on Katz centrality, which is not typically a standard tool in neural network training, introduces additional computational overhead. Furthermore, the paper does not provide sufficient guidelines or discussion about how CoNNect scales with larger and more complex neural network architectures, particularly in real-world environments.

2. **Lack of Comprehensive Comparisons**:
   While the paper demonstrates that CoNNect outperforms traditional L1 and L2 regularization and some pruning methods like SynFlow, it lacks a comparison with other contemporary pruning techniques. For instance, techniques like Lottery Ticket Hypothesis-based pruning (Frankle & Carbin, 2019), DLTH  (Bai et al. 2022),  or other structured pruning methods using adaptive sparsity (e.g., Movement Pruning by Sanh et al., 2020) are notably absent. To strengthen the empirical validation, the authors should include experiments that compare CoNNect with these other pruning frameworks.

3. **Limited Architectural Diversity in Experiments**:
   The paper focuses mainly on feedforward networks, convolutional neural networks (CNNs), and large language models (LLMs), but does not include experiments on other widely-used architectures such as recurrent neural networks (RNNs), transformers, or graph neural networks (GNNs). Since the authors state that CoNNect can be applied across various neural network architectures, demonstrating its effectiveness on a broader range of models would help substantiate this claim. 

4. **Insufficient Exploration of Hyperparameters**:
   The paper uses fixed hyperparameter values for CoNNect across different experiments, but there is limited discussion on how sensitive CoNNect is to these hyperparameters or how they should be tuned for different models or datasets. Given that regularization methods can be highly sensitive to parameter tuning, especially in complex neural networks, it would be beneficial for the authors to conduct an ablation study or sensitivity analysis on the regularization coefficients used in CoNNect.
This paper presents CoNNect, a novel regularizer for efficiently inducing sparsity in (practical scale) neural networks that approximates the $L_0$ norm and outperforms $L_1$ and $L_2$ regularization.
@inproceedings{
anonymous2024connect,
title={Co{NN}ect: A Swiss-Army-Knife Regularizer for Pruning of Neural Networks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Se2aTG9Oui},
note={under review}
}
Pruning encompasses a range of techniques aimed at increasing the sparsity of neural networks (NNs). These techniques can generally be framed as minimizing a loss function subject to an $L_0$-norm constraint. In this paper, we introduce CoNNect, a novel differentiable regularizer for sparse NN training, inspired by Katz centrality, which measures connectivity in weighted graphs. Unlike $L_1$-regularization, which is often used as a surrogate for $L_0$-norm regularization, CoNNect ensures that neural networks maintain connectivity between the input and output layers throughout training. We prove that CoNNect effectively approximates $L_0$-regularization and guarantees maximally connected network structures as stable stationary points, avoiding issues such as layer collapse. Our theoretical and numerical results demonstrate that CoNNect outperforms $L_1$-norm regularization. Moreover, we show that CoNNect is applicable to both unstructured and structured pruning, and further validate its scalability and effectiveness through improved one-shot pruning performance in large language models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Connectivity Regularization Pruning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/77c3049fc7e6898050f13c509be5743a5b3df8b6.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/919e3cc676e68c44b352150db656b7cc645b8144.zip
CoNNect: A Swiss-Army-Knife Regularizer for Pruning of Neural Networks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission