Convex Distillation: Efficient Compression of Deep Networks via Convex Optimization
Convex Neural Networks;Convex/Non-Convex Optimization;Knowledge Distillation;Model Compression;Label-Free Training;Classification
Yes
No ethics review needed.
See the weakness.
- Paper is written well and easy to follow.

- The idea of bridging convex optimization, distillation and compression is interesting.
This paper introduces Convex Distillation, a model compression method that replaces the non-convex layers of deep neural networks with convex approximations. By leveraging convex optimization, the method achieves efficient compression without the need for labeled data or post-compression fine-tuning.
- Practical contributions of convex optimization in model compression are limited. The convexity conversation is only valid and tested up to 3-layer DNNs. It significantly restricts the objective landscape. For simple tasks, it might be fine, while for more complex tasks, it often leads sub-optimal performance. 

- Experimental results are not satisfactory to justify the efficacy of the proposed methods. Only small datasets are included. Meanwhile, the ResNet18 baseline seems not well tuned (with low acc less than 90%).
Yes
No ethics review needed.
See Weakness, is there a more reasonable network configuration, more comprehensive experiments?
-  The paper introduces a novel approach to knowledge distillation that leverages convex optimization for efficient compression of deep neural networks. 

- The authors provide extensive empirical evidence to support their claims, demonstrating the effectiveness of their method across multiple standard datasets and in various scenarios.

- The author provided Google Colab code with very detailed experimental instructions.
This article proposes a novel distillation technique that efficiently compresses deep neural network models through convex optimization. This method eliminates intermediate non-convex activation functions and uses only the intermediate activations of the original model, enabling distillation without the need for labeled data, and achieving comparable performance to the original model without fine-tuning. Experimental results show that this approach not only maintains model performance when compressing image classification models on multiple standard datasets but also performs better and optimizes faster compared to traditional non-convex distillation methods. This work opens up new avenues for future research at the intersection of convex optimization and deep learning.
- Using existing convex neural network packages, there is a lack of originality and workload.

- There is an issue with the network configuration. For datasets with small image sizes like CIFAR-10, the configuration used for ResNet on ImageNet should not be applied. It should not downsample by 4x from the start, which results in feature maps that are too small.

- The experiments were only conducted on small datasets and very small networks. Can they be scaled up to larger datasets such as ImageNet?
Yes
No ethics review needed.
See the weaknesses.
1.A simple yet effective to distill classification models via convex networks

2.An effective distillation acceleration tool and polishing are used to improve convex solver.
This paper proposes a convex distillation method by combining the representational power of large non-convex DNNs with the favorable optimization landscape of convex NNs. The proposed method can distill the models in a label-free manner without requiring post-compression fine-tuning on the training data. Experiments on several image classification datasets show that convex student models can achieve high compression rates without sacrificing accuracy and outperform non-convex compression methods in low-sample and high-compression regimes.
1.Activation matching is not novel for knowledge distillation.

2.Experimental comparison is not sufficient to support the effectiveness of the proposed method. It lacks SOTA KD methods for fair comparison.

3. It is not clear that how to distill all blocks. If the proposed convex distillation performs block-wise distillation, it requires a complex and time-consuming knowledge distillation for handling the entire networks.
Yes
No ethics review needed.
1. Why “while DNNs have the capacity to memorize the training dataset, they often end up learning basic solutions that generalize well to test datasets” can “motivate using smaller, compressed models”?
This manuscript is very clear about the background knowledge and the motivation for undertaking this work is clear.
This paper presents a new distillation method that efficiently compresses the model by convex optimization-eliminating intermediate non-convex activation function.
1.	In the section on related work, there is a lack of information on the most recent work, and the related work is introduced too little.
2.	The notation in Eq. 1 and Eq. 3 is used incorrectly ($D_i \in {D}' \in \mathcal{D}_x$).
3.	Some of the textual content in the figures is too small.
4.	The innovative content of the article is not sufficient.
5.	In the experimental part, there is a lack of validation results on large datasets such as ImageNet. Also, using only ResNet18 and MobileNet V3 for experiments is not convincing enough.
6.	The results in Fig. 4 do not intuitively show the superiority of the proposed approach.
7.	There is a lack of experiments to compare with other methods, only ablation experiments are performed.
We introduce a convex neural network distillation method that compresses large pre-trained deep networks without requiring labeled data or fine-tuning, combining convex and non-convex model architectures for efficient deployment on edge devices.
@inproceedings{
anonymous2024convex,
title={Convex Distillation: Efficient Compression of Deep Networks via Convex Optimization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=XCugWIuHR8},
note={under review}
}
Deploying large and complex deep neural networks on resource-constrained edge devices poses significant challenges due to their computational demands and the complexities of non-convex optimization. Traditional compression methods such as distillation and pruning often retain non-convexity that complicates fine-tuning in real-time on such devices. Moreover, these methods often necessitate extensive end-to-end network fine-tuning after compression to preserve model performance, which is not only time-consuming but also requires fully annotated datasets, thus potentially negating the benefits of efficient network compression. In this paper, we introduce a novel distillation technique that efficiently compresses the model via convex optimization -- eliminating intermediate non-convex activation functions and using only intermediate activations from the original model. Our approach enables distillation in a label-free data setting and achieves performance comparable to the original model without requiring any post-compression fine-tuning.  We demonstrate the effectiveness of our method for image classification models on multiple standard datasets, and further show that in the data limited regime, our method can outperform standard non-convex distillation approaches. Our method promises significant advantages for deploying high-efficiency, low-footprint models on edge devices, making it a practical choice for real-world applications. We show that convex neural networks, when provided with rich feature representations from a large pre-trained non-convex model, can achieve performance comparable to their non-convex counterparts, opening up avenues for future research at the intersection of convex optimization and deep learning.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Convex Neural Networks Convex/Non-Convex Optimization Knowledge Distillation Model Compression Label-Free Training Classification
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5d231f615d031c3b51f941bb4228678562756888.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/4b6b67ae1d366aadbce0f65d698fac8849f1eb0d.zip
Convex Distillation: Efficient Compression of Deep Networks via Convex Optimization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission