Multi-loss-aware Channel Pruning of Deep Networks
Yiming Hu Siyang Sun Jianquan Li Jiagang Zhu Xingang Wang Qingyi Gu



Query Term Weighting based on Query Performance Prediction
    
  Haggai Roitman

  IBM Research

  Haifa University Campus

  Haifa, Israel 

   

  
    
===========================================================================================



Channel pruning, which seeks to reduce the model size by removing redundant channels, is a popular solution for deep networks compression. Existing
channel pruning methods usually conduct layer-wise channel selection by directly minimizing the reconstruction error of feature maps between the baseline model and the pruned one. However, they ignore the feature and semantic distributions within feature maps and real contribution of channels to the overall performance. In this paper, we propose a new channel pruning method by explicitly using both intermediate outputs of the baseline model and the classification loss of the pruned model to supervise layer-wise channel selection. Particularly, we introduce an additional loss to encode the differences in the feature and semantic distributions within feature maps between the baseline model and the pruned one. By considering the reconstruction error, the additional loss and the classification loss at the same time, our approach can significantly improve the performance of the pruned model. Comprehensive experiments on benchmark datasets demonstrate the effectiveness of the proposed method.




deep neural networks, object classification, model compression, channel pruning




§ INTRODUCTION



In recent years, deep convolutional neural networks (CNNs) have been widely applied to various computer vision tasks, e.g., image classification, object detection, action recognition, since AlexNet <cit.> won the ImageNet Challenge: ILSVRC 2012 <cit.>. However, it is very hard to deploy these deep models on resource-constrained devices including mobile robots, unmanned aerial vehicles, smartphones, etc. due to their high storage and computational cost.

In order to make CNNs available on resource-constrained devices, many model compression and acceleration methods have been presented including channel pruning <cit.>, network quantization<cit.> and low-rank approximation <cit.>. The method proposed in this paper falls into the category of channel pruning. Different from simply making sparse connections, channel pruning reduces the model size by directly removing redundant channels. In contrast, pruning the entire channel can achieve fast inference without special software or hardware implementation. To prune redundant channels, existing reconstruction-based methods <cit.> usually minimize the reconstruction error of feature maps between the baseline model and the pruned one. DCP <cit.> seeks to conduct channel selection by introducing additional discrimination-aware losses. These methods suffer from two aspects of limitations. First, directly matching feature maps ignores the feature and semantic distributions within them. Specifically, one should be encouraged to focus more on really important channels and spatial activations of feature maps instead of simply aligning every location. Second, the reconstruction error of a layer or additional losses do not always truly reflect the change of the classification loss, which may lead to mistakenly pruning some important channels. 

In this paper, we address these problems from two aspects. First, the feature and semantic correlation loss is defined, through which the pruned model can preserve the feature and semantic distributions within feature maps of the baseline network. Second, we reconsider real contribution of channels to the overall performance. To this end, the classification loss of the pruned model is used to supervise channel selection. Then, we conduct layer-wise channel pruning by considering the reconstruction error, the additional loss and the classification loss at the same time.

Our main contributions are summarized as follows. First, we present a multi-loss-aware channel pruning method for deep networks compression by the introduction of the feature and semantic correlation loss and the classification loss. Second, extensive experiments on benchmark datasets show that the proposed method is theoretically reasonable and practically effective. On CIFAR-10, our approach outperforms the previous state-of-the-art method by 0.55% in accuracy.





§ RELATED WORK


The basic idea of weight pruning is to prune some redundant weights. Han et al. <cit.> proposed a conceptually simple pruning method: weights lower than a certain threshold are considered as low contribution ones that can be pruned, and then fine-tuning is taken for restoring the network accuracy. However, the method needs special software or hardware implementation for fast inference. In view of this issue, Structured Sparsity Learning (SSL) <cit.> imposed regularizations on different levels of structures such as filters, channels or layers. Then weights belonging to the same level of structure would go to zero at the same time, which could be removed. This method obtains extraordinary performance but is computationally expensive and difficult to converge. Afterwards, some inference-based channel pruning methods <cit.> were proposed, and they sought to do channel selection by defining different selection criteria. ThiNet <cit.> explicitly formulated channel selection as an optimization problem, and conducted channel pruning by minimizing the reconstruction error of feature maps between the baseline model and pruned one. DCP <cit.> aimed at selecting the most discriminative channels for each layer by considering both the reconstruction error and the discrimination-aware loss. However, these methods ignore the feature and semantic distributions within feature maps and real contribution of channels to the overall performance.



§ OUR METHOD


For better description of the proposed method, some notations are given first. Considering a layer of the CNN model, X represents the input tensor, W denotes the convolution filter and ⊗ represents the convolution operation. We further use F ∈ℝ^M× HZ to denote output feature maps of the layer. Here, M, H, Z represents the number of output channels, the height and the width of the feature maps respectively.



 §.§ Motivation



Existing methods <cit.> conduct channel pruning by minimizing the reconstruction error of a layer, which can be defined as the Euclidean distance of feature maps between the original model and the pruned one:


    ℒ_r = 1/2T||X⊗ W-X⊗ W_𝒫||^2_2

where T = M × H × Z, 𝒫 denotes the index set of the selected channels, W_𝒫 represents the submatrix indexed by 𝒫. Jiang et al. <cit.> explore the correlation between the layer-wise reconstruction error and the classification loss, and they demonstrate these two variables show positive correlation. However, the correlation coefficient varies with different layers and some layers show low correlation. Consequently, simply minimizing the reconstruction error may mistakenly remove some important channels, even if they are very sensitive to the classification loss. Moreover, assuming each channel of F corresponds to a type of feature and each spatial activation encodes a type of semantic information, the feature and semantic distributions reflect which channels and spatial areas a CNN model focuses on. However, directly matching feature maps is not a good choice to preserve the feature and semantic distributions of the baseline model.

These limitations of existing methods inspire us to define a new loss: the feature and semantic correlation loss. The pruned model can preserve the feature and semantic distributions within feature maps of the baseline network using this loss. Meanwhile, we will explicitly use the classification loss to supervise layer-wise channel selection.





 §.§ Formulation



To transfer rich semantic information of the baseline model, one can align the semantic distribution of two feature maps <cit.>. In this paper, we consider aligning both the feature and semantic distributions by introducing the feature and semantic correlation loss. Here, the loss can be defined as the square error of two Gram matrices:

    ℒ_s = 1/4N^2M^2(||G^f - G^f_𝒫||^2_2 + ||G^s - G^s_𝒫||^2_2)

where N = H × Z, the Gram matrix G^f∈ℝ^M× M and G^s∈ℝ^N× N encode the feature and semantic correlation for a layer of the baseline model respectively. Similarly, G^f_𝒫 and G^s_𝒫 are Gram matrices corresponding to the pruned model. G^f∈ℝ^M× M is computed by the inner product between different vectorized features. G^s∈ℝ^N× N is computed by the inner product between different vectorized spatial activations cross channels:

    G^f_ij = ∑_k=1^NF_ik F_jk, G^s_i^'j^' = ∑_k^'=1^MF_k^'i^' F_k^'j^'


To ensure important channels being kept, we explicitly use the classification loss to supervise layer-wise channel selection. By considering the reconstruction error, the feature and semantic correlation loss and the classification loss at the same time, the problem of channel pruning can be formulated to minimize the following joint loss function:


    min_W_𝒫     ℒ(W_𝒫) = ℒ_r(W_𝒫) + αℒ_s(W_𝒫) + βℒ_c(W_𝒫)
       s.t.    ||𝒫||_0 ≤𝒦

where α and β are two positive coefficients balancing the regularization terms, ℒ_c is the classification loss and 𝒦 is the number of channels to be selected.





 §.§ Optimization Algorithm


It is a combinatorial optimization problem to choose a subset of channels according to Eq. <ref>. Directly searching competitive network structures in the whole solution space is NP-hard. Existing methods <cit.> seek to do channel selection with greedy algorithm by only considering the weight magnitude or the gradient of Eq. <ref> w.r.t. the channel. In this paper, we take both of them into consideration. To be specific, the sensitivity of the k-th channel is defined as: 

    δ_k = ∑_i=1^H∑_j=1^W||ℒ/W_k,i,jW_k,i,j||^2_2

which is the square sum of weights of the k-th channel multiplying gradients of Eq. <ref> w.r.t. the channel. Then we retain the channels with the i largest sensitivity and remove others. After this, Eq. <ref> w.r.t. the selected channels is further optimized according to stochastic gradient descent (SGD). W_𝒫 is updated by:


    W_𝒫 = W_𝒫 - η∂ℒ/∂ W_𝒫

where η represents the learning rate. After updating W_𝒫, the channel pruning of a single layer is finished. The pruning process of the whole model is described in Algorithm <ref>.











§ EXPERIMENTS



In this section, our approach is evaluated on CIFAR-10 <cit.> and CIFAR-100 <cit.>. The proposed method is compared with several state-of-the-art methods including ThiNet <cit.>, CP <cit.> and DCP <cit.>. Our approach is implemented using the PyTorch <cit.> framework. For the experiments on CIFAR-10 and CIFAR-100, we adopt the same experiments setting as in <cit.>. Besides, α and β
(Eq. <ref>) are set to 0.001 and 1 respectively.


 §.§ Overall Comparison Results



To verify the effectiveness of the proposed approach, we conduct comparison experiments with several state-of-the-art methods on VGGNet and ResNet-56 . Table <ref> summarizes overall comparison results on CIFAR-10. It can be observed that the accuracy gap between our baseline model and the comparison reference model is very small. For VGGNet, the proposed approach has a higher speedup and compression rate than other methods with the same accuracy drop. For ResNet-56, our approach achieves the best performance compared with previous state-of-the-art methods. With a compression rate of 1.97× and a speedup of 1.99×, the proposed method outperforms the DCP by 0.55% in accuracy drop. Moreover, our pruned ResNet-56 even outperforms the baseline model by 0.24% in accuracy.



 §.§ Analysis of the Proposed Approach



To study effects of different losses, we prune 30% channels of ResNet-56 on CIFAR-10 with different losses without fine-tuning. As shown in Table <ref>, the test error of the pruned model gets an immediate decline when introducing the feature and semantic correlation loss or the classification loss. It's worth noting that the pruned model obtains the best performance with three losses together. This demonstrates multi-loss fusion is an effective strategy for channel pruning and three losses can complement each other in some aspects. 




Going one step further, we verify the convergence of our method. The loss function in Eq. <ref> is optimized using SGD when pruning a single layer. Fig. <ref> shows three losses (Eq. <ref>) change over epochs for the first twelve layers of VGGNet on CIFAR-100. It can be found that models with different losses can converge to a minor error after 50 epochs. At Conv5-1 and Conv5-2, curves of three losses do not drop after 10 epochs. The above results demonstrate the proposed method has a fast rate of convergence. 

To further explore limits of the proposed approach, we prune ResNet-18 and ResNet-34 with different pruning rate on CIFAR-100. Experimental results in Table <ref> show that the pruned model gets an immediate accuracy drop with increase of pruning rate. Besides, ResNet-18 is more sensitive to channel pruning than ResNet-34 due to its fewer parameters.



§ CONCLUSION



In this paper, a multi-loss-aware channel pruning method is presented for deep networks compression. The proposed approach achieves the state-of-the-art performance on benchmark datasets by considering the reconstruction error, the classification loss and the feature and semantic correlation at the same time. As for future works, we will combine existing pruning strategies with other network compression methods to explore more compact models with less accuracy drop.



§ ACKNOWLEDGMENT

This work has been supported by the National Key Research and Development Program of China No. 2018YFD0400902 and the National Natural Science
Foundation of China (Grant No. 61673376 and 61573349). 





IEEEbib
10
	
	Krizhevsky2012
	Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,
	“Imagenet classification with deep convolutional neural networks,”
	in Advances in neural information processing systems, 2012, pp.
	1097–1105.
	
	Russakovsky2015
	Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
	Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.,
	“Imagenet large scale visual recognition challenge,”
	International Journal of Computer Vision, vol. 115, no. 3, pp.
	211–252, 2015.
	
	He2017
	Yihui He, Xiangyu Zhang, and Jian Sun,
	“Channel pruning for accelerating very deep neural networks,”
	in International Conference on Computer Vision (ICCV), 2017.
	
	Luo2017
	Jian-Hao Luo, Jianxin Wu, and Weiyao Lin,
	“Thinet: A filter level pruning method for deep neural network
	compression,”
	arXiv preprint arXiv:1707.06342, 2017.
	
	jiang2018efficient
	Chunhui Jiang, Guiying Li, Chao Qian, and Ke Tang,
	“Efficient dnn neuron pruning by minimizing layer-wise nonlinear
	reconstruction error.,”
	in IJCAI, 2018, vol. 2018, pp. 2–2.
	
	WangP2018
	Peisong Wang, Qinghao Hu, Yifan Zhang, Chunjie Zhang, Yang Liu, Jian Cheng,
	et al.,
	“Two-step quantization for low-bit neural networks,”
	in Proceedings of the IEEE Conference on Computer Vision and
		Pattern Recognition, 2018, pp. 4376–4384.
	
	Xu2018
	Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin, and Hongkai Xiong,
	“Deep neural network compression with single and multiple level
	quantization,”
	arXiv preprint arXiv:1803.03289, 2018.
	
	Zhou2018
	Aojun Zhou, Anbang Yao, Kuan Wang, and Yurong Chen,
	“Explicit loss-error-aware quantization for low-bit deep neural
	networks,”
	in Proceedings of the IEEE Conference on Computer Vision and
		Pattern Recognition, 2018, pp. 9426–9435.
	
	Wang2017
	Min Wang, Baoyuan Liu, and Hassan Foroosh,
	“Factorized convolutional neural networks.,”
	in ICCV Workshops, 2017, pp. 545–553.
	
	Wang2018
	Wenqi Wang, Yifan Sun, Brian Eriksson, Wenlin Wang, and Vaneet Aggarwal,
	“Wide compression: Tensor ring nets,”
	learning, vol. 14, no. 15, pp. 13–31, 2018.
	
	zhuang2018discrimination
	Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu,
	Junzhou Huang, and Jinhui Zhu,
	“Discrimination-aware channel pruning for deep neural networks,”
	in Advances in Neural Information Processing Systems, 2018, pp.
	883–894.
	
	Han2015
	Song Han, Jeff Pool, John Tran, and William Dally,
	“Learning both weights and connections for efficient neural
	network,”
	in Advances in neural information processing systems, 2015, pp.
	1135–1143.
	
	Wen2016
	Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li,
	“Learning structured sparsity in deep neural networks,”
	in Advances in Neural Information Processing Systems, 2016, pp.
	2074–2082.
	
	Hu2016
	Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang,
	“Network trimming: A data-driven neuron pruning approach towards
	efficient deep architectures,”
	arXiv preprint arXiv:1607.03250, 2016.
	
	huang2017like
	Zehao Huang and Naiyan Wang,
	“Like what you like: Knowledge distill via neuron selectivity
	transfer,”
	arXiv preprint arXiv:1707.01219, 2017.
	
	zagoruyko2016paying
	Sergey Zagoruyko and Nikos Komodakis,
	“Paying more attention to attention: Improving the performance of
	convolutional neural networks via attention transfer,”
	arXiv preprint arXiv:1612.03928, 2016.
	
	Liu2017
	Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui
	Zhang,
	“Learning efficient convolutional networks through network
	slimming,”
	in Computer Vision (ICCV), 2017 IEEE International Conference
		on. IEEE, 2017, pp. 2755–2763.
	
	howard2017mobilenets
	Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang,
	Tobias Weyand, Marco Andreetto, and Hartwig Adam,
	“Mobilenets: Efficient convolutional neural networks for mobile
	vision applications,”
	arXiv preprint arXiv:1704.04861, 2017.
	
	krizhevsky2009learning
	Alex Krizhevsky and Geoffrey Hinton,
	“Learning multiple layers of features from tiny images,”
	Tech. Rep., Citeseer, 2009.
	
	paszkepytorch
	Adam Paszke, Soumith Chintala, Ronan Collobert, Koray Kavukcuoglu, Clement
	Farabet, Samy Bengio, Iain Melvin, Jason Weston, and Johnny Mariethoz,
	“Pytorch: Tensors and dynamic neural networks in python with strong
	gpu acceleration, may 2017” .
	





