Cluster Regularized Quantization for Deep Networks Compression
Yiming Hu Jianquan Li Xianlei Long Shenhua Hu Jiagang Zhu Xingang Wang Qingyi Gu



Domain-Constrained Advertising Keyword Generation
    Hao Zhou^1, Minlie Huang^1,*, Yishun Mao^2, Changlei Zhu^2, Peng Shu^2, Xiaoyan Zhu^1
    December 30, 2023
=========================================================================================



 Deep neural networks (DNNs) have achieved great success in a wide range of computer vision areas, but the applications to mobile devices is limited due to their high storage and computational cost. Much efforts have been devoted to compress DNNs. In this paper, we propose a simple yet effective method for deep networks compression, named Cluster Regularized Quantization (CRQ),  which can reduce the presentation precision of a full-precision model to ternary values without significant accuracy drop. In particular, the proposed method aims at reducing the quantization error by introducing a cluster regularization term, which is imposed on the full-precision weights to enable them naturally concentrate around the target values. Through explicitly regularizing the weights during the re-training stage, the full-precision model can achieve the smooth transition to the low-bit one. Comprehensive experiments on benchmark datasets demonstrate the effectiveness of the proposed method.




deep neural networks, object classification, model compression, quantization




§ INTRODUCTION



In the past few years, DNNs have been widely applied to various computer vision tasks, e.g., image classification, object detection, action recognition. However, it is very difficult to deploy these deep models on resource-constrained mobile devices due to their high storage and computational cost.

In order to make DNNs available on resource-constrained devices, many model compression and acceleration methods have been proposed including weight pruning <cit.>, low-rank approximation <cit.>, and efficient model designs <cit.>. Another line of works <cit.> for network compression is to reduce the presentation precision of the full-precision parameters. Going one step further, restricting weights to some values with special types (e.g., zero or powers of two) can achieve fast inference on embedded devices such as the Field Programmable Gate Array (FPGA). Existing methods <cit.> usually quantize weights by directly approximating the full-precision weights with the low-bit ones. XNOR-Net <cit.> and TWN <cit.> minimize the reconstruction error of the outputs between the full-precision model and the quantized one. These methods suffer from a common limitation: the reconstruction error of weights or intermediate outputs do not always truly reflect the change of the classification loss. Thus, directly cutting or sharing weights according to the reconstruction error of a layer may cause serious quantization error, which would bring significant disturbance to the classification loss and hurt the prediction accuracy of the quantized model.

To address the problem, we introduce an cluster regularization term, through which the full-precision weights can naturally concentrate around the target values during the re-training stage. After this, weights of each layer are quantized into corresponding cluster centers. Consequently, the quantization error can be significantly reduced. 

Our main contributions are summarized as follows. First, we present a new quantization method for deep networks compression by the introduction of the cluster regularization. Second, comprehensive experiments on benchmark datasets show that the proposed method can achieve competitive performance in accuracy and the rate of convergence.



§ RELATED WORK


The goal of network quantization is to replace the full-resolution weights with low-bit ones without significant accuracy drop.  This line of works can be traced back to two representative methods: Expectation Back-Propagation (EBP) <cit.> and BinaryConnect <cit.> in which authors explicitly constrained the weights to either +1 or -1 during propagations. These two methods achieve decent performance on small datasets including MNIST and CIFAR-10, yet they suffer from significant accuracy drop on challenging datasets such as ILSVRC-12 <cit.>. Afterwards, plenty of works were proposed to improve the performance of BinaryConnect. Rastegari et al. presented XNOR-Network <cit.> that approximated the full-precision weights by introducing a scaling factor during binarization. For pursuing higher accuracy, High-Order Residual Quantization (HORQ) <cit.> sought to compensate the information loss of binary quantization by conducting convolutional operations on inputs in different scales and then combined the results. The Ternary Weight Network (TWN) <cit.> introduced zero as a third quantized value and was the first method that achieved decent results on the ILSVRC-12 dataset. Trained Ternary Quantization (TTQ) <cit.> achieved the state-of-the-art performance by taking the scaling factors as trainable parameters. Different from these works, our approach considers further reducing the quantization error to bridge the accuracy gap by introducing the cluster regularization with little computational complexity.



§ OUR METHOD


For better description of the proposed method, some notations are given first. Considering a layer of the DNN model, X represents the input tensor, W denotes the weight of the layer and W_𝒬 represents the quantized weights of the layer.



 §.§ Motivation

Most existing low-bit quantization methods <cit.> can be classified into two categories. The first category such as BinaryNet <cit.> and BinaryConnect <cit.> directly approximates the full-precision weights with the low-bit ones:

    min_W_𝒬 1/2N||W- W_𝒬||^2_2

where N is the size of W. The second category such as XNOR-Net <cit.> and TWN <cit.> usually minimizes the reconstruction error of a layer, which can be defined as the Euclidean distance of the outputs between the full-precision model and the quantized one:

    min_W_𝒬 1/2N||XW-XW_𝒬||^2_2


These methods suffer from a common limitation: the reconstruction error of weights or outputs between the full-precision model and the quantized one don't always truly reflect the change of the classification loss. Thus, directly cutting or sharing weights via the reconstruction error of a layer may cause serious quantization error, which would bring significant disturbance to the classification loss and hurt the prediction accuracy of the quantized model. This inspires us to define an additional regularization term, which is imposed on the weights of the full-precision model to enable them naturally concentrate around the target values by re-training. Particularly, the additional regularization can assist to reduce the quantization error and achieve the smooth transition from the full-precision model to the low-bit one.



 §.§ Formulation

To softly modify the weights distribution by re-training, an cluster regularization term is defined. Han et al. <cit.> apply k-means to assign weights with K clusters layer-wise at first. Then the weights belonging to the same cluster centers share a same value. Unlike them, the values of cluster centers in our method are restricted to T = {-α,0,+α}, where α∈ℝ is a scaling factor. Without loss of generality, assuming W is a row vector in ℝ^1× N and M=[-1, 0, 1]^T is a column vector, where N=c× h × z, c, h and z represents the number of channels, the height and the width respectively. To partition weights W into three clusters whose cluster centers are chosen from T, we solve the following optimization problem:


    J(Z, α)	= ‖ W-α M^TZ ‖ ^2 
    α^*, Z^* = _α, Z J(Z, α) 
    
    s.t.   Z ∈{0, 1}^3× N, ∑_i^3 Z_i,j = 1, ∀ j ∈ [1,N]

where Z ∈{0, 1}^3× N represents the cluster indicator, and the summation of each column is equal to 1. It's worth noting that the optimization problem is different from the traditional k-means, since the cluster centers are restricted to the discrete values chosen from T. 



 §.§ Optimization Algorithm

Eq. <ref> is non-convex, and optimization parameters involve α and Z. We consider using the alternating algorithm to solve this problem, i.e., solve α when Z is fixed and vice versa. 
 

Solve Z with α fixed. By fixing α, Eq. <ref> is written as:

    min_Z  Tr(W^TW) - 2α Tr(W^TM^TZ) + α^2 Tr(Z^TMM^TZ)
    
    s.t.   Z ∈{0, 1}^3× N, ∑_i^3 Z_i,j = 1, ∀ j ∈ [1,N]

Directly solving Z for Eq. <ref> is difficult due to its discrete constraints. Inspired by <cit.>, Z can be optimized column-wise by fixing other columns. In this way, Z can be solved in roughly square time by learning only one column each time.  Solving Eq. <ref> is equivalent to optimizing two following general terms, i.e., Tr(W^TM^TZ) and Tr(Z^TMM^TZ). Let w be the i^th column of W and W̃ indicates the vector of W excluding w, i = 1,2,⋯,N. Similarly, z denotes the i^th column of Z and Z̃ represents the matrix of Z excluding z. Then the following transformation can be got:


    Tr(W^TM^TZ)    =Tr((zw^T+Z̃W̃^T)M^T)
       = C + z^TMw

Here, those terms that don't contain z are set to a constant C. Similarly, we have:


    Tr(Z^TMM^TZ)   =Tr(MM^T(zz^T+Z̃Z̃^T))
       = C + M^Tzz^TM

According to Eq. <ref> and Eq. <ref>, Eq. <ref> can be written as follows:


    min_qα^2q^2 - 2α wq + C 
    
    s.t.   z ∈{0, 1}^3× 1, ∑_i^3 z_i = 1

where q = M^Tz and q, α, w ∈ℝ. Assuming Eq. <ref> is convex, it has the optimal solution when q is equal to w/α. But for this problem, q is discrete. Let H(M,w/α) returns the index of the nearest value of M to w/α, the optimal z is solved as:


    z_i={
    1       i = H(M,w/α) 
    
    0       otherwise
    .

Each column of Z is optimized according to Eq. <ref>. Then Z can be solved after N iterations.


Solve α with Z fixed. Eq. <ref> is convex when Z is fixed. Eq. <ref> has the optimal solution when its derivative w.r.t. α is equal to zero. Thus we have:


    α = WZ^TM/M^TZZ^TM

In this way, α and Z are updated iteratively until convergence. Here, it's not hard to find that solving α via Eq. <ref> is equivalent to computing α as 1/|I|∑_i∈ I |W_i|, where I is the index sets of weights belonging to the cluster centers -α and +α. Thus, solving Eq. <ref> using the alternating algorithm brings little computational complexity. 

As mentioned above, Eq. <ref> is used as a regularization term imposed on W during the re-training stage. Assuming L(W) is the loss function of the full-precision model, the objective function of the re-training stage is defined as follows:


    min_W,Z,α L(W) + λ J(Z,α)
    
    s.t.   Z ∈{0, 1}^3× N, ∑_i^3 Z_i,j = 1, ∀ j ∈ [1,N]

where λ is a positive coefficient balancing the regularization term. Eq. <ref> can be solved by iteratively updating W, Z and α. First, Z and α are solved according to Eq. <ref> - <ref>. Then W is updated by stochastic gradient descent (SGD). We can derive the following weight update method:



    W^t = W^t-1 - η(∂ L/∂ W^t-1 + λ(W^t-1-α M^TZ))

where η is the learning rate. By the interaction among Z, α and W during the re-training stage, the proposed regularization term enables W naturally concentrate around the target values chosen from T.



 §.§ The Whole Quantization Framework

After retraining, weights of each layer are quantized into corresponding cluster centers according to the indicator matrix Z (Eq. <ref>), for parameter sharing. Specifically, the quantized weights are exactly equal to α M^T Z when Eq.<ref> has the optimal solution. In contrast, the re-training strategy with the cluster regularization brings less quantization error than the reconstruction-based methods <cit.>, since the weight W remains in a highly clustered state after re-training. To further reduce the effects of quantization error to the classification loss, we fine-tune the re-trained model for several epochs. As in <cit.>, the quantized weights is used in the forward and backward process, while the full-precision weights are used in the stage of parameters update. The whole quantization framework is shown in Algorithm <ref>.

The pipeline of <cit.> is close to ours, but we have to emphasize that our approach has three critical differences with them. First, we define and solve a new cluster optimization problem, in which the cluster centers are restricted to several discrete values. Solving the problem is very meaningful to the low-bit quantization, while the optimization algorithms in <cit.> are not applicable to the special problem any more. Second, we seek to solve the cluster indicator Z and the scaling factor α by the alternating algorithm, which avoids extremely time-consuming singular factorization operations. Third, our approach introduces fewer hyper-parameters and is easy to converge.



§ EXPERIMENTS



In this section, we conduct comparison experiments on CIFAR-10 <cit.> and ImageNet <cit.>. The proposed CRQ is compared with two state-of-the-art methods including TWN <cit.> and TTQ <cit.>. Our approach is implemented using the PyTorch <cit.> framework. For the experiments on CIFAR-10 and ImageNet, we adopt the same settings as in <cit.>. Besides, λ (Eq. <ref>) is set to 0.001.

 




 §.§ ImageNet

Table <ref> reports the comparison results of different methods on AlexNet. It can be observed that our ternary AlexNet can reach Top-1 error rate of 42.02% and Top-5 error rate of 19.18%, which achieves the best performance compared with other methods and even outperforms its reference model by 0.75% in Top-1 error rate. Next, further comparative experiments are conducted on ResNet models. Table <ref> summarizes the comparison results of different methods on ResNet-18. It's worth noting that our approach can still get a ternary ResNet-18 that has minor increase of the error rate: 1.99% of Top-1 error rate and 1.38% of Top-5 error rate, which are lower than TWN and TTQ.






 §.§ CIFAR-10







Table <ref> shows the comparison results with TTQ on ResNet-32 and ResNet-44. Similar to the results on ImageNet, our method achieves bigger error rate drop than TTQ for the ternary ResNet-32 and ResNet-44. Particularly, our ternary ResNet-44 even outperforms the full-precision model by 0.26% in the error rate, which may be resulted from the ability of the cluster regularization. Above experiments also imply that our CRQ can deal with the problem of ternary quantization with increased network depth.






 §.§ Convergence Analysis

Going one step further, the rate of convergence of different methods is compared. We compare the evolution process of the training accuracy of different methods. Figure <ref> shows the training accuracy of ternary ResNet-20 changes over epochs on CIFAR-10. We can find that TWN, TTQ and our approach can converge to a minor error, but their accuracy curves are not as smooth as ours. Besides, our method converges within fewer epochs. These results demonstrate our approach has a faster rate of convergence.



§ CONCLUSION



This paper proposes Cluster Regularized Quantization (CRQ), a ternary quantization method for deep networks compression. The key contribution is to define a new cluster regularization term, which significantly reduces the quantization error. Comprehensive experiments on CIFAR-10 and ImageNet demonstrate the CRQ can achieve competitive performance in accuracy and the rate of convergence.



§ ACKNOWLEDGMENT

This work has been supported by the National Key Research and Development Program of China No. 2018YFD0400902 and the National Natural Science
Foundation of China (Grant No. 61673376 and 61573349). 






IEEEbib
10
	
	Han2015
	Song Han, Jeff Pool, John Tran, and William Dally,
	“Learning both weights and connections for efficient neural
	network,”
	in Advances in neural information processing systems, 2015, pp.
	1135–1143.
	
	Hu2016
	Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang,
	“Network trimming: A data-driven neuron pruning approach towards
	efficient deep architectures,”
	arXiv preprint arXiv:1607.03250, 2016.
	
	Denton2014
	Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus,
	“Exploiting linear structure within convolutional networks for
	efficient evaluation,”
	in Advances in neural information processing systems, 2014, pp.
	1269–1277.
	
	Wang2017
	Min Wang, Baoyuan Liu, and Hassan Foroosh,
	“Factorized convolutional neural networks.,”
	in ICCV Workshops, 2017, pp. 545–553.
	
	Howard2017
	Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang,
	Tobias Weyand, Marco Andreetto, and Hartwig Adam,
	“Mobilenets: Efficient convolutional neural networks for mobile
	vision applications,”
	arXiv preprint arXiv:1704.04861, 2017.
	
	Zhang2017
	Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun,
	“Shufflenet: An extremely efficient convolutional neural network for
	mobile devices,”
	arXiv preprint arXiv:1707.01083, 2017.
	
	WangP2018
	Peisong Wang, Qinghao Hu, Yifan Zhang, Chunjie Zhang, Yang Liu, Jian Cheng,
	et al.,
	“Two-step quantization for low-bit neural networks,”
	in Proceedings of the IEEE Conference on Computer Vision and
		Pattern Recognition, 2018, pp. 4376–4384.
	
	Park2017
	Eunhyeok Park, Junwhan Ahn, and Sungjoo Yoo,
	“Weighted-entropy-based quantization for deep neural networks,”
	in IEEE Conference on Computer Vision and Pattern Recognition
		(CVPR), 2017.
	
	courbariaux2016binarized
	Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua
	Bengio,
	“Binarized neural networks: Training deep neural networks with
	weights and activations constrained to+ 1 or-1,”
	arXiv preprint arXiv:1602.02830, 2016.
	
	Courbariaux2015
	Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David,
	“Binaryconnect: Training deep neural networks with binary weights
	during propagations,”
	in Advances in neural information processing systems, 2015, pp.
	3123–3131.
	
	Rastegari2016
	Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi,
	“Xnor-net: Imagenet classification using binary convolutional neural
	networks,”
	in European Conference on Computer Vision. Springer, 2016, pp.
	525–542.
	
	Feng2016Ternary
	Fengfu Li, Bo Zhang, and Bin Liu,
	“Ternary weight networks,”
	arXiv preprint arXiv:1605.04711, 2016.
	
	Soudry2014
	Daniel Soudry, Itay Hubara, and Ron Meir,
	“Expectation backpropagation: Parameter-free training of multilayer
	neural networks with continuous or discrete weights,”
	in Advances in Neural Information Processing Systems, 2014, pp.
	963–971.
	
	Russakovsky2015
	Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
	Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.,
	“Imagenet large scale visual recognition challenge,”
	International Journal of Computer Vision, vol. 115, no. 3, pp.
	211–252, 2015.
	
	Li2017
	Zefan Li, Bingbing Ni, Wenjun Zhang, Xiaokang Yang, and Wen Gao,
	“Performance guaranteed network acceleration via high-order residual
	quantization,”
	arXiv preprint arXiv:1708.08687, 2017.
	
	Zhu2016
	Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally,
	“Trained ternary quantization,”
	arXiv preprint arXiv:1612.01064, 2016.
	
	HanS2015
	Song Han, Huizi Mao, and William J Dally,
	“Deep compression: Compressing deep neural networks with pruning,
	trained quantization and huffman coding,”
	arXiv preprint arXiv:1510.00149, 2015.
	
	Shen2015
	Fumin Shen, Chunhua Shen, Wei Liu, and Heng Tao Shen,
	“Supervised discrete hashing,”
	in Proceedings of the IEEE conference on computer vision and
		pattern recognition, 2015, pp. 37–45.
	
	Wu2018
	Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan
	Lin,
	“Deep k-means: Re-training and parameter sharing with harder
	cluster assignments for compressing deep convolutions,”
	arXiv preprint arXiv:1806.09228, 2018.
	
	Ullrich2017
	Karen Ullrich, Edward Meeds, and Max Welling,
	“Soft weight-sharing for neural network compression,”
	arXiv preprint arXiv:1702.04008, 2017.
	
	krizhevsky2009learning
	Alex Krizhevsky and Geoffrey Hinton,
	“Learning multiple layers of features from tiny images,”
	Tech. Rep., Citeseer, 2009.
	
	paszkepytorch
	Adam Paszke, Soumith Chintala, Ronan Collobert, Koray Kavukcuoglu, Clement
	Farabet, Samy Bengio, Iain Melvin, Jason Weston, and Johnny Mariethoz,
	“Pytorch: Tensors and dynamic neural networks in python with strong
	gpu acceleration, may 2017” .
	





