Fixed-point optimization of deep neural networks with adaptive step size retraining
Sungho Shin Yoonho Boo Wonyong Sung



Tars: Timeliness-aware Adaptive Replica Selection for Key-Value Stores
    

Wanchun Jiang, Liyuan Fang, Haiming Xie, Xiangqian Zhou, Jianxin Wang

School of Information Science and Engineering, Central South University, Changsha, Hunan, China 410083

Email: jiangwc@csu.edu.cn


    
================================================================================================================================================================================================================



Fixed-point optimization of deep neural networks plays an important role in hardware based design and low-power implementations. Many deep neural networks show fairly good performance even with 2- or 3-bit precision when quantized weights are fine-tuned by retraining. We propose an improved fixed-point optimization algorithm that estimates the quantization step size dynamically during the retraining. In addition, a gradual quantization scheme is also tested, which sequentially applies fixed-point optimizations from high- to low-precision. The experiments are conducted for feed-forward deep neural networks (FFDNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs).



deep neural networks, recurrent neural networks, fixed-point quantization, step size adaptation




§ INTRODUCTION


Deep neural networks (DNNs) show very high performance in various fields such as speech recognition <cit.> and image classification <cit.>. However, real-time implementation of DNNs usually demands many arithmetic and weight fetch operations. Thus, word-length optimization is needed in embedded applications to reduce the strength of arithmetic and the size of the weight storage. However, reducing the word length too much tends to degrade the performance. Thus, developing optimum quantization methods is greatly needed for efficient implementation of neural network algorithms. 

Direct quantization of deep neural networks usually does not show satisfactory performance with very low precision weights. However, when the quantized weights are optimized by retraining, the fixed-point performance improves dramatically. Even ternary valued weights (+1, 0, and -1) for a DNN have yielded satisfactory performance <cit.>. Recently, several improved fixed-point optimization methods are developed by employing retraining based fine tuning <cit.>. Also, VLSI and FPGA based deep neural networks have been implemented using fixed-point weights <cit.>.
 
In this work, an improved retraining algorithm is developed for fixed-point optimization of deep neural networks. The previous works decide the optimum quantization step size based on the distribution of floating-point weights and freezes the step-size during the retraining period <cit.>. The proposed algorithm adaptively determines the step-size at the re-quantization step during retraining. Since the weight values change much at the beginning of retraining, this approach is especially effective when applied at initial retraining epochs. In order to change the weight values less abruptly, we also propose and evaluate the gradual quantization method.  In this schemes, floating-point weights are converted to, for example, 6-bit weights, which are then converted to 4-bit weights, and so on. We evaluate the proposed schemes in three different networks: feed-forward deep neural networks (FFDNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs). The proposed methods yielded better results compared to the previous retrain-based quantization schemes. 

The rest of this paper is organized as follows. Section <ref> presents the proposed quantization with step size adaptation during the retraining procedure. The gradual quantization scheme is also explained. Experimental results on FFDNN, CNN, and RNN applications are shown in Section <ref>. Concluding remarks follow in Section <ref>.



§ STEP SIZE ADAPTATION AND GRADUAL QUANTIZATION FOR RETRAINING OF DEEP NEURAL NETWORKS


In this section, we explain the conventional retrain based fixed-point optimization algorithm, and present adaptive step size retraining and gradual quantization methods.  



 §.§ Retrain-based fixed-point quantization review


The original retrain based fixed-point optimization algorithm can be represented briefly as shown in  <ref>. Note that, conventional algorithms <cit.> do not compute Δ_new at the `weights update' stage. In this figure, after obtaining the floating-point weights by training, the quantization step size, Δ, is determined by minimizing the L2 error between the floating-point and fixed-point weights. For the convenience of arithmetic, uniform quantization is assumed. Two algorithms have been developed for the quantization step size optimization. One is an exhaustive search, which decides the initial quantization step size Δ_initial by considering the weight distribution, and then searches the best performing step size between Δ_initial/2 and 2·Δ_initial by testing the quantized network with the evaluation set <cit.>. The second approach is deciding the quantization step size by measuring the mean and the variance of the floating-point weights <cit.>. Then, in the second stage of  <ref>, floating-point weights are rounded to fixed-point values by using the determined quantization step size. The third stage is the inferencing or the forward stage with the quantized network, w^(q). The error signal is calculated and used for backward propagation.  The gradient is calculated and weight update is conducted. Note that the floating-point weights, instead of the fixed-point values, are updated because the amount of weight update is usually much smaller than the quantization step size. Then, the fixed-point weight update, yielding w_ij, new^(q), is accomplished by quantizing the updated floating-point weights. Note that determining Δ_new is not performed in the conventional method, and the same quantization step size is used at every iteration.  



 §.§ Step-size adaptation during retraining






























As described in Section <ref>, the conventional method freezes the step size during the retraining. However, in many cases, the weight values change much by retraining. Note that the amount of change decreases as the retraining iteration progresses. Thus, it is advantageous for improving the performance to adjust the quantization step size during the retraining. Especially, the need of step size adaptation is greater at the beginning of retraining. The proposed scheme adds the determination of Δ_new at the weight update stage of  <ref>. 

We do not perform `exhaustive search' anymore but update the quantization step size during retraining by using the L2 error minimization between the floating-point and fixed-point weights. We consider two different quantization step size update timing. The first one is `epoch-level update', and the other is `1 epoch update & fix'. The `epoch-level update' changes the step size at every epoch. The `1 epoch update & fix' updates the step size only during one or two epochs and freezes it for the remaining epochs. In our empirical evaluation, the first scheme is good for FFDNNs, but the second one shows better results  for CNNs and RNNs. The specific results will be given in Section <ref>.



 §.§ Gradual quantization scheme


We also propose another step size adaptation approach which is similar to the curriculum learning. The curriculum learning is a training strategy to move the goal from an easy level to more complex one gradually <cit.>. One of the important points in curriculum learning is how to organize the tasks from easy to complex ones. We consider that the fixed-point optimization with a small number of bits is a more difficult problem than that with a large one. 

In the proposed scheme, we begin fixed-point optimization with a fairly high precision, such as 6 bits, and then keep lowering the word-length by one bit with retraining for each precision. At each retraining process with a given precision, we also combine the proposed quantization step size adaptation scheme. The experiments are conducted for FFDNNs.



§ EXPERIMENTAL RESULTS


The proposed step size adaptation is evaluated for three applications. We employ FFDNNs for phoneme recognition, CNNs for house number recognition, and RNNs for language modeling. To analyze the effect of step size adaptation, we change the size of each network and their word lengths. 


















 §.§ Phoneme recognition using feed-forward deep neural networks






The FFDNN is trained with the TIMIT corpus <cit.>, and the detailed experimental condition for the data preprocessing is the same with <cit.>. We construct 11 consecutive frames as the network input. The output layer supports 61 labels, and the labels are merged into 39 classes for the final evaluation. For performance evaluation, the number of units in each layer increases from 64 to 1024. We train the floating-point networks using the stochastic gradient descent (SGD) with Nesterov momentum <cit.>. The learning rate decreases from 2e-3 to 3.90625e-6 with a factor of 2 when the development set does not show improvements for 4 consecutive evaluations. For fixed-point networks training, all other conditions are the same with the floating-point case but the initial learning rate is 5e-4.

The results of fixed-point optimization for FFDNNs with and without the step size adaptation are reported in  <ref>. The experiments also show the results with batch normalization (BN) <cit.>. The step size is updated using the `epoch-level update' until the end of the retraining.  <ref> shows that the floating-point network performance saturates at 512 units size when BN is applied, and at 256 units when BN is not used. When the unit size in each layer is 512 or smaller, the proposed algorithm yields better performance in both cases. For example, if the 512 units size network is quantized in 2-bit without BN, the differences between the floating-point and the fixed-point networks are 1.82% and 1% for `conventional' and `adaptive' schemes, respectively. In addition, the phoneme error rate of the 3-bit network optimized with the `adaptive' scheme (29.83%) is lower than that of the 4-bit quantized network with the `conventional' scheme (29.95%). 

BN improves the performance of both floating-point and fixed-point networks. Applying the `adaptive' method improves the performance. For example, if the layer unit size is 128 and 2-bit quantization is used, BN brings the performance gain of 3.42% when `adaptive' scheme is used. Therefore, the proposed `adaptive' method can be efficiently used with BN.

When the unit size is large enough, the quantization scheme does not affect the performance much because a larger size network has a better resiliency to quantization <cit.>. Even the 4-bit quantized 512 units size network without BN shows the performance almost comparable to the floating-point 1024 units size network. When the network is trained with BN, it shows a similar trend.

 <ref> shows the step size Δ of the proposed adaptive scheme as the retraining progresses. Note that the step size is renewed at each epoch during retraining. As shown in this figure, the step size of the last layer varies much, while that of the first layer is almost constant. The step size adaptation is much needed for the last layer. 

We also evaluate the performance of the gradual quantization scheme. The results are reported in  <ref>. The floating point results show 29.61% error rate on the test set. The 6-bit word length shows slightly better accuracy than the floating point. Thus, we define the easiest task as the 6-bit quantization. In  <ref>, the `gradual' scheme yields better performance than the `conventional' strategy, but shows worse or similar results compared to the `adaptive' quantization. The combined strategy of the `adaptive' and `gradual' shows slightly better accuracy than the `adaptive' strategy in 4- and 3-bit quantization, but it is worse than the `adaptive' scheme in 2-bit quantization. Since there is no performance difference between the `adaptive' and `adaptive & gradual' scheme, we only employ the 'adaptive' scheme for CNN and RNN experiments. 
 





























 §.§ Image classification using convolutional neural networks



























Image classification experiments are performed on the SVHN dataset <cit.>. The dataset includes 600,000 labeled 32x32 three channel images from real world house numbers. For the data preprocessing, we employ the same method with <cit.>. The output label has ten units which represent the numbers from 0 to 9. For evaluation of the proposed scheme, we employ three different structures. We name the networks as `L', `C', and `V' which have the trainable parameters 60k, 84k, and 435k, respectively. The `L' network is Lenet5 <cit.>, `C' network is from <cit.>, and `V' network is constructed  as VGG style which is from <cit.>. We train the floating-point networks using SGD with Nesterov momentum. The learning rate is decreased from 2-e2 to 3.125e-4 with a factor of 2 when the development set does not show improvement for 4 consecutive evaluations. For the fixed-point network training, the initial learning rate was 5e-4. The effects of step size adaptation in the CNNs are examined in  <ref>. The step size is updated using the `1 epoch update & fix' strategy. Our algorithm works well for `L' and `V' networks regardless of the weight precision, 2, 3, or 4 bits. However, the `C' networks with the conventional retraining show a better result when the weight precision is 4bits.  Overall, the proposed method yields improved performances.  



 §.§ Language modeling using recurrent neural networks








































































































































Character-level language modeling predicts the next character, and is used for speech recognition and text generation. Since the input and output layers consider only alphabets, the input and output complexity are much lower than the word level language model. We adopt English Wikipedia dataset for training the character level language modeling. The dataset contains 100 MB English Wikipedia text. The input and output layers are composed of 256 units for one-hot encoded ASCII code. The RNN consists of three Long Short-Term Memory (LSTM) layers with a different number of memory cells ranging from 64 to 256 <cit.>. We train the RNNs using AdaDelta based SGD with 64 parallel input streams. The networks are unrolled 256 times and weights update is performed for128 forward steps. The learning rate starts from 5e-4 and decreases until 5e-8. For the step size adaptation, `1 epoch update & fix' strategy is employed. The fixed-point optimization results are reported in  <ref>. As with our previous FFDNN and CNN results, it shows much improved performances on low-precision weights or small size networks.
















§ CONCLUDING REMARKS


We have developed improved fixed-point weight optimization methods for deep neural networks. The first one adaptively determines the quantization step size   by measuring the weight distribution during the retraining procedure. The second one is a curriculum style fixed-point optimization technique, which conducts fixed-point optimization from high- to low-precision gradually. The proposed work yields better quantization results in FFDNN, CNN, and RNN experiments. Especially the effectiveness of the proposed techniques increases when the number of quantization levels is small and the network size is not large enough.















































IEEEbib


