Building Fast and Compact Convolutional Neural Networks for Offline Handwritten Chinese Character Recognition
Xuefeng Xiao Lianwen Jin Yafeng Yang Weixin Yang Jun Sun Tianhai Chang






mymainaddress]Xuefeng Xiao
xiaoxuefengchina@gmail.com

mymainaddress]Lianwen Jinmycorrespondingauthor
[mycorrespondingauthor]Corresponding author
lianwen.jin@gmail.com

mymainaddress]Yafeng Yang
mymainaddress]Weixin Yang
mysecondaryaddress]Jun Sun
mymainaddress]Tianhai Chang

[mymainaddress]School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China
[mysecondaryaddress]Fujitsu Research & Development Center Co. Ltd., Beijing, China


Like other problems in computer vision, offline handwritten Chinese character recognition (HCCR) has achieved impressive results using convolutional neural network (CNN)-based methods. However, larger and deeper networks are needed to deliver state-of-the-art results in this domain. Such networks intuitively appear to incur high computational cost, and require the storage of a large number of parameters, which renders them unfeasible for deployment in portable devices. To solve this problem, we propose a Global Supervised Low-rank Expansion (GSLRE) method and an Adaptive Drop-weight (ADW) technique to solve the problems of speed and storage capacity.  We design a nine-layer CNN for HCCR consisting of 3,755 classes, and devise an algorithm that can reduce the networks computational cost by nine times and compress the network to 1/18 of the original size of the baseline model, with only a 0.21% drop in accuracy. In tests, the proposed algorithm surpassed the best single-network performance reported thus far in the literature while requiring only 2.3 MB for storage. Furthermore, when integrated with our effective forward implementation, the recognition of an offline character image took only 9.7 ms on a CPU. Compared with the state-of-the-art CNN model for HCCR, our approach is approximately 30 times faster, yet 10 times more cost efficient.



Convolutional neural network Handwritten Chinese character recognition CNN Acceleration CNN Compression





 

§ INTRODUCTION

	Offline handwritten Chinese character recognition (HCCR) has been applied to a number of applications, such as for recognizing historical documents, mail sorting, transcription of handwritten notes, and so on. Offline HCCR has drawn the attention of many researchers for over half a century <cit.>. In the last few years, a number of traditional offline approaches have been proposed to improve HCCR performance but have yielded scant progress; the modified quadratic discriminant function (MQDF) <cit.>-based methods are exemplary. There is hence a recognition in the literature that even the best traditional methods are far from mimicking human performance in this domain <cit.>. Due to the availability of better computational hardware and massive amounts of training data in recent years, convolutional neural networks (CNNs), proposed by LeCun in the 1990s <cit.>, have been used to attain state-of-the-art performance in character recognition <cit.>. The multi-column deep neural network (MCDNN) <cit.>, composed of several CNNs, was the first CNN used for HCCR. Zhang et al. <cit.> recently reported an accuracy of 96.95% for recognition by extracting the traditional normalization-cooperated direction-decomposed feature map as input with a CNN. However, the computational cost and storage requirements still prevent the use of CNNs in portable devices, where power consumption and storage capacity are the major challenges.

Many researchers have tried to build fast and compact networks. In this vein, low-rank expansion <cit.> aims to reduce computational cost by decomposing the convolutional layer. According to <cit.>, network pruning is the most effective way to compress the CNN; it eliminates the redundant connections in each layer, following which weight quantization and Huffman encoding are applied to further reduce storage capacity. Although <cit.> achieved impressive performance in accelerating and compressing the network, only a few studies have combined these methods to address the dual challenge of speed and storage capacity. Furthermore, to the best of our knowledge, no study has investigated whether these methods are still feasible for large-scale handwritten Chinese character recognition involving more than 3,700 classes of characters.



In this paper, we propose a method to build a fast and compact CNN-based HCCR classifier. The method is shown in Fig. <ref>; it unifies the advantages of low-rank expansion and network pruning. The first part employs low-rank expansion to decompose the convolutional layer for acceleration purposes, which renders the CNN deeper but compact. The motivation underlying the second part is to remove redundant connections in each layer's to further reduce the storage allocated to parameters and, hence, the computational cost of the entire network. However, in a previous study <cit.> on network pruning, the authors used a fixed threshold to prune the connections of each layer. Instead, we propose an Adaptive Drop-weight (ADW) technique that dynamically increases the threshold and gradually prunes out the parameters of each layer. Here, it comes another problem of the pruning ratio of each layer in previous work <cit.>, which may require numerous attempts for the determination of a suitable threshold for each layer in the context of a trade-off between accuracy drop and compression ratio, especially for some deep networks. Hence, to better address this problem, we propose Connection Redundancy Analysis (CRA) that can analyze redundancy in the connections of each layer and help maximize the pruning ratio of each layer with a tolerable reduction in accuracy.

In experiments involving offline HCCR, the proposed framework reduced by nine times the computational cost of, and by 18 times the parameter storage needed for, the designed CNN; and it degraded accuracy only by 0.21%, which still surpassed the results for the best single-network CNN, reported in the literature thus far, on the ICDAR 2013 Offline HCCR Competition database. The network required only 2.3 MB of storage and took only 9.7 ms to process an offline character image on a single-thread CPU. Moreover, in order to further boost performance, we can increase the width and depth of the networks, or use a new CNN model such as GoogLeNet <cit.> or Deep ResNet <cit.>. This may help finally obtain new benchmarks for offline HCCR, but this is not our main concern in this paper.

The remainder of this paper is organized as follows: Section 2 reviews related work, and Section 3 elaborates on the architecture of the baseline network of the CNN used in our system. Section 4 introduces the adaptive drop-weight technique, whereas Section 5 details the connection redundancy analysis method. Section 6 describes global supervised low-rank expansions in detail, and Section 7 presents the experimental results, which include run time, parameter storage, and accuracy. The conclusions of this study and our future work are summarized in Section 8.



§ RELATED WORK




 §.§ Offline HCCR


Due to the success of CNNs, MQDF-based methods for offline HCCR have already reached their limit. The multi-column deep neural network (MCDNN) <cit.>, consisting of several CNNs, was the first CNN used for offline HCCR. In an offline HCCR competition subsequently organized by ICDAR in 2013 <cit.>, the method developed by the team from Fujitsus R&D Center won with an accuracy of 94.77%. In 2014, they improved accuracy to 96.06% by voting on four alternately trained relaxation convolutional neural networks (ATR-CNN) <cit.>. Zhong et al. <cit.> subsequently proposed combining traditional Gabor features with offline Chinese character images as network inputs, and used a streamlined version of GoogLeNet called HCCR-Gabor-GoogLeNet. They reported an accuracy of 96.35%, and then that of 96.74% for ensembling ten model and become the first one beyond human performance. The framework proposed by Zhou et al. <cit.> is based on HCCR-GoogLeNet <cit.>; they used a Kronecker fully connected (KFC) layer to replace the layers after the four inception groups, and then followed by two fully connected layers, finally obtaining an accuracy of 96.63%. Zhang et al. <cit.> recently combined traditional normalization-cooperated direction-decomposed feature maps and CNNs to obtain accuracy values of 96.95% and 97.12% by voting on three models.



 §.§ Accelerating and Compressing


Most CNN structures, such as VGGNet <cit.>, AlexNet <cit.>, CaffeNet <cit.>, and GoogLeNet <cit.>, have similar properties: for example, the convolutional layers incur most of the computational cost and the fully connected layers contain the most network parameters. Despite the different potential avenues, existing approaches mainly concentrate on accelerating the convolutional layers and compressing the fully connected ones.



To reducing the computational cost of the convolutional layers, Cong and Xiao <cit.> used the Strassen algorithm for fast matrix multiplication to reduce the arithmetic complexity of the convolutional layer without loss in accuracy. Mathieu et al. <cit.> adopted the fast Fourier transform (FFTs) to convert convolutional calculations into pointwise products in the frequency domain for fast computation. Lavin et al. <cit.> proposed using Winograds minimal filtering algorithms to reduce the multiplication in the convolutional layers. Wu et al. <cit.> recently proposed quantized convolutional neural networks that quantize the weights and transform computations into inner products in the convolutional layer. Nevertheless, computations of the convolutional layer are transformed into matrix multiplication by using the im2col algorithm <cit.> and the BLAS (Basic Linear Algebra Subprograms) library. These tools are useful for faster CPU implementation of CNNs and cannot be used with the previously proposed method in <cit.>. In this paper, we use the low-rank expansion-based method that can combine the matrix multiplication method by using the BLAS library. Jaderberg et al. <cit.> exploited the cross-channel or filter redundancy to formulate a low-rank basis for filters, and proposed filter and data reconstruction techniques for optimization. Zhang et al. <cit.> improved their work by considering a non-linear case and asymmetric reconstruction for multiple layers to mitigate reconstruction error.

For fully connected layers, HashedNets proposed by Chen et al. <cit.> uses the hash function to group weights into hash buckets, where connections in the same hash buckets share parameter value. Vanhoucke et al. <cit.> used an eight-bit fixed-point integer to replace the 32-bit floating point. Matthieu et al. <cit.> proposed binarized neural networks that constrain weights and activations to +1 or -1, and replace most floating-point multiplications by one-bit exclusive-NOR operations. It is clear that this can reduce computational cost and parameter storage but, on the other hand, degrades network performance. Lin et al. <cit.> used SVD-based low-rank expansions to compress the fully connected layers, and then used global error reconstruction to fine-tune the entire network. However, both these methods have low compression ratios, or seriously deteriorate network performance. Methods based on network pruning <cit.> can significantly reduce parameter storage by learning important connections without compromising network performance. Deep compression was proposed by Han et al. <cit.> to further reduce storage by combining network pruning, weight quantization, and Huffman coding. Guo et al. <cit.> proposed dynamic network surgery that can dynamically prune and splice connections based on Han's work <cit.>.



§ ARCHITECTURE OF CONVOLUTIONAL NEURAL NETWORK


As shown in Fig. <ref>, we designed a nine-layer (only accounting for the convolutional layer and the fully connected layer) network for offline HCCR consisting of seven convolutional layers and two fully connected layers. Each of the first three convolutional layers are followed by a max-pooling layer. Following this, every two convolutional layers are followed by a max-pooling layer. The last max-pooling layer is followed by a fully connected layer, which contains 1,024 neurons. The last fully connected layer contains 3,755 neurons, and is used to perform the final classification. The overall architecture can be represented as Input-96C3-MP3-128C3-MP3-160C3-MP3-256C3-256C3-MP3-384C3-384C3-MP3-1024FC-Output.

We found that within a certain range, increasing the size of the input character image improved classification performance, but incurred higher computational cost. Hence, we fixed this effect of increasing size and computational cost by resizing the input characters into 96×96. In our baseline networks, all convolutional filters were 3×3, and a pixel was added to retain the size. Finally, the max-pooling operation was carried out over a 3×3 window with a stride of 2.

In our proposed network, the parametric rectified linear unit (PReLU) <cit.>, slightly different from the rectified linear unit (ReLU) <cit.>, was used to enable the network to easily converge and minimize the risk of overfitting to boost performance. Ioffe et al. <cit.> proposed batch normalization (BN), which can normalize nonlinear inputs and stabilize the distribution by reducing the internal covariate shift. It not only provides the liberty of using higher learning rates to expedite network convergence, but also ameliorates network performance at a negligible computational cost and storage. Moreover, for some deep networks, BN can effectively solve the problem of vanishing gradients. Therefore, all convolutional layers and the first fully connected layer were equipped with a BN layer, and the PReLU were added to each BN layer. Since the fully connected layers are quite redundant, we added the dropout <cit.> layer between the two fully connected layers for regularization, where the ratio was set to 0.5.

The main difference between our proposed model and other available models for offline HCCR is that the former involves BN and PReLU in the network; hence, we refer to this baseline network as the HCCR-CNN9Layer. Although the CNN model used is quite simple, it is yielded state-of-the-art performance for HCCR.



§ ADAPTIVE DROP-WEIGHT




Our pruning scheme is shown in Fig. <ref>; it consists of two parts. The first a new technique called Adaptive Drop-weight (ADW), which can gradually prune out the weighted connections of each layer by dynamically increasing the pruning threshold. When the pruning ratio reaches a value determined by the results of the CRA for the layer, we remember the threshold for further pruning.




 §.§ Pruning Threshold


In the previous work on network pruning <cit.>, a fixed threshold was determined as follows:

    P_th = α/N∑_i = 1^N | w_i|  + β√(1/N∑_i = 1^N (w_i - 1/N∑_i = 1^N w_i)^2)  + λ.


The pruning threshold P_th depends on the weight of the layer w_i in terms of average absolute value and variance. In order to render P_th suitable for each layer, parameters α,β,λ are selected by their empirical rules <cit.>. However, if the fixed threshold is too high, it leads to the pruning of a large number of connections at the outset, which results in drastic drop in performance. On the contrary, if the fixed threshold is too low, the compression ratio may be far from the desired value. In order to solve this problem, we propose using a dynamically increased threshold that gradually prunes out the weighted connections of each layer. This methodology can gradually lead the network toward self-adaptive pruning connections.



 §.§ Pruning Training Scheme


In order to gradually prune redundant connections from each layer, we prune connections after every I iterations (in experiments, we set I = 10). If we intend to prune the ratio r_i in each layer that contains N_i weights within T_1 iterations, the pruned number p_i is increased by r_iN_iI/T_1 in each pruning iteration. The threshold is also gradually increased. During the iterations without the pruning process, the weights are updated with a gradient, and the pruned weights never come back. Once the desired pruning ratio is reached, the increasing threshold is fixed and noted for further pruning of the layer until pruning ends after T_2 iterations. This pruning process has been described in detail in Algorithm <ref>.

In order to further compress the network and improve performance, we employ the strategy proposed in <cit.> to quantize weights. A k-means clustering algorithm is used to cluster weights for each layer of a pruned network. The quantized pruned network is then fine-tuned, which may result in better network performance.






§ CONNECTION REDUNDANCY ANALYSIS


The deep neural network consists of many layers, and each plays a significant role in the network. There are inevitably various redundancies in each layer, especially in the large gap between the convolutional layer and the fully connected layer. It makes sense that pruning ratio be determined by the redundant connections of each layers, and that the same pruning ratio not be applied to all layers. Nevertheless, retrospective work by <cit.> involved a fixed threshold P_th based on the relevant layer's weights to prune the connection. Hence, numerous experiments are needed to find the pertinent values of α,β, and λ for the pruning threshold P_th for each layer; as is self-evident, this is very time consuming, especially for deep networks.

In order to better address the above issue, we propose a Connection Redundancy Analysis (CRA) method that analyzes each layer's redundancy and can help us set a suitable value for the pruning ratio r_i for it. Inspired by <cit.>, a sensitivity analysis was carried out to analyze the importance of a layers parameters for network performance. Iandola et al. <cit.> implemented the strategy of directly pruning half the parameters with the smallest absolute values and carried out the experiment separately for each layer. After testing the pruned networks, network performance was examined. However, this strategy can only highlight the important parameters of a given layer with regard to performance, and cannot help determine how many connections are redundant.

To carry out the Connection Redundancy Analysis, we separately conducted an experiment for each layer. While carrying out the experiment on a layer, we fixed the weights of other layers and pruned only that layer. By using our proposed Adaptive Drop-weight as pruning strategy, we gradually pruned each layer's redundant connections, which were thought to gradually degrade network performance. When the drop in accuracy was beyond a given tolerance level, we knew how many connections had been pruned, which guided us in further pruning the network.

Since the proposed CRA was implemented to prune out the layers separately, it is difficult to analyze the scenario where all layers are pruned together. However, it may guide us in setting a proper pruning ratio for each layer. The ultimate goal of the CRA is to maximize the compression ratio under a tolerable reduction of accuracy rate, which desires further research.



§ GROBAL SUPERVISED LOW RANK EXPANSION




 §.§ Decomposition Scheme




For the original convolutional layer illustrated in Fig. <ref>, the input feature map is a three-dimensional (3D) vector X ∈ℝ ^C × H × W, where C is the channel of the input feature map, and H and W are its height and width, respectively. The output feature map is also a 3D vector Y ∈ℝ ^N × H' × W', where N is the channel of the output feature map, and H' and W' are its height and width, respectively. The kernel matrix is a 4D vector W ∈ℝ ^C × N × K × K, where the size of the kernel is K × K. The output feature map can be calculated by

    Y   (n,h',w') = 
       ∑_c = 1^C ∑_i = 1^K ∑_j = 1^K W(n,c,i,j)X(c,h' + i - 1,w' + j - 1).

We know that the computational cost of the direct convolutional layer is O(CNK^2H'W').

By carrying out the low-rank expansion shown in Fig. <ref>, that the input feature map originally convolved with the square filter, will be transformed into the input feature map convolved with two low rank filters. The first one is the input convolved with the vertical kernel T ∈ℝ ^C × D × K × 1, where D is the output feature number the decomposed layer. The first output is

    M   (d,h',w) = 
       ∑_c = 1^C ∑_i = 1^K T(d,c,i,1)X(c,h' + i - 1,w),
 where the computational cost by the first convolution is O(CDKH'W).
Then the output M ∈ℝ ^D × H' × W convolves with horizontal kernel V ∈ℝ ^D × N × 1 × K, and the final output is calculated by

    Y   (n,h',w') =  
       ∑_d = 1^D ∑_j = 1^K V(n,d,1,j)M(d,h',w' + j - 1).

The computational cost by the second convolution is given by O(NDKH'W'). If the two low-rank expansions are considered together, the computational cost is  O(DKH'(NW' + CW)).
So if we want to accelerate x time for the convolutional layer, D can be determined as

    D = CNKW'/(CW + NW')x.




 §.§ Training scheme


In past work <cit.>, the output of each layer was used as a supervisor to learn the low-rank filter for that layer. This method was mainly devised to minimize the reconstruction error between the local output and the low-rank approximation output, as shown in Fig. <ref>. We refer to this strategy as Local Supervised Low-rank Expansion (LSLRE). We think that while using the output of the local layer to guide the low-rank expansion is a reasonable and straightforward strategy, it does not a direct relationship with the performance of global classification.



Thus, we propose Global Supervised Low-rank Expansion (GSLRE), which uses the label as supervisor. The training scheme is shown in Fig. <ref>. The training process is conducted in a layer-by-layer manner. For a specific layer, the original convolutional layer, say, the second layer Conv2 is decomposed into two smaller layers, Conv2_de1 and Conv2_de2  (see Fig. <ref>). The parameters of Conv2_de1 and Conv2_de2 are determined through back-propagation using the SGD algorithm, based on the loss function for the entire network. It is worth mentioning that during the training of the specific convolutional layer, the parameters of other convolutional layers are kept fixed. This is because our network is equipped with the BN layer, which can enable gradients to smoothly pass into lower-level layers even though the network deepens.

Since the first convolutional layer is hard to approximate (as mentioned in <cit.>), and plays an important role in extracting features from the original image, we begin our low-rank training scheme at the second convolutional layer. Once the second layer has been decomposed and trained adequately, we begin decomposing the third layer. When training the third convolutional layer, the parameters of both the second and the third layers are learned and updated according to the SGD-BP algorithm. In this way, all convolutional layers are decomposed and trained. Finally, since the first convolutional layer and the parameters of the fully connected layer are always fixed during the above low-rank expansion training process, we need to fine-tune the whole network to further improve overall performance.



§ EXPERIMENT


We evaluated our method on an offline HCCR task. The training data was taken from the CASIA-HWDB1.0 and CASIA-HWDB1.1 <cit.> databases, written by 300 and 420 people, respectively. The datasets contained 2,678,424 samples in total. The test data were datasets used in the ICDAR-2013 offline competition <cit.>, which contained 224,419 samples written by 60 people. The training and testing databases were written by different people, and contained 3,755 classes.



 §.§ Baseline Model


We trained our baseline network (Fig. <ref>) on the Caffe <cit.> deep learning platform. We used the method of mini-batch gradient descent with momentum for training. The mini-batch size was set to 128 and momentum to 0.9. In order to make training data independent of each mini-batch, we shuffled the data before training. Since our proposed network is equipped with a BN layer, it allowed us to use higher learning rates to accelerate converge. We hence initialize the learning rate at 0.1, and then reduced it × 0.1 every 70,000 iterations. Training was completed after 300,000 iterations, and we obtained an accuracy of 97.30%.

Fig. <ref> shows our proposed network. We realized that the first convolutional layer plays an important role in extracting features from the original image <cit.>, and incurs lower computational cost and smaller parameter storage (less than 1% of the entire network). Thus, it is intuitive that the parameters of this layer should not be modified.



 §.§ The evaluation of the Global Supervised Low Rank Expansions






By using our proposed architecture, the baseline network was accelerated fourfold. Using Eq. <ref>, we calculated the number of feature maps each convolutional layer after decomposing it. In Table <ref>, it is clear that we were able to accelerate the network fourfold with a negligible drop in accuracy by integrating our devised Global Supervised Low-rank Expansion training scheme. It was also shown that our decomposing training scheme can obtain better results than the direct training of a decomposed network architecture.



 §.§ The evaluation of the Adaptive Drop-weight


We first applied our method to the MNIST database with the LeNet-300-100 and the LeNet-5 networks <cit.>. The MNIST dataset was designed for character recognition of handwritten digits. LeNet-5 is a convolutional network that contains two convolutional layers and two fully connected layers. LeNet-300-100 is a fully connected network with two hidden layers. The baseline models were trained on the Caffe <cit.> deep learning platform without any data augmentation. We directly trained the LeNet-5 using the settings for the training parameters provided by Caffe. In this way, an accuracy of 99.11% was obtained after training for 10,000 iterations. The training parameter settings of LeNet-300-100 were nearly identical to those for LeNet-5, and yielded an accuracy of 98.33%.



As shown in Table <ref>, with our proposed pruning strategy, we compressed LeNet-5 by a factor of 133 and LeNet-300-100 by that of 60 using the proposed ADW method. It surpassed the results in <cit.>, which was the first application of network pruning for compression. Compared with recent work <cit.>, we achieved a higher pruning ratio and better accuracy, especially for LeNet-300-100. We also combined the methods of quantizing weights for further compression. Finally, we obtained a state-of-the-art compression ratio of 250 times for LeNet-5 and 113 times for LeNet-300-100 without any loss in accuracy.

Following this, we applied our proposed compression framework on an offline HCCR network that was accelerated fourfold, and contained 13 convolutional layers and two fully connected layers. In Table <ref>, it is evident that the entire network was compressed to approximately a quarter of its size using only the proposed ADW method. When we integrated the weight quantization, storage was further reduced approximately 14-fold with a drop of only 0.18% in accuracy.

While pruning LeNet-5, we noticed that separately pruning the convolutional and the fully connected layers was a better choice to deal with the vanishing gradient problem than pruning these layers together, which was the strategy used in past work <cit.>. However, since our accelerated network was equipped with the BN layer, it enabled the gradient to smoothly pass in both forward and backward propagations, as also demonstrated in <cit.>. In our experiment, we were able to prune the convolutional layers and fully connected layers together. This not only reduced training time for the pruning process, but also yielded higher accuracy and compression ratio at the same time.



 §.§ The evaluation of the Connection Redundancy Analysis


We implemented our CRA on each layer in LeNet-5 and LeNet-300-100. With our proposed Adaptive Drop-weight pruning strategy, we gradually pruned the connections of each layer.

In Fig. <ref> and <ref>, we see that at the start of the experiment, the network was less susceptible to the pruning ratio; but later on, drastically decreases down with the higher values of this pruning ratio. However, since each layer had different redundant connections, the pruning ratio of each was different. In our experiment, CRA was implemented with a tolerable accuracy drop of 0.1% for each layer. Then, the pruning ratio was used to guide the pruning of the network.



In the same way, CRA was applied to an offline HCCR network, which reduced computational cost fourfold in the convolutional layer. We then analyzed each layer's redundancy with a tolerable accuracy drop of 0.1% to guide us in pruning. Since the convolutional layer had been accelerated four times, the redundancy therein was significantly eliminated. As shown in Fig. <ref>, we set a much lower pruning ratio than the CRA results in convolutional layer to maintain accuracy at par.





 §.§ The results of accuracy


Table <ref> illustrates the results of different methods that achieved performance beyond human level on the ICDAR-2013 offline competition database as well as their network storage and FLOPs (multiply-adds) in detail.

We use our nine-layer network, shown in Fig. <ref>, and achieved an accuracy of 97.30%. We realized that the BN and the PReLU layers are quite effective for offline HCCR. Using our proposed GSLRE and ADW, we further reduced computational cost by nine times and parameter storage by 18 times with a only drop of 0.21% in accuracy. This result still surpassed the best single-network performance in line 7 in Table <ref>, but our model simultaneously involved considerably less parameter storage and incurred lower computational cost.



It was clear that the larger and deeper the network, the better it performed. Hence, based on our proposed network, we added more convolutional layers: Input-96C3-MP3-128C3-128C3-MP3-192C3-192C3-MP3-256C3-256C3-MP3-384C3-384C3-384C3-MP3-1024FC-Output. We refer to this large network as HCCR-CNN12Layer, which yielded an accuracy of 97.59%, as shown on line 12 in Table <ref>. Then, combining our GSLRE and ADW, we were still able to reduce computational cost 16-fold and parameter storage 10-fold with only a drop of 0.19% in accuracy.



 §.§ The results of the forward implementation


The run time of the network is crucial for applying offline HCCR to deal with real-time tasks. Other techniques can be deployed for accelerating CNNs for real-time applications. Loop unrolling (LU) is a well-known and efficient strategy to improve speed, especially for large loops. Using the im2col algorithm, convolutional computations were converted into matrix-matrix multiplication using the BLAS library[In the following experiments, we used Intel MKL as the BLAS library, available at https://software.intel.com/en-us/intel-mkl.], which has been shown to be an efficient way for CPU-based implementation of CNNs. By using the BLAS library, the fully connected layers were directly implemented into matrix-vector multiplication. Moreover, when we eliminated connections in each layer using our proposed ADW method, we used sparse matrix-matrix multiplication and sparse matrix-vector multiplication, respectively, for the convolutional layer and the fully connected layer. However, we found that if the layer was not sparse enough, performance degraded. In our proposed network, we simply applied sparse matrix-vector multiplication only to compute the fully connected layer.



We compared the forward run time with different strategies on a single-threaded CPU. The experiments were carried out on a single desktop PC equipped with 3.60 GHz Intel Core i7-6700 and 16 GB of memory. From Table <ref>, we see that when we did not use a technique to accelerate the CNN, the run time was long (1369 ms per character). When we simply used loop unrolling for all layers, the run time was reduced. When we used our acceleration method and reduced computational cost fourfold, the run time was also reduced approximately fourfold (from 492 ms to 118 ms). Then, all convolutional layers and the fully connected layer were computed by adopting matrix-matrix and matrix-vector multiplication, respectively, with the BLAS library. Loop unrolling was also applied to all other layers. The run time decreased significantly. Finally, using our compression method to prune redundant connections in the convolutional layers and the fully connected layers, we employed sparse matrix-vector multiplication to implement the computations in the fully connected layer. In this way, we achieved a fast and compact CNN model for large-scale HCCR with a speed of 9.7 ms/char but only 2.3 of MB storage. We compared the forward run time implemented by Zhang et al. <cit.>(the last row) with that of our model. The proposed forward implementation method was clearly more effective than Zhang's <cit.> method: it was approximately 30 times faster but 10 times smaller. The source code of our fast and compact CNN model's forword implementation will soon be made publicly available.



§ CONCLUSION


In this paper, we proposed an effective approach for accelerating and compressing a CNN for large-scale HCCR involving 3,755 classes of Chinese characters. We proposed a Global Supervised Low-rank Expansion to accelerate calculations in the convolutional layers, and an Adaptive Drop-weight method to remove redundant connections by using a dynamic increase in the pruning threshold of each layer. We also proposed Connection Redundancy Analysis technology to analyze redundant connections in each layer in order to guide the pruning of the CNN without compromising the performance of the network.

In future work, we plan to apply the proposed framework to other fields, such as image classification and object detection. These ideas can also be used to address deep recurrent neural networks <cit.>, especially for long short-term memory, as they are viable deep-learning models to deal with such time sequence-based problems as online handwritten character/text recognition <cit.>.



model2-names
10
url<#>1urlprefixURL href#1#2#2 #1#1

kimura1987modified
F. Kimura, K. Takashina, S. Tsuruoka, Y. Miyake, Modified quadratic
  discriminant functions and the application to chinese character recognition,
  IEEE Trans. Pattern Anal. Mach. Intell. 9 (1) (1987) 149–153.

jin2000deformation
L. Jin, J. Huang, J. Yin, Q. He, Deformation transformation for handwritten
  chinese character shape correction, in: Proceedings of Advances in Multimodal
  Interfaces (ICMI), 2000, pp. 450–457.

dai2007chinese
R. Dai, C. Liu, B. Xiao, Chinese character recognition: history, status and
  prospects, Frontiers of Computer Science in China 1 (2) (2007) 126–136.

long2008building
T. Long, L. Jin, Building compact MQDF classifier for large character set
  recognition by subspace distribution sharing, Pattern Recognition 41 (9)
  (2008) 2916–2925.

liu2013online
C. Liu, F. Yin, D. Wang, Q. Wang, Online and offline handwritten chinese
  character recognition: Benchmarking on new databases, Pattern Recognition
  46 (1) (2013) 155–162.

zhang2017online
X. Zhang, Y. Bengio, C. Liu, Online and offline handwritten chinese character
  recognition: A comprehensive study and new benchmark, Pattern Recognition
  61 (2017) 348–360.

yin2013icdar
F. Yin, Q. Wang, X. Zhang, C. Liu, ICDAR 2013 chinese handwriting recognition
  competition, in: Proceedings of International Conference on Document Analysis
  and Recognition (ICDAR), 2013, pp. 1464–1470.

le1990handwritten
Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E. Hubbard,
  L. D. Jackel, Handwritten digit recognition with a back-propagation network,
  in: Proceedings of Advances in Neural Information Processing Systems (NIPS),
  Morgan-Kaufmann, 1990, pp. 396–404.

lecun1998gradient
Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to
  document recognition, Proceedings of the IEEE 86 (11) (1998) 2278–2324.

cirecsan2013multi
D. C. Ciresan, U. Meier, Multi-column deep neural networks for offline
  handwritten chinese character classification, in: Proceedings of
  International Joint Conference on Neural Networks (IJCNN), 2015, pp. 1–6.

denton2014exploiting
E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, R. Fergus, Exploiting linear
  structure within convolutional networks for efficient evaluation, in:
  Proceedings of Advances in Neural Information Processing Systems (NIPS),
  2014, pp. 1269–1277.

jaderberg2014speeding
M. Jaderberg, A. Vedaldi, A. Zisserman, Speeding up convolutional neural
  networks with low rank expansions, in: Proceedings of British Machine Vision
  Conference (BMVC), 2014.

zhang2015efficient
X. Zhang, J. Zou, X. Ming, K. He, J. Sun, Efficient and accurate approximations
  of nonlinear convolutional networks, in: Proceedings of Computer Vision and
  Pattern Recognition (CVPR), 2015, pp. 1984–1992.

zhang2015accelerating
X. Zhang, J. Zou, K. He, J. Sun, Accelerating very deep convolutional networks
  for classification and detection, IEEE Trans. Pattern Anal. Mach. Intell.
  38 (10) (2016) 1943–1955.

han2015learning
S. Han, J. Pool, J. Tran, W. J. Dally, Learning both weights and connections
  for efficient neural network, in: Proceedings of Advances in Neural
  Information Processing Systems (NIPS), 2015, pp. 1135–1143.

guo2016dynamic
Y. Guo, A. Yao, Y. Chen, Dynamic network surgery for efficient dnns, in:
  Proceedings of Advances in Neural Information Processing Systems (NIPS),
  2016, pp. 1379–1387.

szegedy2015going
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan,
  V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: Proceedings
  of Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–9.

He_2016_CVPR
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition,
  in: Proceedings of Computer Vision and Pattern Recognition (CVPR), 2016, pp.
  770–778.

wu2014handwritten
C. Wu, W. Fan, Y. He, J. Sun, S. Naoi, Handwritten character recognition by
  alternately trained relaxation convolutional neural network, in: Proceedings
  of International Conference on Frontiers in Handwriting Recognition (ICFHR),
  2014, pp. 291–296.

zhong2015high
Z. Zhong, L. Jin, Z. Xie, High performance offline handwritten chinese
  character recognition using googlenet and directional feature maps, in:
  Proceedings of International Conference on Document Analysis and Recognition
  (ICDAR), 2015, pp. 846–850.

zhou2015exploiting
S. Zhou, J. Wu, Y. Wu, X. Zhou, Exploiting local structures with the kronecker
  layer in convolutional networks, CoRR abs/1512.09194.

Simonyan2014VeryDC
K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale
  image recognition, in: Proceedings of International Conference on Learning
  Representations (ICLR), 2014.

krizhevsky2012imagenet
A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep
  convolutional neural networks, in: Proceedings of Advances in Neural
  Information Processing Systems (NIPS), 2012, pp. 1106–1114.

jia2014caffe
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. B. Girshick,
  S. Guadarrama, T. Darrell, Caffe: Convolutional architecture for fast feature
  embedding, in: Proceedings of International Conference on Multimedia (ICM),
  2014, pp. 675–678.

cong2014minimizing
J. Cong, B. Xiao, Minimizing computation in convolutional neural networks, in:
  Proceedings of International Conference on Artificial Neural Networks
  (ICANN), 2014, pp. 281–290.

mathieu2013fast
M. Mathieu, M. Henaff, Y. LeCun, Fast training of convolutional networks
  through ffts, CoRR abs/1312.5851.

lavin2015fast
A. Lavin, S. Gray, Fast algorithms for convolutional neural networks, in:
  Proceedings of Computer Vision and Pattern Recognition (CVPR), 2016, pp.
  4013–4021.

wu2016quantized
J. Wu, C. Leng, Y. Wang, Q. Hu, J. Cheng, Quantized convolutional neural
  networks for mobile devices, in: Proceedings of Computer Vision and Pattern
  Recognition (CVPR), 2016, pp. 4820–4828.

Yanai2O16
K. Yanai, R. Tanno, K. Okamoto, Efficient mobile implementation of A
  cnn-based object recognition system, in: Proceedings of International
  Conference on Multimedia(ACM MM), 2016, pp. 362–366.

chen2015compressing
W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, Y. Chen, Compressing neural
  networks with the hashing trick, in: Proceedings of International Conference
  on Machine Learning (ICML), 2015, pp. 2285–2294.

vanhoucke2011improving
V. Vanhoucke, A. Senior, M. Z. Mao, Improving the speed of neural networks on
  cpus, in: NIPS Deep Learning and Unsupervised Feature Learning Workshop,
  Citeseer, 2011.

courbariaux2016binarynet
M. Courbariaux, Y. Bengio, Binarynet: Training deep neural networks with
  weights and activations constrained to +1 or -1, CoRR abs/1602.02830.

Lin2016TowardsCN
S. Lin, R. Ji, X. Guo, X. Li, Towards convolutional neural networks compression
  via global error reconstruction, in: Proceedings of International Joint
  Conference on Artificial Intelligence (IJCAI), 2016, pp. 1753–1759.

Han2015DeepCC
S. Han, H. Mao, W. J. Dally, Deep compression: Compressing deep neural network
  with pruning, trained quantization and huffman coding, in: Proceedings of
  International Conference on Learning Representations (ICLR), 2016.

He2015DelvingDI
K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing
  human-level performance on imagenet classification, in: Proceedings of
  International Conference on Computer Vision (ICCV), 2015, pp. 1026–1034.

Nair2010RectifiedLU
V. Nair, G. E. Hinton, Rectified linear units improve restricted boltzmann
  machines, in: Proceedings of International Conference on Machine Learning
  (ICML), 2010, pp. 807–814.

ioffe2015batch
S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training
  by reducing internal covariate shift, in: Proceedings of International
  Conference on Machine Learning (ICML), 2015, pp. 448–456.

srivastava2014dropout
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov,
  Dropout: a simple way to prevent neural networks from overfitting, Journal of
  Machine Learning Research 15 (1) (2014) 1929–1958.

iandola2016squeezenet
F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, K. Keutzer,
  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < 0.5mb
  model size, CoRR abs/1602.07360.

liu2011casia
C. Liu, F. Yin, D. Wang, Q. Wang, CASIA online and offline chinese
  handwriting databases, in: Proceedings of International Conference on
  Document Analysis and Recognition (ICDAR), 2011, pp. 37–41.

chen2015beyond
L. Chen, S. Wang, W. Fan, J. Sun, S. Naoi, Beyond human recognition: A
  cnn-based framework for handwritten character recognition, in: Proceedings of
  Asian Conference on Pattern Recognition (ACPR), 2015, pp. 695–699.

Graves2012Supervised
A. Graves, Supervised Sequence Labelling with Recurrent Neural Networks, Vol.
  385 of Studies in Computational Intelligence, Springer, 2012.

zhang2016drawing
X. Zhang, F. Yin, Y. Zhang, C. Liu, Y. Bengio, Drawing and recognizing chinese
  characters with recurrent neural network, CoRR abs/1606.06539.

xie2016learning
Z. Xie, Z. Sun, L. Jin, H. Ni, T. Lyons, Learning spatial-semantic context with
  fully convolutional recurrent network for online handwritten chinese text
  recognition, CoRR abs/1610.02616.




