{'fraction_non_alphanumeric': 0.04034279717833615, 'fraction_numerical': 0.015973882119745816, 'mean_word_length': 4.624262295081968, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Attention-Based Handwritten Mathematical Expression Recognition with Scale Augmentation and Drop Attention

Handwritten mathematical expression recognition (HMER) is an important research direction in handwriting recognition. The performance of HMER suffers from the two-dimensional structure of mathematical expressions (MEs). To address this issue, in this paper, we propose a high-performance HMER model with scale augmentation and drop attention. Specifically, tackling ME with unstable scale in both horizontal and vertical directions, scale augmentation improves the performance of the model on MEs of various scales. An attention-based encoder-decoder network is used for extracting features and generating predictions. In addition, drop attention is proposed to further improve performance when the attention distribution of the decoder is not precise. Compared with previous methods, our method achieves state-of-the-art performance on two public datasets of CROHME 2014 and CROHME 2016.


I. INTRODUCTION
Handwritten mathematical expression recognition (HMER) has been researched for more than 50 years [1], and has wide applications in practice, such as human-computer interaction, office automation, and intelligent education. As a significant branch of handwriting recognition, HMER faces many challenges, which attract researchers' interest. Challenges of HMER include indistinguishably similar symbols, various handwriting styles, and lack of data, which are typical for handwriting recognition. However, a mathematical expression (ME) differs from isolated characters or texts in its unique twodimensional structure. Therefore, HMER models should also phase the structure, which entails recognizing the spatial relations between two mathematical symbols or sub-expressions.
Traditional grammar-based methods, such as [2], [3], and [4], recognized MEs through symbol segmentation, symbol recognition, and structural analysis sequentially. In the past few years, deep neural networks (DNNs) have made breakthrough progress in handwriting recognition [5] [6]. Many DNN models for HMER were proposed by researchers, such as [7], [8], [9], and [10]. These studies, based on the encoderdecoder framework [11], considered HMER as an image to sequence problem, as shown in Fig. 1. Handwriting text recognition is also a typical image to sequence task, whereas in most previous works, the text are normalized to a fixed height, which is a critical process before it is fed to DNN models [5]. However, as ME has a complex two-dimensional structure with symbols of various sizes, it is inappropriate to normalize all ME images to the same size. As shown in Fig. 2(a), some MEs have a multi-line structure, and the size of superscript and subscript symbol are smaller than that of other symbols. Normalizing these ME images to the same height will degrade the DNN performance because some symbols will be too small or too large to be recognized, as shown in Fig. 2(b). To address this issue, we propose a scale augmentation method to generate ME images for training the DNN model. Before being fed to DNN, MEs are augmented to another scale randomly but keeping the original aspect ratio. Subsequently, MEs are zero-padded to a fixed size, as shown in Fig. 2(c). The DNN model is then trained to generate the correct predictions from MEs of various scales.
We use an attention-based encoder-decoder framework to recognize MEs, including symbol prediction and structure phasing. The encoder extracts features from the input image and the decoder predicts one symbol at each time step to output a sequence. The decoder generates an attention weight map, which represents the importance of each feature for predicting the symbol at the current time step. When attention neglects key features, the model generates an incorrect prediction and performs worse [12]. Inspired by [13], we propose a drop attention module applied to the decoder in the training phase to alleviate this issue. The drop attention module can assist our model to predict the correct symbol or spatial relationship when attention is imprecise through suppressing or abandoning features. These two complementary methods, named "scale augmentation" and "drop attention", are both applied in the training phase and improve performance.
The rest of this paper is organized as follows. In Section II, we briefly reviews the related works of HMER. The proposed methods are introduced in details in Section III. In Section IV, we present and analyze the experimental results. Finally, we conclude our paper in Section V.

II. RELATED WORKS
Many traditional methods for HMER were based on grammars, such as graph grammars [14], definite clause grammars [15], and relational grammars [16]. Yamamoto et al. [3] used stochastic context-free grammars and employed a CockeY-oungerKasami algorithm to parse MEs. Simistira et al. [2] also proposed a method based on stochastic context-free grammars and used a probabilistic support vector machine classifier to recognize spatial relations between two mathematical symbols. Hidden Markov models have been used to recognize mathematical symbols byÁlvaro et al. [17] and an integrated grammar-based method has been proposed by them [4].
With the rapid development of deep learning, many methods based on the encoder-decoder framework [11] have achieved excellent performance in HMER in the past few years. Deng et. al. [18] pioneered using the encoder-decoder model in ME recognition. Zhang et. al. [8] further improved the performance by using a deep convolutional neural network (CNN) as an encoder and adopting coverage-based attention. Moreover, Zhang et. al. [19] added an extra DenseNet [20] branch to deal with different sizes of symbols. Zhang et. al [21] also used a recurrent neural network (RNN) as an encoder for online HMER. Wang et. al. [22] proposed a multi-modal network with both online and offline encoder branches. Hong et. al. [23] used Markovian transition probability matrix for decoding. Le et. al. [9] proposed pattern generation strategies to augment training data and improve the recognition performance. To learn semantic-invariant features of ME, Wu et. al. [10] proposed a paired adversarial learning method.

III. PROPOSED METHODS
In this paper, we treat HMER as an image-to-sequence problem. Specifically, given an image I containing an ME, our recognition model outputs the LaTeX sequence Y of the ME.

A. Encoder-decoder network
Our recognition model consists of a CNN encoder and an attention-based decoder, as shown in Fig. 3.
We build the encoder by modifying ResNet-18 [24] because CNNs have excelled in visual feature extraction. To extract more precise features and avoid neglecting features of smallscale symbols (e.g., dot, superscript, or subscript), we set  [24]. Moreover, max pooling layers are adopted for down-sampling. Dropout layers are also applied to alleviate network overfitting. The detailed network configuration is shown in Tab. I, where c, k, s, pa, n, and p mean the kernel number, kernel size, stride, padding, block number, and dropout probability, respectively. An ME image I is fed to CNN; subsequently, the output features F are fed to the decoder.
The attention-based decoder is based on RNN and iteratively generates the target sequence Y from features F . At time step t, the probability of generating symbol y t depends on context c t , and the current hidden state h t from the RNN output, as shown in Eq. 1: where g(·) is a linear function followed by a soft-max function. We denote the size of features F as L, where L = H × W ; H and W are the height and width of features F , respectively. Context c t is computed as a weighted sum of features F , as illustrated by Eq. 2, and α t,l is the weight of the lth features of F at time step t. As the human visual mechanism does, an attention-based decoder concentrates only on a subset of features at every time step. Trained by the back-propagation algorithm using gradient descent, the decoder can be endowed with the capability to determine which features are important for generating word y t at current time step t. We choose a linear function with the activation function tanh to compute the attention weights and normalization by soft-max function in Eq. 3 and Eq. 4, where W e , W h , W f , W q , and W s are trainable parameters. Referring to [10], when calculating the attention weights, position embeddings q l are considered to make the decoder position sensitive. As shown in Eq. 5, E p,h and E p,v are absolute position embeddings in horizontal and vertical direction, respectively. Referring to [8], coverage features s l are adopted to alleviate over-attention and underattention. As shown in Eq. 6, s l are the sum of the past attention weights.
The hidden state h t is computed by a RNN with long shortterm memory (LSTM) cells. At time step t, the RNN input is the word embedding of y t−1 concatenated with context c t−1 , which is a weighted sum of not only CNN features F but also position embeddings Q at last time step, as illustrated by Eq. 8.

B. Scale augmentation
Unlike handwritten texts, an ME has a complex twodimensional structure and symbols of various sizes. If multiline MEs are normalized to the same scale though keeping the aspect ratio, some symbols (e.g., superscript, subscript, and dot) are smaller than other symbols as shown in Fig. 2(b), which increases the recognition difficulty. Instead of normalizing MEs to the same scale, we augment MEs according to Eq. 9, x = kx, y = ky, where k is the scaling factor and we keep the aspect ratio constant. In the training phase, MEs are augmented to another scale randomly and are zero-padded to the fixed size, as shown in Fig. 2(c). In the test phase, MEs are zero-padded to the fixed size without augmentation. The encoder-decoder network is trained to adapt to MEs of various scales and generate correct predictions.

C. Drop attention
As illustrated previously, attention weight α has a great impact on recognition performance. When attention neglects the key features, the model will generate an incorrect prediction and perform worse. Inspired by [13], we propose a drop attention module in the decoder. First, we randomly suppress the features where the corresponding attention weight α is the highest. Subsequently, we randomly abandon spots on the feature maps, as in Eq. 10.
where γ is the suppress factor; r p and r s are random values and obey the Bernoulli distribution. Features f l replace f l in Eq. 2 and Eq. 8 in training phase, which assists our model to predict the correct symbol or spatial relationship when attention is imprecise.

A. Datasets
The CROHME competition dataset is the largest dataset for HMER [25] [26]. This dataset contains 101 classes of math symbols. In this paper, our model was trained on CROHME 2014 training set (containing 8834 MEs) and validated on CROHME 2013 test set (containing 671 MEs). We report the test results of expression recognition rate (ExpRate) on CROHME 2014 test set and CROHME 2016 test set, which contains 986 and 1147 MEs, respectively. ExpRate ≤ 1(%), ≤ 2(%), and ≤ 3(%) denote the expression recognition rates when one, two, or three symbol-level errors are tolerable. We convert the output LaTeX strings to label graphs and evaluate the performance with the official tools provided by the CROHME 2019 organizers [27]. The experiment settings are consistent with the settings used by the competition participants [25] [26] and in most previous studies [28] [8] [29] [9] [10].

B. Implementation
Our model was implemented in PyTorch and optimized on Nvidia TITAN Xp GPU. The batch size was set to 8 for parallel computing. ME images underwent scale augmentation in the training phase and were zero-padded to a fixed size 256 × 1024. Few ME images were larger than this size before zero-padding, and they were downsized to be smaller than this size. Scaling factor k were set within [0.5, 2] randomly at every training iteration. Features were extracted by CNN from the input image, with configurations shown in Tab. I, and then fed to attention-based decoder. A drop attention module was applied in the training phase. The suppress factor was set as 0.1. Bernoulli probabilities of r p and r s were 0.8 and 0.4, respectively. We calculated the cross-entropy loss between the generated symbols and ground truth. Adam optimizer was used to train the model, and the learning rate was 0.0001.

C. Effect of scale augmentation
We conducted comparative experiments about methods for processing ME images and without applying a drop attention module, as shown in Tab. II. The first method involves normalizing ME images to a fixed height of 256 while keeping the aspect ratios, as shown in Fig. 2(b). The second method is zero-padding ME images to a fixed size while the third method is applying scale augmentation before zero-padding as shown in Fig. 2(c). As illustrated in Tab. II, the results with fixed height normalization are slightly better than that with zero-padding on CROHME 2016 but worse on CROHME 2014. However, both these two methods achieve worse result than the proposed scale augmentation because they are weak on MEs of various scales. The results with scale augmentation outperform zero-padding about 5% in ExpRate, which proves that scale augmentation is effective. The model trained with MEs using scale augmentation could extract ME discriminative features from various scales, which is important for generating a correct prediction. Therefore, we applied scale augmentation in the subsequent experiments.

D. Effect of drop attention
We conducted the comparative experiments to verify the effect of drop attention, and the results are shown in Tab. III. As illustrated in Tab. III, when equipped with the drop attention module, the accuracy of our model improves by 1.42% on CROHME 2016 and 2.1% on CROHME 2014. The drop attention module through suppressing or abandoning some features trains model generating correct predictions when attention neglects the key features. To further improve the performance of our model, the drop attention module was applied in the training phase of the subsequent experiments.

E. Comparison with the proposed methods
Tab. IV compares the results of our model with the systems submitted at CROHME 2014 and recent offline attention-based HMER models. Systems I to VII were participating systems except for system III because of using private training data. We do not present results from [21], [22] and [23] because their models were trained on handwriting trajectory data. A trajectory provides handwriting order information, which is useful for distinguishing visually similar symbols (e.g., "α" and "a"). The attention-based models listed in Tab. IV are all trained on offline ME images with only LaTeX level labels, including our method. The sign " * " in Tab. IV denotes using an ensemble of five differently initialized recognition models to improve the performance [8]. In the case of with and without ensemble, our method outperforms the state-of-the-art model PAL-v2 by 5.58% and 7.71%, respectively.
Tab. V compares the results of our model with the systems submitted at CROHME 2016 and recent offline attention-based   [18] 28.70 ---End-to-end [28] 35.19 ---WAP * [8] 44.40 58.40 62.20 63.10 PAL [29] 39.66 ---PAL * [29] 47.06 ---DenseMSA * [19] 52 HMER models. Our method outperforms "Wiris" system with a large margin, which won the first place in CROHME 2016 using only the official handwritten MEs training data. Moreover, our method also performs better than the stateof-the-art model PAL-v2 by 4.97% in the case of without ensemble and slightly better than with ensemble by 0.17%. In general, the results have shown that our method has excellent performance and potential to achieve higher performance.  features from the CNN encoder are summed according to the attention weight. The model outputs one prediction symbol and composes a LaTex string gradually until the output symbol is "\eos", which means "end of string." Thus, our model can correctly predict symbols of various sizes and the relationships between them. The decoder can focus on discriminative features of symbols and be "distracted" when outputting grammar symbols, such as "{". Because we used RNN network to compose our decoder, there is an implicit language model learned in training phase. Therefore, when generating symbol " " or "∧" at the last time step, our model can generate "{" without referring to the attention features.

V. CONCLUSION
In this paper, we propose a scale augmentation method to address the recognition problem of MEs of various scales, which is caused by the complex two-dimensional structure of ME. Moreover, a new drop attention module is proposed to further improve performance by training the model to generate predictions when attention is imprecise. The experimental results indicated that these two technologies assist our method achieving state-of-the-art performance, compared with the previous methods without an ensemble. In future work, we plan to extend our method to deal with online HMER.