The Impact of Preprocessing on Arabic-English Statistical and Neural Machine Translation
Mai Oudah Amjad Almahairi Nizar Habash

[
    C. Figueira de Morisson Faria and 
	A. S. Maxwell

    December 30, 2023
======================================================




utf8

Neural networks have become the state-of-the-art approach for machine translation (MT) in many languages. 

While linguistically-motivated tokenization techniques were shown to have significant effects on the performance of statistical MT, it remains unclear if those techniques are well suited for neural MT. 


In this paper, we systematically compare neural and statistical MT models for Arabic-English translation on data preprecossed by various prominent tokenization schemes. Furthermore, we consider a range of data and vocabulary sizes and compare their effect on both approaches.
Our empirical results show that the best choice of tokenization scheme is largely based on the type of model and the size of data. We also show that we can gain significant improvements using a system selection that combines the output from neural and statistical MT.





§ INTRODUCTION


Neural machine translation (NMT) has been rapidly attracting the attention of the research community for its promising results <cit.>. NMT is composed of two neural networks, an encoder and a decoder, where the encoder is fed a sentence from the source language and the decoder generates its translation, word by word, in the target language. Recently, NMT has been shown to outperform other MT systems in many language pairs, e.g. German-English, French-English and Basque-English <cit.>. While Arabic MT has been mostly developed under statistical MT (SMT), NMT has also been applied and studied recently <cit.>. 

Linguistically-motivated tokenization has shown to have a significant effect on SMT, particularly in the case of morphologically rich languages like Arabic <cit.>. However, it remains unclear if such techniques are well suited for NMT, where language-agnostic tokenizations, e.g. byte-pair encoding (BPE) <cit.>, are widely used.
almahairi2016first has looked into Arabic SMT and NMT, achieving the highest accuracy using the Penn Arabic Treebank (ATB) tokenization, with 51.2  and 49.7 BLEU points for SMT and NMT, respectively. 

In this paper, we study the impact of different preprocessing techniques in Arabic-English MT on both SMT and NMT, by examining various prominent tokenization schemes.
We conduct learning curve experiments to study the interaction between data size and the choice of tokenization scheme. We study the performance under morphology-based and frequency-based tokenization schemes, provided by MADAMIRA <cit.> and BPE, respectively, on in-domain data. In addition, we evaluate the best performing models on out-of-domain data. Our results show that the utilization of BPE for SMT can be effective and allows achieving a good performance even with a small vocabulary size of 20K. Moreover, the results show that the performance of NMT is especially sensitive to the size of data. We notice that NMT suffers with long sentences, and thus, we utilize system selection, which yields significant improvements over both approaches. Our best system significantly outperforms previous results reported on the same in-domain test data by +4 BLEU points <cit.>.

The rest of the paper is organized as follows. The related work is presented in Section 2. Section 3 describes our proposed approach. Section 4 illustrates the experimental settings. The results are reported in Section 5. In Section 6, we discuss our findings. Finally, we conclude the paper and mention the future work in Section 7.




 



§ RELATED WORK


Many studies have compared the performance of different MT models on translation tasks <cit.>. However, the data preprocessing was not unified across those models. For example, BPE is only applied to the training data utilized by the NMT system, but not SMT <cit.>.

HabashS06 investigated and compared across some preprocessing schemes for Arabic, describing and evaluating different methods for combining them. The main preprocessing schemes were Simple Tokenization, Decliticization (degrees 1 to 3), and Arabic Treebank Tokenization. Decliticization of degree 2 outperformed the rest when applied individually. They reported improvement in MT performance when combining different schemes together.

almahairi2016first compared NMT and SMT on Arabic translation, and showed that NMT performs comparably to SMT. The best performance is achieved when Penn Arabic Treebank (ATB) tokenization is used with 51.19  and 49.70 BLEU points for SMT and NMT, respectively. 

The idea of system selection for MT exists in the literature, but mostly for model selection under the same approach (SMT or NMT) <cit.>. 

 



§ APPROACH

In our study, we systematically compare SMT and NMT on the following dimensions.



 §.§ Source Language Tokenization

Much research has shown the importance of tokenization and orthographic normalization for SMT and NMT, as they deal with data sparsity <cit.>. Tokenization schemes can either be morphology-based or statistical/frequency-based <cit.>. We investigate both in the context of Arabic MT, both separately and in combination, to observe their interaction. We normalize Alif `ا' and Ya `ي' in all schemes, where Hamza is removed from the variants of Hamzated Alif (e.g.`أ',`إ') to become `<a>', the Alif Maqsura `ى' is replaced with Ya `ي' and the diacritics are removed. 
 
 Morphology-based This tokenization scheme relies on the linguistic rules of the source language. We explore three schemes under this category <cit.>:  1) Simple Tokenization (Raw) that splits off punctuation and numbers; 2) Penn Arabic Treebank (ATB) Tokenization, which splits all clitics except definite articles; 3) Decliticization (D3), which splits all clitics.
 
 Frequency-based We use byte-pair encoding (BPE) <cit.>, which is an iterative compression approach that replaces the most frequent pair of characters in a sentence with a unique sequence of characters. It allows for a fixed-size vocabulary representation. Figure <ref> shows an example across Raw and Tok schemes with/without BPE on top.

 




 §.§ Training Data Size

We conduct a learning curve experiment to explore how much both Arabic-English SMT and NMT can benefit from adding more training data with each tokenization scheme. HabashS06 have conducted a similar learning curve study for SMT.
Each tokenization scheme may result in a different number of tokens per sentence; hence, a sentence-length filter will discard more sentences from more verbose schemes. This would lead to some schemes having access to more words than others. 
Therefore, we adopt el2012orthographic's approach of filtering training parallel data based on the D3 scheme as a reference scheme for selecting sentences of length up to 100 tokens. Thus, the same sentences will be selected across different tokenization schemes. 



 §.§ Target Language Resources

 We design the training so that both systems will have access to the same additional target language resources besides the target side of the training parallel corpus. In SMT, target language resources are used to build language models for fluency improvement. Whereas, many works have proven pretrained word embeddings to be useful in neural network models <cit.>, and therefor, the same additional TLR are used to learn pretrained word embeddings that support the decoder in NMT. 
Here, the _tgt++ designation next to the system name indicates the use of additional TLR. 



 §.§ Input Length and System Selection

Many have reported NMT performing worse with long sentences <cit.>, which was caught in our error analysis and thus we explored combining the two MT systems via a system selection approach, where the selection of either translation is based on which is closer to the input length as a criterion. Whereas the sentence BLEU score is the criterion in the Oracle system selection.




§ EXPERIMENTAL SETTINGS

 





 §.§ Datasets

The training dataset contains 1.2M sentence pairs in newswire (NW) domain from three Linguistic Data Consortium (LDC) resources: LDC2004T18, LDC2004T14, and LDC2007T08. For tuning, we use LDC2010T12 (MT04), which consists of 1,075 sentence pairs in NW and government documents. As for the in-domain testing, we use LDC2010T14 (MT05), which consists of 1,056 sentence pairs in NW, and has four English reference translations. We look into the performance of the systems in out-of-domain data using LDC2014T02 (MT12), which consists of 1,535 sentence pairs mostly web collection, and has four English reference translations.



 §.§ Preprocessing
 
MADAMIRA <cit.> is utilized for morphology-based tokenization of the source side. bpe's BPE implementation is used for learning and applying BPE models. We set vocabulary size to 20K in BPE learning after exploring multiple vocabulary sizes, including 10K, 20K and 30K, where the 20K setting achieved comparable results to the 30K and outperformed the 10K. Each BPE model is trained on source side of training data of the respective experiment. While Moses' <cit.> tokenizer and lowercaser are used for preparing the target side. 



 §.§ SMT settings

We use Moses 3.0 <cit.> to train SMT models with maximum phrase length of 8 tokens. Two versions of the language model are examined: 1) trained solely on the target side of the training dataset, and 2) trained on the target side and the English Gigaword 5th edition. 



 §.§ NMT settings

We use the encoder-decoder with the general global attention architecture as introduced by Luong-2015.

All the NMT models have been trained using OpenNMT toolkit <cit.> with no restriction on input's vocabulary.

We use long short-term memory units (LSTM) <cit.>, with hidden units of size 500 and two layers in both the encoder and decoder. The word embedding vector size for source/target is 300. 

English pretrained word embeddings were trained as skip-gram model <cit.> via gensim tool <cit.> with settings: (size=300, window=8, min count=5) on English Gigaword 5th edition <cit.> dataset. Arabic embeddings were trained on the Arabic Gigaword 5th edition <cit.> via FastText <cit.>, which showed better performance with morphologically rich languages <cit.>. We give the designation of _src/tgt++ to the system that uses both embeddings.






 §.§ Evaluation Metrics

The evaluation results are reported in case insensitive BLEU scores <cit.> with their confidence intervals (CI) and p-values. Bootstrap resampling is used to compute statistical significance intervals <cit.>.













§ RESULTS



 §.§ Preprocessing and Learning Curve

We examine Raw, ATB and D3 with and without BPE applied on top, across a learning curve where smaller sets of our training data (1.2M) are considered at 25% (300K) and 6.25% (75K) tokens. Figure <ref> illustrates the learning curve results for Raw (baseline) and ATB (overall best), with and without BPE. Figure <ref> shows the importance of training data availability, especially for NMT, and also that BPE impact can be seen in both systems, which we find interesting. Moreover, SMT is shown to be far more sensitive to preprocessing than NMT. 

Table <ref> shows the best systems' results when 100% of the training data is tokenized by Raw, ATB and D3, with and without BPE on top of it, for SMT and NMT. It shows ATB+BPE and ATB to achieve the best results for SMT and NMT, respectively, which we find interesting as BPE is usually associated with NMT. The p-value indicates whether the difference between SMT and NMT results under the same tokenization scheme is statistically significant or not. The statistical significance is illustrated with p-value <0.05. 
 and  have comparable results at the baseline.

As expected, using more data for LM produces better results as well as the increase in training data size. 

Using pretrained word embeddings for both languages improve the NMT results significantly compared to only target ones by two BLEU points. As shown in Table <ref>, the best  model (using ATB) outperforms  model (using ATB+BPE) by 1.5 BLEU against MT12 in the out-of-domain testing. 



 §.§ Error analysis

Error analysis has shown that NMT output is more fluent than SMT's, especially with short sentences (<50 tokens), in contrast to long sentences where coverage and accuracy drop, which support related work. Figure <ref> shows dropping in NMT output size as the input size increases, especially <40 tokens, while SMT keeps more consistent output to the input size. 
So we explore system selection based on the closeness to the input length as well as Oracle results, where the selection is based on the highest output BLEU score. 
Tables <ref> and <ref> show the results of length-based system selection on best models in SMT and NMT when applied to in-domain test (MT05) and out-of-domain testing (MT12), respectively, which illustrate improvements over the original BLEU scores.

Figure <ref> illustrates five examples, where either SMT or NMT output is selected based on the output length compared to the source input size. 
In Example 1, the SMT output is selected over the NMT one as the NMT system drops the phrase after the comma and only translates the part before, however, fluently. Example 2, which represents much longer sentence, SMT output is selected over NMT, which translates the saying and drops the rest of the sentence. On the other hand, in Examples 3 and 4, NMT output is selected over SMT's for being the closest to the source input in terms of length. Furthermore, Example 5 represents a case where the system selection approach fails to select the better prediction (in terms of BLEU score) for the final output based on the source-output length comparison.




§ DISCUSSION

We notice that morphology-based tokenization schemes improve the performance of MT systems regardless of the MT approach, but in different levels. The difference in scheme choice is less impactful on NMT; compared with SMT. The improvement range for NMT is 1.13 BLEU, while for SMT the range is 2.86 BLEU. While Raw results are almost the same for SMT and NMT; ATB improves both NMT and SMT; but the improvement is higher for SMT. Adding BPE helps SMT, while lowering vocabulary size. The effect of BPE on NMT is insignificant, which is a surprising result since BPE is often associated with NMT. 



Also, we significantly improve on almahairi2016first's results by more than three BLEU points. 

Length-based system selection improves over both NMT and SMT results in in-domain and out-of-domain cases, significantly in the later, which indicates a hybrid MT system may be promising. Moreover, the huge jump in performance with Oracle selection shows that there is still room for potential improvement in system designs, for better accuracy and fluency. More TLR allow for better results in MT systems. When both Arabic and English pretrained word embeddings are used, the performance improves by more than two BLEU points compared to English only. 



§ CONCLUSION AND FUTURE WORK

In this paper, we study the impact of various preprocessing techniques to Arabic-English MT under SMT and NMT, where  various  prominent tokenization schemes are examined. 
We conduct a learning curve analysis of the different preprocessing settings with incremental training data size, where ATB scheme performs consistently well along the learning curve. Moreover, we implemented a length-based system selection to deal with NMT's struggle with short sentences, and significant improvements. 
The empirical results show that the choice of tokenization scheme can be optimized based on the type of model to train and the data available. We also gain significant improvements using length-based system selection that combines the output from neural and statistical MT.
Our results significantly outperform the ones reported in the prior work when applied to in-domain test (MT05). As future work, we plan to examine training data of general domain with linguistically-motivated tokenization schemes to study further their impact on NMT under different neural models. Also, exploring sophisticated system selection schemes for potential improvement.
 



§ ACKNOWLEDGMENTS

The support and resources from the High Performance Computing Center at New York University Abu Dhabi are gratefully acknowledged.

mtsummit2019


