Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Hai Wang Dian Yu Kai Sun Janshu Chen Dong Yu

	
UTF8gkai

The Infall Motion in the Low-Mass Protostellar Binary NGC 1333 IRAS 4A1/4A2
    I-Ta Hsieh
     
===========================================================================



* This work was done when H. W. and K. S. were at Tencent AI Lab, Bellevue, WA. 




Recently, pre-trained language models have achieved remarkable success in a broad range of natural language processing tasks. However, in multilingual setting, it is extremely resource-consuming to pre-train a deep language model over large-scale corpora for each language. Instead of exhaustively pre-training monolingual language models independently, an alternative solution is to pre-train a powerful multilingual deep language model over large-scale corpora in hundreds of languages. However, the vocabulary size for each language in such a model is relatively small, especially for low-resource languages. This limitation inevitably hinders the performance of these multilingual models on tasks such as sequence labeling, wherein in-depth token-level or sentence-level understanding is essential.

In this paper, inspired by previous methods designed for monolingual settings, we investigate two approaches (, joint mapping and mixture mapping) based on a pre-trained multilingual model BERT for addressing the out-of-vocabulary (OOV) problem on a variety of tasks, including part-of-speech tagging, named entity recognition, machine translation quality estimation, and machine reading comprehension. Experimental results show that using mixture mapping is more promising. To the best of our knowledge, this is the first work that attempts to address and discuss the OOV issue in multilingual settings. 

















 



§ INTRODUCTION




It has been shown that performance on many natural language processing tasks drops dramatically on held-out data when a significant percentage of words do not appear in the training data, , out-of-vocabulary (OOV) words <cit.>. A higher OOV rate (, the percentage of the unseen words in the held-out data) may lead to a more severe performance drop <cit.>.
OOV problems have been addressed in previous works under monolingual settings, through replacing OOV words with their semantically similar in-vocabulary words <cit.> or using character/word information <cit.> or subword information like byte pair encoding (BPE) <cit.>. 




Recently, fine-tuning a pre-trained deep language model, such as Generative Pre-Training (GPT) <cit.> and Bidirectional Encoder Representations from
Transformers (BERT) <cit.>, has achieved remarkable success on various downstream natural language processing tasks. Instead of pre-training many monolingual models like the existing English GPT, English BERT, and Chinese BERT, a more natural choice is to develop a powerful multilingual model such as the multilingual BERT. 


However, all those pre-trained models rely on language modeling, where a common trick is to tie the weights of softmax and word embeddings <cit.>. Due to the expensive computation of softmax <cit.> and data imbalance across different languages, the vocabulary size for each language in a multilingual model is relatively small compared to the monolingual BERT/GPT models, especially for low-resource languages. Even for a high-resource language like Chinese, its vocabulary size 10k in the multilingual BERT is only half the size of that in the Chinese BERT. Just as in monolingual settings, the OOV problem also hinders the performance of a multilingual model on tasks that are sensitive to token-level or sentence-level information. For example, in the POS tagging problem (Table <ref>), 11 out of 16 languages have significant OOV issues (OOV rate ≥ 5%) when using multilingual BERT.





According to previous work <cit.>, it is time-consuming and resource-intensive to pre-train a deep language model over large-scale corpora. To address the OOV problems, instead of pre-training a deep model with a large vocabulary, we aim at enlarging the vocabulary size when we fine-tune a pre-trained multilingual model on downstream tasks. 





We summarize our contributions as follows: (i) We investigate and compare two methods to alleviate the OOV issue. To the best of our knowledge, this is the first attempt to address the OOV problem in multilingual settings. (ii) By using English as an interlingua, we show that bilingual information helps alleviate the OOV issue, especially for low-resource languages. (iii) We conduct extensive experiments on a variety of token-level and sentence-level downstream tasks to examine the strengths and weaknesses of these methods, which may provide key insights into future directions[Improved models will be available at <https://github.com/sohuren/multilingul-bert>.].











































 

 

 





 














§ APPROACH



We use the multilingual BERT as the pre-trained model. We first introduce the pre-training procedure of this model (Section <ref>) and then introduce two methods we investigate to alleviate the OOV issue by expanding the vocabulary (Section <ref>). Note that these approaches are not restricted to BERT but also applicable to other similar models.




 §.§ Pre-Trained BERT



Compared to GPT <cit.> and ELMo <cit.>, BERT <cit.> uses a bidirectional transformer, whereas GPT pre-trains a left-to-right transformer <cit.>; ELMo <cit.> independently trains left-to-right and right-to-left LSTMs <cit.> to generate representations as additional features for end tasks. 

In the pre-training stage, devlin2018bert use two objectives: masked language model (LM) and next sentence prediction (NSP). In masked LM, they randomly mask some input tokens and then predict these masked tokens. Compared to unidirectional LM, masked LM enables representations to fuse the context from both directions. In the NSP task, given a certain sentence, it aims to predict the next sentence. The purpose of adding the NSP objective is that many downstream tasks such as question answering and language inference require sentence-level understanding, which is not directly captured by LM objectives.

After pre-training on large-scale corpora like Wikipedia and BookCorpus <cit.>, we follow recent work <cit.> to fine-tune the pre-trained model on different downstream tasks with minimal architecture adaptation. We show how to adapt BERT to different downstream tasks in Figure <ref> (left). 








 §.§ Vocabulary Expansion







devlin2018bert pre-train the multilingual BERT on Wikipedia in 102 languages, with a shared vocabulary that contains 110k subwords calculated from the WordPiece model <cit.>. If we ignore the shared subwords between languages, on average, each language has a 1.1k vocabulary, which is significantly smaller than that of a monolingual pre-trained model such as GPT (40k). The OOV problem tends to be less serious for languages (, French and Spanish) that belong to the same language family of English. However, this is not always true, especially for morphologically rich languages such as German <cit.>. OOV problem is much more severe in low-resource scenarios, especially when a language (, Japanese and Urdu) uses an entirely different character set from high-resource languages.



We focus on addressing the OOV issue at subword level in multilingual settings. Formally, suppose we have an embedding E_bert extracted from the (non-contextualized) embedding layer in the multilingual BERT (i.e., the first layer of BERT). And suppose we have another set of (non-contextualized) sub-word embeddings {E_l_1, E_l_2, …, E_l_n}∪{E_en}, which are pre-trained on large corpora using any standard word embedding toolkit. Specifically, E_en represents the pre-trained embedding for English, and E_l_i represents the pre-trained embedding for non-English language l_i at the subword level. We denote the vocabulary of E_bert, E_en, and E_l_i by V_bert, V_en, and V_l_i, respectively. For each subword w in V_bert, we use E_bert(w) to denote the pre-trained embedding of word w in E_bert. E_l_i(·) and E_en(·) are defined in a similar way as E_bert(·). For each non-English language l ∈{l_1, l_2, …, l_n}, we aim to enrich E_bert with more subwords from the vocabulary in E_l_i since E_l_i contains a larger vocabulary of language l_i compared to E_bert. 


As there is no previous work to address multilingual OOV issues, inspired by previous solutions designed for monolingual settings, we investigate the following two methods, and all of them can be applied at both word/subword level, though we find subword-level works better (Section <ref>). 







Joint Mapping For each non-English language l, we first construct a joint embedding space E'_l through mapping E_l to E_en by an orthogonal mapping matrix B_l (, E'_l=E_lB_l). When a bilingual dictionary f_l:V_l→ V_en is available or can be constructed based on the shared common subwords (Section <ref>), we obtain B_l by minimizing:

    ∑_w'∈ V_l∩{ w: f_l(w) ∈ V_en}E_l(w')B_l - E_en(f_l(w'))_F^2



where ·_F denotes the Frobenius norm. Otherwise, for language pair (, English-Urdu) that meets neither of the above two conditions, we obtain B_l by an unsupervised word alignment method from MUSE <cit.>.


We then map E'_l to E_bert by an orthogonal mapping matrix A'_l, which is obtained by minimizing

    ∑_w∈ f_l(V_l) ∩ V_bertE'_l(w)A'_l - E_bert(w)_F^2

We denote this method by M_J in our discussion below, where the subscript J stands for “joint”.


Mixture Mapping Following the work of gu2018universal where they use English as “universal tokens” and map all other languages to English to obtain the subword embeddings, we represent each subword in E'_l (described in joint mapping) as a mixture of English subwords where those English subwords are already in the BERT vocab V_bert. This method, denoted by M_M, is also a joint mapping without the need for learning the mapping from E'_l to E_bert. Specifically, for each w∈ V_l, we obtain its embedding e(w) in the BERT embedding space E_bert as follows. 

    e(w) = ∑_u ∈𝒯(w) p(u | w)E_bert(u)

where 𝒯(w) is a set to be defined later, and the mixture coefficient p(u|w) is defined by

    p(u | w) = exp(CSLS(E_l(u), E_en(w)))/∑_v ∈𝒯(w)exp(CSLS(E_l(v), E_en(w)))

where CSLS refers to the distance metric Cross-domain Similarity Local Scaling <cit.>. We select five v ∈ V_en∪ V_bert with the highest CSLS(E_l(v), E_en(w)) to form set 𝒯(w). In all our experiments, we set the number of nearest neighbors in CSLS to 10. We refer readers to conneau2017word for details. Figure <ref> (right) illustrates the joint and mixture mapping. 























































§ EXPERIMENT





 §.§ Experiment Settings



We obtain the pre-trained embeddings of a specific language by training fastText <cit.> on Wikipedia articles in that language, with context window 5 and negative sampling 5. Before training, we first apply BPE <cit.> to tokenize the corpus with subword vocabulary size 50k. For joint mapping method M_J, we use bilingual dictionaries provided by conneau2017word. For a language pair where a bilingual dictionary is not easily available, if two languages share a significant number of common subwords (this often happens when two languages belong to the same language family), we construct a bilingual dictionary based on the assumption that identical subwords have the same meaning <cit.>. We add all unseen subwords from 50k vocabulary to BERT. We define a word as an OOV word once it cannot be represented as a single word. For example, in BERT, the sentence “Je sens qu' entre ça et les films de médecins et scientifiques" is represented as “je sens qu ##' entre [UNK] et les films de [UNK] et scientifiques", where qu' is an OOV word since it can only be represented by two subword units: qu and ##', but it is not OOV at subword level; ça and médecins cannot be represented by any single word or combination of subword units, and thus they are OOV at both word and subword level.    



We use the multilingual BERT with default parameters in all our experiments, except that we tune the batch size and training epochs. To have a thorough examination about the pros and cons of the explored methods, we conduct experiments on a variety of token-level and sentence-level classification tasks: part of speech (POS) tagging, named entity recognition (NER), machine translation quality estimation, and machine reading comprehension. See more details in each subsection.    





 §.§ Discussions about Mapping Methods

Previous work typically assumes a linear mapping exists between embedding spaces of different languages if their embeddings are trained using similar techniques <cit.>. However, it is difficult to map embeddings learned with different methods <cit.>. Considering the differences between BERT and fastText: , the objectives, the way to differentiate between different subwords, and the much deeper architecture of BERT, it is very unlikely that the (non-contextualized) BERT embedding and fastText embedding reside in the same geometric space. Besides, due to that BERT has a relatively smaller vocabulary for each language, when we map a pre-trained vector to its portion in BERT indirectly as previous methods, the supervision signal is relatively weak, especially for low-resource languages. In our experiment, we find that the accuracy of the mapping from our pre-trained English embedding to multilingual BERT embedding (English portion) is lower than 30%. In contrast, the accuracy of the mapping between two regular English embeddings that are pre-trained using similar methods (e.g., CBOW or SkipGram <cit.>) could be above 95% <cit.>.

Besides the joint mapping method M_J (Section <ref>), another possible method that could be used for OOV problem in multilingual setting is that, for each language l, we map its pre-trained embedding space E_l to embedding E_bert by an orthogonal mapping matrix A_l, which is obtained by minimizing ∑_w∈ V_l∩ V_bertE_l(w)A_l - E_bert(w)_F^2. This approach is similar to <cit.>, and is referred as independent mapping method below. However, we use examples to demonstrate why these kind of methods are less promising. In Table <ref>, the first two rows are results obtained by mapping our pre-trained English embedding (using fastText) to the (non-contextualized) BERT embedding. In this new unified space, we align words with CSLS metric, and for each subword that appears in English Wikipedia, we seek its closest neighbor in the BERT vocabulary. Ideally, each word should find itself if it exists in the BERT vocabulary. However, this is not always true. For example, although “however" exists in the BERT vocabulary, independent mapping fails to find it as its own closest neighbor. Instead, it incorrectly maps it to irrelevant Chinese words “盘” (“plate”) and “龙” (“dragon”). A similar phenomenon is observed for Chinese. For example, “册” is incorrectly aligned to Chinese words “书” and “卷”.



Furthermore, our POS tagging experiments (Section <ref>) further show that joint mapping M_J does not improve (or even hurt) the performance of the multilingual BERT. Therefore, we use mixture mapping M_M to address and discuss OOV issues in the remaining sections.













































 §.§ Monolingual Sequence Labeling Tasks


POS Tagging: We use the Universal Dependencies
v1.2 data <cit.>. For languages with token segmentation ambiguity, we use the gold segmentation following plank2016multilingual. We consider languages that have sufficient training data and filter out languages that have unsatisfying embedding alignments with English (accuracy is lower than 20.0% measured by word alignment accuracy or 0.25 by unsupervised metric in MUSE <cit.>). Finally, we keep 16 languages. We use the original multilingual BERT (without using CRF <cit.> on top of it for sequence labeling) to tune hyperparameters on the dev set and use the fixed hyperparameters for the expanded multilingual model. We do not tune the parameters for each model separately. As shown in Table <ref>, at both the word and subword level, the OOV rate in this dataset is quite high. Mixture mapping improves the accuracy on 10 out of 16 languages, leading to a 1.97% absolute gain in average. We discuss the influence of alignments in Section <ref>.

















Chinese NER: We are also interested in investigating the performance gap between the expanded multilingual model and a monolingual BERT that is pre-trained on a large-scale monolingual corpus. Currently, pre-trained monolingual BERT models are available in English and Chinese. As English has been used as the interlingua, we compare the expanded multilingual BERT and the Chinese BERT on a Chinese NER task, evaluated on the Weibo NER dataset constructed from social media by peng2015named. In the training set, the token-level OOV rate is 2.17%, and the subword-level OOV rate is 0.54%. We tune the hyperparameters of each model based on the development set separately and then use the best hyperparameters of each model for evaluation on the test set. 

As shown in Table <ref>, the expanded model outperforms the multilingual BERT on the Weibo NER dataset. We boost the F1 score from 59.0% to 61.4%. Compared to the Chinese BERT (66.9%), there still exists a noticeable performance gap. One possible reason could be the grammatical differences between Chinese and English. As BERT uses the language model loss function for pre-training, the pre-trained Chinese BERT could better capture the language-specific information comapred to the multilingual BERT.










 §.§ Code-Mixed Sequence Labeling Tasks



As the multilingual BERT is pre-trained over 102 languages, it should be able to handle code-mixed texts. Here we examine its performance and the effectiveness of the expanded model in mixed language scenarios, using two tasks as case studies. 

Code-Switch Challenge: We first evaluate on the CALCS-2018 code-switched task <cit.>, which contains two NER tracks on Twitter social data: mixed English&Spanish (en-es) and mixed Modern Standard Arabic&Egyptian (ar-eg). Compared to traditional NER datasets constructed from news, the dataset contains a significant portion of uncommon tokens like hashtags and abbreviations, making it quite challenging. For example, in the en-es track, the token-level OOV rate is 44.6%, and the subword-level OOV rate is 3.1%; in the ar-eg track, the token-level OOV rate is 64.0%, and the subword-level OOV rate is 6.0%. As shown in Table <ref>, on ar-eg, we boost the F1 score from 74.7% to 77.3%. However, we do not see similar gains on the en-es dataset, probably because that English and Spanish share a large number of subwords, and adding too many new subwords might prevent the model from utilizing the well pre-trained subwords embedding. See Section <ref> for more discussions.




 





<cit.> code switch on en and es, method is simple and potentially we can use the data.
<cit.>
train: 
en-es:
OOV@token: 0.446437
OOV@subword: 0.030605
en-es (enriched)
OOV@token: 0.416830
OOV@subword: 0.013943
train:
ar-et:
OOV@token: 0.64
OOV@subword: 0.06
ar-et(enriched)
OOV@token:0.54 
OOV@subword:0.00888






Machine Translation Quality Estimation:
 All previous experiments are based on well-curated data. Here we evaluate the expanded model on a language generation task, where sometimes the generated sentences are out-of-control. 

 We choose the automatic Machine Translation Quality Estimation task and use Task 2 – word-level quality estimation – in WMT18 <cit.>. Given a source sentence and its translation (, target), this task aims to estimate the translation quality (“BAD" or “OK") at each position: , each token in the source and target sentence, each gap in the target sentence. We use English to German (en-de) SMT translation. On all three categories, the expanded model consistently outperforms the original multilingual BERT (Table <ref>)[Our evaluation is based on the development set since the test set is only available to participants, and we could not find the submission teams' performance on the developmenet set.].
 

 




 §.§ Sequence Classification Tasks


 
Finally, we evaluate the expanded model on sequence classification in a mixed-code setting, where results are less sensitive to unseen words.  











































Code-Mixed Machine Reading Comprehension:
We consider the mixed-language machine reading comprehension task. Since there is no such public available dataset, we construct a new Chinese-English code-mixed machine reading comprehension dataset based on 37,436 unduplicated utterances obtained from the transcriptions of a Chinese and English mixed speech recognition corpus King-ASR-065-1[http://kingline.speechocean.com.]. We generate a multiple-choice machine reading comprehension problem (, a question and four answer options) for each utterance. A question is an utterance with an English text span removed (we randomly pick one if there are multiple English spans) and the correct answer option is the removed English span. Distractors (, wrong answer options) come from the top three closest English text spans, which appear in the corpus, based on the cosine similarity of word embeddings trained on the same corpus. For example, given a question “突然听到 21     ，那强劲的鼓点，那一张张脸。” (“Suddenly I heard 21     , and the powerful drum beats reminded me of the players.”) and four answer options { “forever”, “guns”, “jay”, “twins” }, the task is to select the correct answer option “guns” (“21 Guns” is a song by the American rock band Green Day). We split the dataset into training, development, and testing of size 36,636, 400, 400, respectively. Annotators manually clean and improve the quality problems by generating more confusing distractors in the dev and testing sets to guarantee that these problems are error-free and challenging.



In this experiment, for each BERT model, we follow its default hyperparameters. As shown in Table <ref>, the expanded model improves the multilingual BERT (38.1%) by 1.2% in accuracy. Human performance (81.4%) indicates that this is not an easy task even for human readers. 

 











 §.§ Discussions



In this section, we first briefly investigate whether the performance boost indeed comes from the reduction of OOV and then discuss the strengths and weaknesses of the methods we investigate. 

First, we argue that it is essential to alleviate the OOV issue in multilingual settings. Taking the POS tagging task as an example, we find that most errors occur at the OOV positions (Table <ref> in Section <ref>). In the original BERT, the accuracy of OOV words is much lower than that of non-OOV words, and we significantly boost the accuracy of OOV words with the expanded BERT. All these results indicate that the overall improvement mostly comes from the reduction of OOV.


 


We also observe that the following factors may influence the performance of the expanded model.




Subwords: When expanding the vocabulary, it is critical to add only frequent subwords. Currently, we add all unseen subwords from the 50k vocabulary (Section <ref>), which may be not an optimal choice. Adding too many subwords may prevent the model from utilizing the information from pre-trained subword embedding in BERT, especially when there is a low word-level overlap between the training and test set. 

Language: We also find that languages can influence the performance of the vocabulary expansion through the following two aspects: the alignment accuracy and the closeness between a language and English. For languages that are closely related to English such as French and Dutch, it is relatively easy to align their embeddings to English as most subword units are shared <cit.>. In such case, the BERT embedding already contains sufficient information, and therefore adding additional subwords may hurt the performance. On the other hand, for a distant language such as Polish (Slavic family), which shares some subwords with English (Germanic family), adding subwords to BERT brings performance improvements. In the meantime, as Slavic and Germanic are two subdivisions of the Indo-European languages, we find that the embedding alignment methods perform reasonably well. For these languages, vocabulary expansion is usually more effective, indicated by POS tagging accuracies for Polish, Portuguese, and Slovenian (Table <ref>). For more distant languages like Arabic (Semitic family) that use different character sets, it is necessary to add additional subwords. However, as the grammar of such a language is very different from that of English, how to accurately align their embeddings is the main bottleneck. 

Task: We see more significant performance gains on NER, POS and MT Quality Estimation, possibly because token-level understanding is more critical for these tasks, therefore alleviating OOV helps more. In comparison, for sequence level classification tasks such as machine reading comprehension (Section <ref>), OOV issue is less severe since the result is based on the entire sentence.   












hyper parameter: batch size, 24, epoch 3.0 tuned on en dev (F1: 94.6).
en + cased base: 

OOV: train: OOV@token: 0.168229 OOV@subword: 0.000000
OOV: test:  OOV@token: 0.178206 OOV@subword: 0.000000
en + uncased big: 

OOV: train: OOV@token: 0.291443, OOV@subword: 0.000000
OOV: test: OOV@token: 0.307742, OOV@subword: 0.000000


en + multi: 

OOV: train: OOV@token: OOV@token: 0.317202, OOV@subword: 0.000007
OOV: test: OOV@token: OOV@token: 0.331323, OOV@subword: 0.000000

en-riched + multi: 

train: OOV@token: 0.293040, OOV@subword: 0.000007
test: OOV@token: 0.309659, OOV@subword: 0.000000

train: OOV@token: 0.262650, OOV@subword: 0.000000
test: OOV@token: 0.298525, 

hyper parameter: batch size, 24 , epoch 4.0 tuned on en dev (F1:96.8 ).
de + multi: 

train: OOV@token: 0.426422, OOV@subword: 0.052064
valid: OOV@token: 0.392126, OOV@subword: 0.049834
test: OOV@token: 0.456496, OOV@subword: 0.051448

de-riched + multi: 

train: OOV@token: 0.382026, OOV@subword: 0.026904
test: OOV@token: 0.416531, OOV@subword: 0.026580


hyper parameter: batch size, 24, epoch 4.0 tuned on en dev (F1: 87.43).
nl + multi: 

train:OOV@token: 0.340449, OOV@subword: 0.004703
valid:OOV@token: 0.344124 OOV@subword: 0.004273
test:OOV@token: 0.330381, OOV@subword: 0.003996


hyper parameter: batch size, 12 , epoch 4.0 tuned on en dev (F1: 81.52 ).
es + multi: 

train: OOV@token: 0.321081,OOV@subword: 0.080659
valid:OOV@token: 0.337169, OOV@subword: 0.075910
test: OOV@token: 0.319058, OOV@subword: 0.080874




NER: For NER, we use the following data sets: Dutch (nl) and Spanish (es) data from the CoNLL 2002 shared task <cit.>, English (en) and German (de) from the CoNLL 2003 shared task <cit.>. 
To give an intuition on how serious out-of-vocabulary problem is, on German training set, with the original multilingual BERT model, the OOV rate at token level is 42.6% and OOV rate at subword is 5.21%. On other language, we observed similar or even higher OOV rate. We use the original multilingual pre-trained BERT model to tune parameters based on dev set, then use those fixed parameters for all other models, including our multilingual model with expanded vocabulary, i.e., we do not tune the parameters for each model separately. We replicate the sequence labeling experiments for BERT, measured on English, we cannot get exactly the same performance due to the unknown hyper parameter (as far we we know, similar problem also existed in other groups). From Table.<ref>, we can see the model M_M can improve the multilingual BERT model. On German, we boost the f1 score from 82.2 to 82.6. On Spanish, we boost the f1 score from 79.8 to 80.68, while this is still lower than previous state-of-the-art (85.88), we suspect this is the issue of pre-trained multilingual BERT model. 















 




 §.§ Sequence Classification

XNLI:  Cross-lingual Natural Language Inference
corpus, or XNLI, by extending these NLI
corpora to 15 languages. XNLI consists of 7500
human-annotated development and test examples
in NLI three-way classification format in English,
French, Spanish, German, Greek, Bulgarian, Russian,
Turkish, Arabic, Vietnamese, Thai, Chinese,
Hindi, Swahili and Urdu, making a total of
112,500 annotated pairs. These languages span
several language families, and with the inclusion
of Swahili and Urdu, include two lower-resource
languages as well









GLUE: The General Language Understanding Evaluation
(GLUE) benchmark (Wang et al., 2018) is a collection
of diverse natural language understanding
tasks. Most of the GLUE datasets have already
existed for a number of years, but the purpose
of GLUE is to (1) distribute these datasets
with canonical Train, Dev, and Test splits, and
(2) set up an evaluation server to mitigate issues
with evaluation inconsistencies and Test set overfitting.
GLUE does not distribute labels for the
Test set and users must upload their predictions to
the GLUE server for evaluation, with limits on the
number of submissions


















  §.§.§ English MRC


RACE
en big, batch size 7 
en base, batch size 12 










  §.§.§ English Bio medical Relation Extraction


drug gene var: english(big/base, uncased) model: OOV@token: 0.229722, OOV@subword: 0.000032
drug gene: OOV@token: 0.202691, OOV@subword: 0.000071


 








WeiboNER: 

multilingual model: 
train: OOV@token: 0.021687, OOV@subword: 0.005400
dev: OOV@token: 0.021297, OOV@subword: 0.007819
test: OOV@token: 0.023110, OOV@subword: 0.006808
best tuned parameters on dev: batch 24 epoch 32
precision:63.83; recall:67.61; FB: 65.67
chinese model: 
train: OOV@token: 0.020819, OOV@subword: 0.004565
test: OOV@token: 0.021224, OOV@subword: 0.004992
sinhanWS: 

multilingual model: train: OOV@token: 0.011758, OOV@subword: 0.010623
dev: OOV@token: 0.010641, OOV@subword: 0.009596
test: OOV@token: 0.011758, OOV@subword: 0.011003
best tuned parameters on dev: batch 18, epoch 6
precision:	0.965, recall:	0.964, f1 0.965
chinese model: 
train: OOV@token: 0.011383, OOV@subword: 0.010307
test: OOV@token: 0.011608, OOV@subword: 0.010856





















§ RELATED WORK





 §.§ Generative Pre-Training

Great success has been achieved by pre-training deep neural networks with language modeling and then fine-tuning the models on downstream tasks <cit.>. It significantly outperforms methods that use pre-trained language models as auxiliary features <cit.>. 


However, it's time-consuming and resource-intensive to pre-train a deep language model such as GPT <cit.> and BERT <cit.>. Considering the computation limitation, it is necessary to pre-train a multilingual model such as Multi-BERT (Section <ref>) that leverage rich linguistic knowledge from large-scale corpus in different languages. However, the vocabulary size is inevitably limited. To get a more powerful pre-trained multilingual model, instead of training from scratch, in this paper, we focus on improving it by enlarging the vocabulary size during the fine-tuning stage. 






OOV poses challenges for many tasks <cit.> such as machine translation <cit.> and sentiment analysis <cit.>. Even for tasks such as machine reading comprehension that are less sensitive to the meanings of each word, OOV still hurts the performance <cit.>. We now discuss previous methods in two settings.        



 §.§ Monolingual Setting


Most previous work address the OOV problems in monolingual settings. Before more fine-grained encoding schema such as BPE <cit.> is proposed, prior work mainly focused on OOV for token-level representations <cit.>. Besides simply assigning random embeddings to unseen words <cit.> or using an unique symbol to replace all these words with a shared embedding <cit.>, a thread of research focuses on refining the OOV representations based on word-level information, such as using similar in-vocabulary words <cit.>, mapping initial embedding to task-specific embedding <cit.>, using definitions of OOV words from auxiliary data <cit.>, and tracking contexts to build/update representations <cit.>.

Meanwhile, there have been efforts in representing words by utilizing character-level <cit.> or subword-level representations <cit.>. To leverage the advantages in character and (sub)word level representation, some previous work combine (sub)word- and character-level representations <cit.> or develop hybrid word/subword-character architectures <cit.>. However, all those approaches assume monolingual setting, which is different from ours.





 §.§ Multilingual Setting

Addressing OOV problems in a multilingual setting is relatively under-explored, probably because most multilingual models use separate vocabularies <cit.>. While there is no direct precedent, previous work show that incorporating multilingual contexts can improve monolingual word embeddings <cit.>.   

madhyastha2017learning increase the vocabulary size for statistical machine translation (SMT). Given an OOV source word, they generate a translation list in target language, and integrate this list into SMT system. Although they also generate translation list (similar with us), their approach is still in monolingual setting with SMT. cotterell2017cross train char-level taggers to predict morphological taggings for high/low resource languages jointly, alleviating OOV problems to some extent. In contrast, we focus on dealing with the OOV issue at subword level in the context of pre-trained BERT model.


















 §.§ Generative Pre-Training

Pre-trained word embeddings <cit.> have been used as an effective initialization method in NLP for a long time. Basically, we can view word embedding as an approximation to language modeling. In days when large scale deep learning models were not widely used, word embeddings is the main pre-training method that succeed in NLP community. However, word embedding only captures the shallow semantics.

Along with the progress in hardware and deep learning model <cit.>, various large scale generative pre-training have been proposed <cit.>, and finally we get BERT <cit.>, in which deep bidirectional transformers are pre-trained through language model with lots of hardware resources.

Since release, BERT has shown significant boost compared with previous state-of-the-art on quite a lot NLP tasks. It seems it will have the same wide-ranging impact on NLP as pretrained ImageNet models had on computer vision <cit.>. However, different from vision, pre-trained model in NLP is language specific and for lots of low resource languages, even finding the large scale monolingual corpus is difficult. To make large-scale pre-trained model really make a big impact on NLP, pre-training in multilingual fashion is necessary. BERT release a pre-trained multilingual model with vocabulary computed from mixed 100 languages. However, due to softmax bottleneck and limitation on hardware, in this vocabulary, the coverage for each language is not big enough, and this is the main problem we want to solve in this paper.  

<cit.>
<cit.>



 §.§ OOV

 
 Out of vocabulary problem has drawn lots of attention, especially for generation task like machine translation <cit.> and  has identified that OOV is one of the main bottleneck for machine translation. However, in classification tasks like machine reading comprehension,
nature language inference and sequence labelling, OOV problem is less serious compared with that in generation task, however, it still influence the downstream task performance significantly. Here, we briefly previous approaches that dealing with this problem.        
 
Monolingual setting: Most approaches for OOV only consider the single language. Before more fine-grained encoding schema introducing, most OOV are based on token level <cit.>. In this case, the key is how to get the embedding for unseen token. madhyastha2016mapping train a transformation matrix to map the initial embedding(from pre-trained Glove) to task specific embedding space. madhyastha2016mapping train a transformation matrix to map the initial embedding(from pre-trained wordvec) to task specific embedding space. tafforeau2015adapting induce embeddings for unseen words by combining the embeddings
of the k nearest neighbors. santos2014learning,kim2016character introduce the char level representation and represent the token as a sequence of char. sennrich2016neural introduce the Byte Pair Encoding (BPE) to encode the word through composition of subword unit. Similar idea also used in chitnis2015variable,stratos2017sub. Since then, several variations and hybrid sub-word/char approaches have been proposed, such as pinter2017mimicking, chung2016character,yu2017joint,luong2015addressing,kobayashi2017neural,zhao2018addressing,kim2018learning,li2017acoustic. li2018subword,kim2018learning,ling2015finding address different composition functions for combining the char level information to get the word representation. 
bahdanau2017learning trained a RNN to compute the embedding for rare word on the fly, if a rare word encountered, they will retrieval its definition in dictionary and use the definition to compute the embedding.   
Several other work target achieving open vocab model in machine translation through hybrid approach   <cit.>

 <cit.>
 <cit.>

Multilingual setting:

OOV problems in multilingual settings is relatively under explored (perhaps due to most previous multilingual model use separate vocabulary and multilingual task focus more on multilingual perspective?). For pre-training a large scale multilingual deep model perspective, OOV problems is more serious. While there is no direct work deal with OOV in multilingual setting. However, several work show that multilingual information can help improving the word embedding for each language. lu2015deep showed adding multilingual context when learning embeddings via (deep) canonical correlation analysis (CCA) can improve word embedding for each language.
ruder2017survey gave a survey on cross-lingual word embedding models and found most bilingual information helps the monolingual word embedding.  
madhyastha2017learning expand the vocabulary through word embedding alignment in statistical machine translation (SMT) setting, i.e., for any source OOV, calculate a probabilistic list of target language and use this list as additional feature to improve the SMT's performance.  
cotterell2017cross train character-level taggers to predict morphological
taggings for high-resource languages and transfer to low-resource languages. Through character-level representation, it will alleviate the OOV issue, but it doesn't directly address the OOV problem.
More references here? I didn't found.




§ CONCLUSION


We investigated two methods (, joint mapping and mixture mapping) inspired by monolingual solutions to alleviate the OOV issue in multilingual settings. Experimental results on several benchmarks demonstrate the effectiveness of mixture mapping and the usefulness of bilingual information. To the best of our knowledge, this is the first work to address and discuss OOV issues at the subword level in multilingual settings. Future work includes: investigating other embedding alignment methods such as Gromov-Wasserstein alignment <cit.> upon more languages; investigating approaches to choose the subwords to be added dynamically.










§ ACKNOWLEDGMENTS

We thank the anonymous reviewers for their encouraging and helpful feedback.





acl_natbib


