Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer
Jianyu Zheng Fengfei Fan Jianquan Li







§ INTRODUCTION

Unsupervised cross-lingual transfer refers to the process of leveraging knowledge from one language, and applying it to another language without explicit supervision <cit.>. Due to the free requirement of the labeled data in target language, it is highly preferred for low-resource scenarios. Recently, unsupervised cross-lingual transfer has been widely applied in various natural language processing (NLP) tasks, such as part-of-speech (POS) tagging <cit.>, named entity recognition (NER) <cit.>, machine reading comprehension <cit.>, and question answering (QA) <cit.>.

The success of unsupervised cross-lingual transfer can be attributed to its ability to exploit connections across languages, which are reflected in various linguistic aspects such as lexicon, semantics, and syntactic structures. Consequently, many studies have sought to enhance models by encouraging them to learn these cross-lingual commonalities. For instance, in the lexical domain, <cit.> utilize bilingual dictionaries to randomly replace certain words with their translations in other languages, thereby encouraging models to implicitly align representations between the source language and multiple target languages.
In the area of syntax, several works have developed novel neural architectures to guide models in encoding the structural features of languages. <cit.>, for example, proposes a graph neural network (GNN) to encode the structural representation of input text and fine-tune the GNN along with the multilingual BERT (mBERT) for downstream tasks. Both lexical and syntactic approaches facilitate the alignment of linguistic elements across different languages, thereby enhancing the performance of cross-lingual transfer tasks.




However, language is a highly intricate system <cit.>, with elements at various levels being interconnected. For example, sentences are composed of phrases, which in turn are composed of words. In cross-lingual transfer, we hypothesize that merely guiding models to focus on a single linguistic aspect is inadequate. Instead, by simultaneously directing models to learn linguistic knowledge across diverse levels, their performance can be further improved.
Table <ref> presents some example sentences extracted from the XNLI dataset <cit.>. These parallel sentence pairs demonstrate that the multilingual model makes incorrect predictions for sentence pairs in the target languages (French and German) when only one aspect of linguistic knowledge, such as lexical or syntactic knowledge, is incorporated. However, when both types of knowledge are integrated into the model, the correct prediction is obtained. Despite this, most previous studies have focused on either syntactic or lexical information alone, without considering the integration of both types of information.

In this work, we aim to enhance unsupervised cross-lingual transfer by integrating knowledge from different linguistic levels. To achieve this, we propose a framework called "Lexicon-Syntax Enhanced Multilingual BERT" ("LS-mBERT"), based on a pre-trained multilingual BERT model. Specifically, we first preprocess the input source language sequences to obtain each word's part-of-speech information and dependency relationships between words in each sentence. Then, we replace some words in the sentence with their translations from other languages while preserving the established dependency relationships. Furthermore, we employ a graph attention network<cit.> to construct a syntactic module, the output of which is integrated into the attention heads of the multilingual BERT. This integration guides the entire model to focus on syntactic structural relationships. Finally, during the fine-tuning process, we simultaneously train the multilingual BERT and the syntactic module with the pre-processed text. As a result, our framework enables the multilingual BERT to not only implicitly learn knowledge related to lexical alignment but also encode knowledge about syntactic structure.

To validate the effectiveness of our framework, we conduct experiments on various tasks, including text classification, named entity recognition (ner), and semantic parsing. The experimental results show that our framework consistently outperforms all baseline models in zero-shot cross-lingual transfer across these tasks. For instance, our method achieves the improvement of 3.7 points for mTOP dataset. Our framework also demonstrates significant improvements in generalized cross-lingual transfer. Moreover, we examine the impact of important parameters, such as the replacement ratio of source words, and languages for replacement. To facilitate further research explorations, we release our code at <https://github.com/Tian14267/LS_mBert>.



§ RELATED WORK

Cross-lingual transfer is crucial in the field of natural language processing (NLP) as it enables models trained on one language to be applied to another. To enhance performance in transfer tasks, numerous studies focus on addressing the characteristics of various languages and their relationships.



 §.§ Incorporating Lexical Knowledge for Cross-lingual Transfer

A group of studies aims to incorporate lexical alignment knowledge into cross-lingual transfer research <cit.>. For example, <cit.> and <cit.> employ bilingual dictionaries to establish word alignments and subsequently train cross-lingual models by leveraging explicit lexical associations between languages. Other methods <cit.> involve substituting a portion of words in a sentence with their equivalents from different languages, a technique commonly known as "code-switching." By increasing the diversity of input text, these approaches promote implicit alignments of language representations. However, this group of studies mainly offers insights into lexical translation across languages, while neglecting the learning of language-specific structural rules.



 §.§ Incorporating Syntactic Knowledge for Cross-lingual Transfer

Another research category focuses on integrating syntactic knowledge for cross-lingual transfer <cit.>. Many studies in this group <cit.> develop graph neural networks to encode syntactic structures, a category to which our work also belongs. Taking inspiration from <cit.>, we adopt a similar architecture, specifically using a graph attention network to encode syntactic knowledge. Other methods <cit.> extract sparse syntactic features from text and subsequently incorporate them into the overall model. Although these approaches consider the relationships between language elements, they frequently overlook the alignments across languages, which impedes the effective transfer of linguistic elements and rules between languages.

Consequently, we combine the strengths of these two categories of approaches. First, we replace the input sequence with translated words from other languages, which aids in guiding the entire model to acquire implicit alignment information. Then, we introduce an additional module to assist the model in encoding syntax. 





§ METHODOLOGY

In this section, we provide a detailed introduction to our framework "LS-mBERT", as illustrated in Figure <ref>. Our objective is to enhance the cross-lingual transfer capabilities of multilingual BERT (mBERT) by incorporating both lexical and syntactic knowledge. Given an input sequence, we first pre-process it using a part-of-speech tagger and a universal parser(Section <ref>). This yields the part-of-speech tag for each word and dependency relationships among words in the sequence. To enable mBERT to implicitly encode word alignment information, we substitute some words with their translations from other languages using a code-switching technology (Section <ref>). Moreover, to guide mBERT in attending to syntactic relationships, we construct a graph attention network (GAT), introduced in Section <ref>. The output of the graph attention network is then used as input to the attention heads within BERT, effectively biasing attention information between words. Finally, to integrate both syntactic and lexical knowledge, we pass the code-switched text into both the GAT network and mBERT, which are trained simultaneously (Section <ref>).




 §.§ Pre-processing Input Sequence


The initial step involves pre-processing the input data to obtain prior knowledge for subsequent training. As our framework incorporates syntactic knowledge, we opt for an off-the-shelf parser with high accuracy to process the input text. In this case, we employ the UDPipe toolkit<cit.> to parse the inputs sentences, and Stanza<cit.> to annotate the part-of-speech information of each word. By utilizing both tools, given a sentence, we can obtain the dependency relationships between words and their part-of-speech information, which are then utilized to provide syntactic knowledge and enhance word representations, respectively.



 §.§ Code-switching for Text (lexical knowledge)


As our objective is to improve unsupervised cross-lingual transfer, introducing explicit alignment signals would be inappropriate. Therefore, we employ an implicit strategy to guide the entire model to encode word alignment information. Inspired by the work of <cit.>, we opt for the code-switching strategy. Specifically, we first randomly select a proportion α of words within each source sentence. Then, for each selected word, we use a high-quality bilingual dictionary to substitute it with a corresponding translation from another target language. This method not only promotes the implicit alignment of representations across diverse languages within our model, but also enhances the model's robustness when processing input text.



 §.§ Graph Attention Network (syntactic knowledge)


To guide mBERT in acquiring syntactic knowledge better, we construct an external syntactic module by referring to the method introduced by <cit.>. The overview of this module is displayed in Figure <ref>. Given that there are n tokens in the input sequence, we first represent each token by combining its embedding representation with part-of-speech (POS) information. The representation of the i-th token can be calculated: x_i=c_iW_c+pos_iW_pos,
where c_i and pos_i represent the token representation and the part-of-speech representation of the i-th token, respectively; while W_c and W_pos denote the token parameter matrix and the part-of-speech parameter matrix.
Then, the encoded sequence s' = [x_1, x_2, ⋯ , x_n] is passed into the subsequent syntactic module, which is designed with a graph attention network (GAT) <cit.>. The GAT module comprises a total of L layers, each with m attention heads. These attention heads play a crucial role in generating representations for individual tokens by attending to neighboring tokens in the graph. Each attention in GAT operates as follows: O = Attention(T, T, V, M), wherein T denotes the query and key matrices, and V represents the value matrix.  Besides, M signifies the mask matrix, determining whether a pair of words in the dependency tree can attend each other. Notably, the relationships between words in the attention matrix are modeled based on the distances between words in the dependency tree, rather than the positional information within the word sequence. Subsequently, the resulting representations produced by all attention heads are concatenated to form the output representations for each token. Finally, the output sequence from the final layer can be denoted as Y = [y_1, y_2, ⋯, y_n], where y_i represents the output representation for the i-th token.
To maintain the lightweight nature of the architecture, certain elements in GAT have been excluded. Specifically, we do not employ feed-forward sub-layers, residual connections, or positional representations. We found that these modifications do not result in a significant performance gap.




 §.§ Summary of the Framework: Lexicon-syntax Enhanced Multilingual BERT


In this subsection, we provide an overview of our "LS-mBERT" framework, as illustrated in Figure <ref>. We first select multilingual BERT (mBERT) as the base model. Then, we process the input sequence using the code-switching strategy in Section <ref>, resulting in the code-switched sequence s'. It is important to note that despite some words in each sentence being replaced with other languages, the original dependency relationships between words are still preserved in s'. Next, we feed the code-switched text into both mBERT and the syntactic module (GAT), facilitating the fusion of the two types of knowledge. Furthermore, this step guides the entire model to better align different languages within the high-dimensional vector space during training. After GAT processes the code-switched sequence, the output from the final layer is utilized to bias the attention heads of mBERT. The calculation process can be described as follows: O=Attention(Q+YW_l^Q,K+YW_l^K,V), where Q, K, and V represent the query, key, and value matrices, respectively; While W_l^Q and W_l^K are new parameters to learn for biasing the query and key matrices. 







§ EXPERIMENTS
 


 §.§ Experimental Settings

As above mentioned, we use UDPipe <cit.> and Stanza <cit.> for parsing sentences and obtaining words' part-of-speech information in all languages, and employ MUSE <cit.> as the bilingual dictionary for word substitution. For all tasks, we identify the optimal parameter combinations by searching within the candidate sets. The learning rate is set to 2e-5, utilizing AdamW as the optimizer. The batch size is 64, and the maximum length for input sequences is 128 tokens. For code-switching, we vary the replacement ratio (α) from 0.3 to 0.7 with a step of 0.1. For the GAT network, we adopt the identical parameter values as employed in the work of <cit.>. Specifically, we set L to 4 and k to 4.



 §.§ Tasks


Our framework is evaluated on the following tasks, using English as the source language. Some statistics are summarized in Table <ref>, along with the detailed descriptions provided below.

Text Classification. Text Classification is a task that assigns predefined categories to open-ended text. In our experiment, we utilize two publicly available dataset: XNLI and PAWS-X. In XNLI <cit.>, models need to predict whether a given pair of sentences is entailed, contradicted, or neutral; In PAWS-X <cit.>, models are required to determine whether two given sentences or phrases convey the same meaning. When implementing the two tasks, to establish connections between the dependency trees of the two sentences, we introduce two edges from the [CLS] token to the root nodes. Subsequently, we apply the code-switching technique to randomly replace certain words in the sentence pairs.


Named Entity Recognition. Named Entity Recognition (NER) is a task that involves the automatic identification and categorization of named entities. In our experiment, we employ the Wikiann <cit.> dataset. Wikiann consists of Wikipedia articles annotated with person, location, organization, and other tags in the IOB2 format. Our method is evaluated across 15 languages. To ensure that the models can obtain complete entity information, we exclusively substitute words that do not constitute named entities during the code-switching process.


Task-oriented Semantic Parsing. In this task, the models are required to determine the intent of the utterance and then fill the relevant slots. The dataset for the experiment is mTOP <cit.>, which is an almost parallel corpus, containing 100k examples in total across 6 languages. Our experiments cover 5 languages.








 §.§ Baselines

We choose the following methods as baselines to compare:


  * mBERT. We exclusively utilize the multilingual BERT model to perform zero-shot cross-lingual transfer for these tasks. 

  * mBERT+Syn. A graph attention network (GAT) is integrated with multilingual BERT, and these two components are jointly trained for all tasks. 

  * mBERT+Code-switch. The multilingual BERT model is fine-tuned with the code-switched text across various languages. 












§ RESULTS AND ANALYSIS



 §.§ Cross-Lingual Transfer Results

The main experimental results are displayed in Table <ref>. Our method consistently demonstrates superior performance across all tasks compared to other baselines. This indicates our method's effectiveness for cross-lingual transfer, achieved through the incorporation of lexical and syntactic knowledge. Especially for the tasks Wikiann and mTOP, our method exhibits a significant improvement, with an increase of 2.2 and 3.7 points, respectively, when compared to the baseline with the best performance.  In addition, since code-switching technique blends words from various language, we calculate the results across the languages excluding English, as shown in the column "AVG/en" in Table <ref>. We find that the performance gap between our method and each baseline in most tasks becomes wider. This also indicates that our method can more effectively align non-English languages within the same vector space implicitly.  

For each task, we discover most of languages can gain improvement by using our method, as compared to the top-performing baseline. Specifically, 84.6% (11/13), 100.0% (7/7), 80.0% (12/15) and 100.0% (5/5) languages demonstrate improvement in XNLI, PAWS-X, Wikiann and mTOP respectively. Furthermore, our method also provides improvement for non-alphabetic languages in many tasks, such as Chinese, Japan and Korean. This reflects that our method can be effectively generalized into various target languages, even in cases where significant differences exist between the source and target languages.








 §.§ Generalized Cross-Lingual Transfer Results

In practical scenarios, cross-lingual transfer could involve any language pair. For example, in a cross-lingual question-answering (QA) task, the context passage may be in German, while the multilingual model is required to answer the question in French. Considering on this, we conduct zero-shot cross-lingual transfer experiments within a generalized setting. Since PAWS-X and mTOP are completely parallel, we evaluate the performance of our method and "mBERT" baseline on generalized cross-lingual transfer tasks using the two dataset. The experimental results are illustrated in Figure <ref>.

For both classification and semantic parsing benchmarks, we have observed improvements among most language pairs. This reflects that our method is very effective for generalized cross-lingual transfer. Furthermore, when English is included in the language pair, there is a substantial enhancement in performance. Specifically, when English serves as the source language, the average performance of target languages is increased over 10% and 3% in mTOP and PAWS-X dataset, respectively. This reflects the effectiveness of the code-switching in aligning other languages with English. For the PAWS-X dataset, we find that some non-Indo-European languages such as Japanese, Korean, and Chinese can achieve improvements, even when the source languages belong to the Indo-European language family, including English, Spanish, French, and German. It reflects that syntactic knowledge can effectively narrow the gap of language structures for this task, especially for the language pairs without close linguistic relationships.







§ ANALYSIS AND DISCUSSION



 §.§ Impact on Languages

We investigate whether our method can improve the performance of specific languages or language groups. As shown in Figure <ref>, we display the performance improvement of our method by comparing the "mBERT" baseline. We find that almost languages can obtain benefits from our method. Particularly, when the target language, such as German, Spanish and French, belongs to the Indo-European language family, the improvement is very significant. Furthermore, the performance in the mTOP task is improved significantly by our method among all languages. This may be because that our method consider both syntax and lexicon simultaneously, which is beneficial for the semantic parsing task.




 §.§ Representation Similarities across Languages

To evaluate the effectiveness of our method in aligning different languages, we employ the representation similarity between languages as the metric. Specifically, we utilize the testing set of XNLI <cit.> as the dataset, which consists of parallel sentences across multiple languages. Then we take the vector of [CLS] token from the final layer of our model, as well as the vectors from two baselines ("mBERT+Syn" and "mBERT+code-switch) for each sentence. Following <cit.>, the centroid vector for representing each language is calculated by averaging these sentence representations. Finally, we adopt cosine similarity as the indicator to assess the degree of alignment between English and each target language.

Figure <ref> illustrates the similarities between languages by using our method and the other two baselines. It can be easily found that our method outperforms the other two baselines in aligning language representations. This suggests that infusing two types of knowledge is indeed effective in reducing the disparities in language typologies, which improve cross-lingual transfer performance. In addition, we observe that "mBERT+code-switch" performs better than "mBERT+Syn", which reflects that lexical knowledge is more useful than syntactic knowledge for this task.





 §.§ Impact of Code-switching

The replacement ratio α for code-switching is an important hyper-parameter in our method. Hence, we explore its impact on mTOP and PAWS-X, by varying α from 0 to 0.9 in increments of 0.1, shown in Figure  <ref>. When α is set to 0, it represents the results of the baseline "mBERT+Syn". As α increases, more source words are substituted with their equivalent words from other languages. The performance improvement certificates the effectiveness of code-switching technique. Notably, when about half of the words are replaced (0.5 for PAWS-X and 0.4 for mTOP), the performance reaches their peaks. After that, both tasks experience a decline in performance. This decline might be because the expression of meaning and sentence structure are influenced severely as too many words are replaced. Therefore, it is a optimal choice to set α between 0.4 to 0.5 for code-switching.



Furthermore, we investigate whether the choice of the replacement language in code-switching impacts our model's performance. We select mTOP and PAWS-X as the testing tasks. In code-switching, we devise three different measures for language replacement: "Exclusively replacing with the target language", "Replacing with languages from the same language family as the target language"; and "Replacing with languages selected randomly". The experimental results are illustrated in Figure  <ref>. We can easily observe that "Exclusively replacing with the target language" performs best, while "Replacing with randomly selected languages" yields the poorest results. Hence, this also underscores the importance of selecting languages closely related to each target language for substitution when employing the code-switching technique.








 §.§ Performance with XLM-R

To validate the universality of our method, we substitute multilingual BERT with XLM-R in our framework. XLM-R is a more robust multilingual pre-trained model known for its  exceptional cross-lingual transfer capabilities. Subsequently, we test its performance on the PAWX-S dataset, and the experimental results are displayed in Table  <ref>.

In Table  <ref>, we also observe that our framework outperforms the other three baselines. This indicates that integrating lexical and syntactic knowledge is beneficial for enhancing performance, irrespective of the base model employed. Notably, our framework only achieves the slight performance improvement when utilizing XLM-R as the base model compared to employing multilingual BERT. It may be because that the base model, XLM-R, adopt larger corpus during pre-training, resulting in preserving richer language information. Consequently, XLM-R itself has possessed superior cross-lingual transfer capabilities. The assistance by incorporating external linguistic knowledge appears to be relatively minor in comparison.




 §.§ Limitations and Challenges

In our study, we adopt a bilingual dictionary, such as MUSE <cit.>, to substitute words in other languages. However, we randomly choose a target language word when there exist multiple translations for a source language word. This approach, although convenient, neglect the context of the source language word, potentially leading to inaccurate translations. This also highlights us to explore more precise word alignment methods in the future.

Furthermore, the tasks we have evaluated are quite limited, with some of them involving only a few languages. In the future, we will extend our method to more cross-lingual tasks. Meanwhile, we also develop dataset for these tasks to support more languages.



§ CONCLUSION

In this paper, we present a framework called "lexicon-syntax enhanced multilingual BERT" ("LS-mBERT"), which infuses lexical and syntactic knowledge to enhance cross-lingual transfer performance. Our method employs code-switching technology to generate input text mixed in various languages, enabling the entire model to capture lexical alignment information during training. Besides, a syntactic module consisting of a graph attention network (GAT) is introduced to guide mBERT in encoding language structures. The experimental results demonstrate that our proposed method outperforms all the baselines across different tasks, which certificates the effectiveness of integrating both types of knowledge into mBERT for improving cross-lingual transfer. In the future, we plan to incorporate different linguistic knowledge into large language models (LLMs) to further enhance cross-lingual transfer performance.




§ ACKNOWLEDGEMENTS

The authors would like to thank the anonymous reviewers for their feedback and suggestions. Additionally, this work was supported by the Major Program of the National Social Science Fund of China (18ZDA238), the National Social Science Fund of China (No.21CYY032), Beihang University Sponsored Projects for Core Young Researchers in the Disciplines of Social Sciences and Humanities(KG16183801) and Tianjin Postgraduate Scientific Research Innovation Program (No.2022BKY024).      





*


§ BIBLIOGRAPHICAL REFERENCES


lrec-coling2024-natbib








