No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement
Mateusz Klimaszewski Piotr Andruszkiewicz Alexandra Birch























A,B]Mateusz KlimaszewskiCorresponding Author. Email: mateusz.klimaszewski.dokt@pw.edu.pl
A]Piotr Andruszkiewicz
B]Alexandra Birch 

[A]Institute of Computer Science, Warsaw University of Technology
[B]School of Informatics, University of
Edinburgh




Modular deep learning is the state-of-the-art solution for lifting the curse of multilinguality, preventing the impact of negative interference and enabling cross-lingual performance in Multilingual Pre-trained Language Models. However, a trade-off of this approach is the reduction in positive transfer learning from closely related languages. In response, we introduce a novel method called language arithmetic, which enables training-free post-processing to address this limitation. Inspired by the task arithmetic framework, we apply learning via addition to the language adapters,  transitioning the framework from a multi-task to a multilingual setup. The effectiveness of the proposed solution is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual schemes, acting as a post-processing procedure. Language arithmetic consistently improves the baselines with significant gains in the most challenging cases of zero-shot and low-resource applications. Our code and models are available at <https://github.com/mklimasz/language-arithmetic>.








§ INTRODUCTION


The recent progress of large language models has led to a huge interest in how well they can perform not just in English but across multiple languages and invoked the rise of Multilingual Pre-trained Language Models (MLLMs) <cit.>.
These models serve as general-purpose solutions that can be adapted and applied to various Natural Language Processing tasks. Notably, MLLMs demonstrate zero-shot cross-lingual capabilities, allowing them to generalise effectively to downstream tasks even when trained in a language different from the target language.

The positive transfer of abilities from both related languages and high-quality training data from unrelated languages has meant that MLLMs have reported state-of-the-art performance in low-resourced languages <cit.>.
However, this benefit does not always extend to high-resourced languages <cit.>. In such cases, the quality of MLLMs tends to decrease compared to their monolingual counterparts <cit.> due to negative interference phenomena <cit.>. Additionally, the curse of multilinguality <cit.> reveals the existence of a trade-off between language coverage and model capacity. Consequently, MLLMs must carefully limit the number of languages included during the pre-training phase.

Modular deep learning (MDL) <cit.> methods come to the rescue to avoid the abovementioned limitations, enabling the extension of MLLMs to support any number of languages
<cit.>.
MDL methods adapt the model to arbitrary tasks and languages by isolating components from each other (and the backbone MLLM) via parameter-efficient extensions.
Examples of parameter-efficient modules are LoRA <cit.> and adapters <cit.>, which are low-budget (in terms of parameters) bottleneck layers that increase an MLLM size by just a fraction. Language adapters especially allow the modularisation of language-specific knowledge by training on a raw, unlabelled corpus in a target language.

The limitation of the MDL and language adapters is their isolation. While they lift the curse of multinguality and prevent negative interference, at the same time, language adapters limit the possible impact of positive transfer. Previous attempts to address these challenges — such as training bilingual <cit.> or language-family <cit.> adapters — do not scale effectively. In our work, we tackle this limitation as a post-processing step. Leveraging recent insights from task arithmetic <cit.>, specifically learning via addition, we augment language adapters with missing related language knowledge – a concept we term language arithmetic. Remarkably, this training-free approach enhances language adapters across various scenarios, from zero-shot settings to low-resource regimes.

To summarise, our contributions are as follows:

    
  * A novel training-free post-processing method named language arithmetic that enhances language adapters.
    
  * We conduct a cross-lingual evaluation on three downstream tasks and Multilingual Pre-trained Language Models with test cases that include zero-shot and low-resource setups.
    
  * We provide an analysis of language arithmetic internal components.






§ BACKGROUND

Our research builds upon the contributions of <cit.> and <cit.>. The following Section provides the background and serves as a gentle introduction to the concept of task vectors and task arithmetic.



 §.§ Task Vectors & Task Arithmetic

Let us assume that we have access to a pre-trained model with its weights denoted θ_pre∈ R^d  and a fine-tuned version of the same model on a task t represented by θ^t_ft∈ R^d. The task vector τ_t∈ R^d is an element-wise difference between models' weights.

    τ_t = θ^t_ft - θ_pre

The task vectors can be part of multiple arithmetic operations, e.g. learning via addition. This operation is an addition operation between two task vectors and the base model, i.e. we add two differences between the fine-tuned models and the pre-trained version with weights controlling the impact.

    θ_multi-task = θ_pre + λ_1 τ_t_1 + λ_2 τ_t_2

The lambdas can be further normalised to sum to one, i.e. λ_2 = 1 - λ_1 and simplifying notation with just λ.

    θ_multi-task = θ_pre + λτ_t_1 + (1 - λ) τ_t_2

While we define learning via addition for two tasks, the same procedure can be applied to multiple tasks.

Task arithmetic allows us to forge a multi-task model from a separate, task-specific set of fine-tuned models, preserving high accuracy (although a shared pre-trained starting point is required, e.g. the same Language Model). Moreover, vectors from different tasks are typically close to orthogonal, and <cit.> speculate that this enables the combination of task vectors via addition with minimal interference.

In the Parameter-Efficient Fine-tuning (PEFT) regime, we can reduce a task vector to only newly introduced PEFT parameters <cit.>. Another way of looking at this is a task vector that is a zero-valued vector for all the parameters apart from PEFT weights, i.e. a task vector would not modify the base, pre-trained checkpoint.



§ METHOD


We propose language arithmetic that transitions the task arithmetic concept from a multi-task to a multilingual setup. In this Section, we describe the language arithmetic alongside its application as a training-free, post-processing step to a MAD-X cross-lingual framework <cit.>.



 §.§ Language Arithmetic

We formulate a language arithmetic (LA) concept by substituting the task in task vectors and arithmetic with a language. This approach means that instead of merging downstream tasks, we target a problem of cross-lingual performance. We propose to apply learning via addition to languages, and in Section <ref>, we demonstrate the discrepancies when comparing language and task vectors. Our study focuses specifically on the language adapters <cit.>. Due to overlapping abbreviations, we use the LA exclusively as the former, i.e. language arithmetic. In the learning via addition, we limit the parameters to language adapters and simplify the notation that θ represents the adapters' weights and τ is referred to as a language vector. As we operate in a language space instead of a task, the t is replaced with a language, i.e. its language code in the notation. The example equation describes a language arithmetic operation between an English and a Spanish adapter.

    θ_LA = θ_pre + λτ_en + (1 - λ) τ_es

Throughout the paper, the equation above is abbreviated as a function: LA(en, es) with lambda as a default parameter. Whenever we use language arithmetic in a zero-shot context, we add a subscript to the function name, i.e. LA_0s(en, es) means that language arithmetic between English and Spanish is applied to a different language, e.g. French.

Language arithmetic is a training-free method, taking advantage of already pre-trained modules. The sole requirement is a validation dataset on which the λ parameter can be established. While in our work, we use a pretty fine-grained step (0.05) to determine the λ value (i.e. we run evaluations for λ∈ [0, 1] with a provided step), our analysis showcased that it is possible to increase the value and limit the computation burden even more (details in Section <ref>).



 §.§ Application

We apply the language arithmetic as an enhancement method for language adapters. We assume two scenarios: (i) zero-shot case, where we do not have a language adapter in the desired language and (ii) “regular” enhancement case, where we improve the existing language adapter (in high- and low-resourced languages).

We evaluate our post-processing method as an extension of the MAD-X framework <cit.> to challenge our solution in a cross-lingual manner. The overview of the schema is presented in Figure <ref>.



The MAD-X consists of the following:

    
  * Choosing MLLM
    
  * Training language adapter 1
    
  * Training task adapter 1
    
  * Training more language adapters 2


In our work, we introduce an additional step:
enumi4
    
  * Post-processing via Language Arithmetic 3




  §.§.§ MLLM

The first step usually assumes the availability of a Multilingual Pre-trained Language Model. In our work, we focus on mBERT[] <cit.> and XLM-R[] <cit.>.



  §.§.§ Training language & task adapters

The following two steps depend on the downstream dataset. In the most extreme case, we might have a dataset in one specific language and need to take advantage of cross-lingual performance. Therefore, step number 2 trains a language adapter on raw corpora corresponding to the task language using regular masked language modelling loss in a self-supervised manner. During this step, the MLLM is frozen. The third step freezes both the backbone and language adapter and trains a task adapter on a downstream task dataset.
Given a set of tasks or if a new task appears, we can repeat this step as long as a required language adapter exists, i.e. a language adapter that matches the task's language.



  §.§.§ Additional language adapters

The last, fourth step is the heart of the MAD-X framework. We can now train language adapters in any language, leverage the pre-trained task adapter(s) and obtain a cross-lingual performance by connecting a new language adapter with a task adapter (i.e. routing first via language adapter and then task) to obtain language-task pair unseen during training. The growing pool of pre-trained adapters can be accessed at public repositories like AdapterHub <cit.> and reused for further use cases.



  §.§.§ Post-processing via Language Arithmetic

First, we assume the use-case where the pool of language adapters does not contain the target languages, i.e. zero-shot scenario. In this study, we add English as a task-specific language to a language related to the target. In the second language adapter enhancement scenario, we apply language arithmetic in its default form, where the goal is to enhance the existing target language adapter (via either  English or a related language). At last, we investigate (simulated and actual) low-resource cases.




§ EXPERIMENTS




 §.§ Experimental setup

We use the AdapterHub library <cit.> for all our experiments. For language adapters, we train on the Wikipedia corpora[20331101.xx checkpoint <https://huggingface.co/datasets/wikimedia/wikipedia>] for 250k steps with a learning rate of 1e-4, an effective batch size of 64 using a single GPU[Nvidia A100/RTX3090/V100] and the same initialisation. For task-specific training, we train for 100 epochs with the same learning rate and a batch size set to 16. We choose the final checkpoint based on validation dataset performance  (for language adapters, we evaluate on a held-out subset of Wikipedia).

Downstream evaluation is performed on three tasks: Named Entity Recognition (NER), Natural Language Inference (NLI) and Question Answering (QA), covering jointly 13 languages[ar, bg, de, el, es, fr, hi, ru, sw, tr, ur, vi, zh; XQuAD does not cover 4 languages (bg, fr, sw, ur)], while the training - to perform cross-lingual evaluation - is performed on the English data. For the NER task, we use the WikiANN <cit.> dataset and for NLI - XNLI <cit.>. The QA evaluation is done on XQuAD <cit.> (we split data 50/50 into valid/test datasets), and the training uses SQuAD 1.1 <cit.>. Additionally, to evaluate a low-resource scenario for a language not covered during MLLM pre-training, we leverage the Assamese subset from IndicXNLI <cit.>.



 §.§ Zero-shot evaluation




The zero-shot evaluation assumes a scenario where the language adapter pool does not contain the desired target language. In such a case, one can use an English adapter as a proxy (i.e. 1 in Figure <ref>). An alternative is the usage of a related language (2). Language arithmetic serves a solution that, instead of choosing, combines the adapter's tuple: LA_0s(en, REL), where REL symbolises a related language (we provide the related language list in Appendix <ref>).

Table <ref> presents the results of the zero-shot experiment. Language arithmetic consistently outperforms the proxy baselines, reaching up to an improvement of over 3 F1 points in the XLM-R NER setup and almost 1.5 F1 for QA. Additionally, we looked at the λ parameter derived on a validation dataset.

While most cases set the value to over 0.5 (i.e. preferring the English side, given LA_0s(en, REL)), the preferred values did not showcase any consistency and pattern.
We analyse that phenomenon in Section <ref>.




 §.§ Language Arithmetic evaluation


This evaluation assumes that the target language adapter exists in the adapter pool. We test two cases, i.e. LA(T, en) and LA(T, REL), where REL is once again a related language and T is the target language. The results are presented in Table <ref>. Compared to the baseline direct application of a target language adapter (i.e. MAD-X method), the gains are not as significant as in the case of the zero-shot scenario.
Given that a target language adapter is trained on a significant corpus, it gives less room for improvement (this is not the case in the low-resource regimes, as shown in the following Section).
However, considering the cost-to-performance ratio and the minimal fatigue that our post-processing method enforces on existing MAD-X pipelines, we can see a constant but minor gain.



Notably, during the evaluation of XLM-R NLI, we observed that the outcomes remain consistent across different methods (e.g. 67.8 for Swahili). This behaviour is a built-in property of the proposed method. The method has a fail-safe mechanism, which keeps the extreme values during the evaluation of λ∈ [0, 1]. Suppose the method does not find a better lambda different than the extreme values (i.e. λ∈{0, 1}). In that case, it means that we are using a proxy language directly, and the language arithmetic did not provide an improvement (but kept the final result the same).





 §.§ Low resource evaluation




Training a language adapter might be troublesome for high-resourced languages due to massive corpora requiring significant computational resources.[Although in our experimental setup, we train each adapter for the same number of steps and choose the best checkpoint based on the validation performance, for low-resourced languages, one could apply an early stopping mechanism in a production-level pipeline.] On the other hand, in most languages, we lack data to train a strong language adapter, i.e. language-specific corpora might be either too small or unavailable  <cit.>. We investigate whether language arithmetic can help in such cases. We test our solution in three cases and define three (actual and simulated) evaluation scenarios:

    
  * Assamese (as) - low resource language, additionally not used in the pre-training of mBERT and XLM-R,
    
  * Swahili (sw) - low resource language, used in the pre-training,
    
  * French (fr) - high resource, used in the pre-training. We simulate cases from low to high resources.

We train a series of language adapters with different token budgets for each language, from 10k to 10M (or 1B for French; we limit this particular study to the XLM-R model). Afterwards, we compare the usage of such adapters directly against language arithmetic with three adapters (we use LA(T,en), where T ∈{as,sw,fr}).


Figure <ref> presents the results of the evaluation performed on the downstream tasks. The most gain is visible in the most challenging scenario, during the evaluation on the Assamese dataset. In this case, the backbone MLLMs did not encounter the language during the pre-training phase. Although the difference becomes less pronounced in the NER task as we approach the limits of available data, there remains a significant margin for NLI - the difference can be explained by the overlap in the corpora (Wikipedia) between NER and language adapter training tasks, following the findings of <cit.>. For Swahili, where the language is part of the pre-training, the flattening effect begins earlier and affects both tasks. Nevertheless, leveraging language arithmetic still yields improvements.

The simulated case of French showcases that even with a relatively weak language adapter (trained on 10k tokens), the language arithmetic can restore existing knowledge and results in high performance for the language. Moreover, comparing the adapters trained with a different token budget, the results remain similar, without significant fluctuations.  We believe that this phenomenon happens because the Multilingual Pre-trained Language Model has seen a much higher amount of French in the pre-training procedure than Swahili (over 35 times more tokens in XLM-R pre-training; moreover, French is in the top 15 represented languages). Therefore, even undertrained French adapters have a relatively easy task once they are merged with a robust English adapter. In practical terms, this finding allows us to prototype new languages quicker by estimating the possible end product quality or might serve as an intermediate solution (until the full-corpora adapter is trained).



§ ANALYSIS




 §.§ Lambda impact


Our study estimates the λ parameter with a small step (0.05). Depending on multiple variables that include, among others, model and evaluation dataset sizes or number of languages, running 20 evaluations might be troublesome (especially when using neural-based metrics, e.g. COMET <cit.>). Therefore, we analysed the potential impact of choosing a suboptimal lambda with a decrease in evaluation count. The breakdown includes a subset of languages on both tasks (using the XLM-R version). We chose the zero-shot scenario where we performed language arithmetic between English and related language adapters.

In Figure <ref>, we plot the validation scores with the corresponding baselines, that is, the scores of using directly the adapters. The dotted lines are based on λ=0 or λ=1 for clarity, meaning we exclusively use the arithmetic equation's left or right side (i.e., a specific language). In most cases, a subset of valid λ values would improve over the baselines. Moreover, the analysis reveals that a coarser evaluation (e.g., with a step of 0.1 or 0.2) would be sufficient, reducing the required number of performed tests up to four times while maintaining most of the improvement. At last, setting the default λ=0.5 would be near optimum for the analysed subset.





 §.§ Language Relatedness


Relatedness of languages is a difficult-to-define concept. At times, in our proposed framework, we might face a choice of multiple, seemingly equally related languages to use for the arithmetic operation. In this analysis, we decided to look at this aspect via a glance at Romance languages. We trained an additional subset of language adapters and formed a pool of 6 languages: Catalan, French, Italian, Portuguese, Romanian and Spanish. Afterwards, we evaluated languages shared in our NER and NLI tasks (Spanish and French) by arithmetic with the entire Romance languages pool. 

The results are presented in Table <ref> and show that given a related language (in this case, defined as coming from the same language family), there might be very little difference in which one we will use. The relative difference between the best and the worst language reaches only a 0.40 F1 score in the NER task and the same value in terms of accuracy points in NLI. Nevertheless, the more sophisticated and hand-crafted language choice would improve the results from Table <ref>.





 §.§ Language vs task vectors



Task vectors exhibit high sparsity and orthogonality, as <cit.> observed. While the former characteristic can be denoted in language vectors (Figure <ref>), the latter displays different properties, in contrast to task vectors. In Figure <ref>, we visualise the cosine similarity between evaluated language vectors of language adapters. Notably, the minimal cosine similarity (0.2) surpasses the maximum (0.18) reported by previous research in the task space <cit.>. Interestingly, most pairs in the task space oscillate within the range of 0.01 to 0.03. At the same time, language vectors surpass 0.2 in each case, indicating that the orthogonality aspect is an inherent property of task adapters.




Based on the cosine similarity observation, we investigated one of the recent task arithmetic extensions, Ties-Merging <cit.>. This work introduces a three-step algorithm that prevents different parameter interferences, improving upon task arithmetic. The algorithm decreases the cosine similarity via a pruning step and alignment of parameter signs to perform arithmetic only on relevant parameters to the merged tasks. On the experimental details note, as Ties-Merging operates on averaging, not addition, we utilise a different lambda range during validation (as suggested by <cit.>,  λ∈ [0.8, 1.8]), and we set Top-K% to the default value of 20 and additionally to 80.



We report the comparison in the zero-shot setting on the NER task (XLM-R version) in Table <ref>. The Ties-Merging decreases the results significantly; it is not only outperformed by language arithmetic in every language pair but also by other baselines (see Table <ref>). Moreover, we note that the pruning operation has the reverse effect; higher pruning (i.e. keeping Top-K% lower)  decreases the performance  (in contrast to task vectors) by making language vectors more sparse and, hence, closer to orthogonal.




One interpretation of the phenomena can be the different goals of the arithmetic: in the multi-task setup, we try to include multiple, often disconnected, tasks into a single task vector. In contrast, the language vectors' goal is to include the knowledge of the closely related language rather than remove the harmful artefacts. Our findings indicate that language arithmetic has different characteristics than task arithmetic, and the follow-up works that improve upon task arithmetic might not be suited for the multilingual context.



§ RELATED WORK

Knowledge composition from multiple, independently trained adapters has been widely discussed in the literature. However, unlike our work, the solutions require substantial changes to the vanilla adapter setup. The previous work either requires additional parameters to learn a parameterised composition function/a gating module to combine/steer the flow through the suitable adapter(s), or needs a specific training procedure that increases the complexity of the overall solution or, in most cases, both <cit.>. Moreover, to prevent specifically negative interference, hyper-adapters <cit.> were proposed using hyper-networks <cit.>, and <cit.> applied sparse fine-tuning to compose task and language masks. Unlike the prior studies mentioned earlier, our attempt is training-free and does not modify the base architecture. The most conceptually similar work is proposed by <cit.>; however, they operate on the notion of sample similarity to a subset of domains in a domain adaptation regime. Additionally, concurrent to our work, <cit.> show intial potential of task arithemtic in cross-lingual transfer based on a full fine-tuning setup. At last, we denote the rise of task arithmetic use cases, e.g. vision tasks or cross-task generalisation <cit.>.



§ CONCLUSION


We have proposed language arithmetic, which enhances language adapters based on task arithmetic learning via addition. It is a training-free method and functions as a post-processing technique for the adapters. Our experiments have shown that LA is particularly beneficial in zero-shot and low-resourced cases. At last, we highlight the differences between language and task arithmetic.

In our future work, we plan to extend language arithmetic by incorporating additional components into the sum. Additionally, we aim to adapt other elements of the task arithmetic framework, initially designed for tasks, i.e. task analogies and forgetting via negation, to a multilingual setup with an analysis of the differences between multi-task and multilingual arithmetic context. Furthermore, we will evaluate LA's performance on various non-classification tasks. 



§ LIMITATIONS

Our work was tested on English-centric task training and could be extended to different languages with more PEFT methods. Moreover, applying multi-source training based on the work of <cit.> could provide better robustness of the task adapters and a more thorough analysis.
At last, our analysis of the differences between task and language vectors is preliminary, without definitive conclusions.









§ LAMBDA IMPACT - QA



Figure <ref> presents the analysis of lambda impact for the Question Answering task. For details, refer to Section <ref>.





§ RELATED LANGUAGES CHOICE



We derived related languages for the language adapters based on the available pool, i.e. we reused the adapters from other languages from the task (we trained two bonus ones, Persian and Italian, apart from the analysis in Section <ref>). While the related language could have been more suitable in a few cases, there is always a trade-off between relatedness, criterion for relatedness and available corpora. Moreover, the perfectly related language matching would be an unfair comparison as it would assume the availability of the ideal adapter's pool, which is unlikely in practice. Table <ref> presents the related language choice used in our experiments.



