Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model
Runzhe Zhan Xinyi Yang Derek F. Wong Lidia S. Chao Yue Zhang

Estimating Metocean Environments Associated with Extreme Structural Response
    [
    April 29, 2024
============================================================================

^*Work was done during a visit to Westlake University.
^ Co-corresponding authors.


While supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language model (LLM) to specific preferences, concerns have been raised about the depth of this alignment, with some critiques suggesting it is merely “superficial”.
We critically examine this hypothesis within the scope of cross-lingual generation tasks, proposing that the effectiveness of SFT may be constrained by its reliance on prior tokens to guide cross-lingual generation.
Based on this crucial insight, and in response to the challenges posed by the costly and limited availability of non-English data for SFT, we introduce a novel training-free alignment method named , which employs minimal task-related prior tokens to bridge the foundation LLM and the SFT LLM, achieving comparable performance without training. Experiments on machine translation and part-of-speech tagging across eight languages demonstrate the efficacy of  in cross-lingual settings. Remarkably, by initiating the decoding process with only one or two prior tokens, foundation LLMs can achieve performance comparable to their SFT counterparts. This method presents a cost-effective alternative to SFT and advances the democratization of multilingual LLMs.

 


§ INTRODUCTION

Supervised fine-tuning (SFT) refines large language models (LLMs) using task-specific instruction data to enhance their capability to follow instructions <cit.> and to align their outputs with human preferences and safety considerations <cit.>.
This process is often termed “alignment”, signifying the tailoring of model outputs to conform to specific downstream requirements. Nevertheless, current research casts doubt on the necessity and potential adverse impacts of SFT. 
But the alignment achieved through SFT is often considered to be “superficial”, with the process potentially repurposing pre-existing knowledge from pre-training to merely reshape outputs to meet specific criteria <cit.>. 
It has been observed that even a small-scale SFT training dataset can produce significant alignment effects <cit.>.
On the other hand, recent empirical studies <cit.> have raised concerns that SFT might hurt the knowledge acquired during its pre-training phase, leading to serious consequences like catastrophic forgetting. 

Not only is there no definitive consensus on the necessity of SFT, but the majority of these studies also focus on monolingual tasks. LLMs still encounter challenges in handling complex cross-lingual generation tasks <cit.>. 
Current research on cross-lingual alignment primarily seeks to extrapolate or align English capabilities to other languages using the SFT paradigm <cit.>, yet there remains a gap in exploring the specific impacts of SFT-based cross-lingual alignment. Furthermore, given the potential risk of SFT leading to the forgetting of pre-training knowledge, the question of how to achieve cross-lingual alignment without training remains underexplored.



To bridge these gaps, our study conducts an in-depth examination of the impact of SFT on cross-lingual generation. We investigate the influence of SFT on the decoding patterns of foundation models in cross-lingual contexts, hypothesizing that the success of SFT largely hinges on the selection of initial prior tokens that are critical for eliciting task-specific generation in the target language.
Furthermore, the observed decoding similarities between foundation and SFT models support the extension of the superficial alignment hypothesis to cross-lingual scenarios.
Responding to these insights, we introduce a training-free alignment method named “” for cross-lingual and non-English tasks. The Prefix TexTs act as a Yarn () linking the foundation LLM and the SFT LLM, eliciting the foundation LLM to exhibit near-SFT performance levels. 
Specifically, we augment the original input with a few tokens that serve as decoding priors, and then prompt the foundation LLM to resume decoding based on this modified input.
In most cases, only one or two task-related prior tokens are needed, and the method for constructing these prior tokens is flexible across various kinds of language resources, fostering the democratization of multilingual LLMs.


We conducted experiments on machine translation <cit.>, cross-lingual summarization <cit.> and non-English part-of-speech (POS) tagging <cit.> tasks across eight languages.
These tasks exemplify cross-lingual generation and multilingual language understanding, and they provide ample non-English test data to evaluate effectiveness across varying levels of resource availability.
The experimental results demonstrate that  can effectively align the foundation model to match SFT model's performance without training, by merely adding two prior tokens in the decoding.


§ ICEBERG MODEL OF SFT




 §.§ Preliminaries



  
Pre-training 
The pre-training (PT) of LLMs is primarily conducted through language modeling tasks on large-scale unlabeled data <cit.>. 
During this phase, given a sequence X_PT of length N and a context window k, the optimization objective is maximizing the joint probability P_LM as:

    P_LM(X_PT)=∏_i=1^N P(x_i|x_i-k:i-1)

which encourages the model to generate text that naturally follows from the preceding context. However, this “text completion” behavior can become a bottleneck when models are prompted to switch languages or follow specific instructions of cross-lingual generation. 
It is frequently observed that when prompted with English input and instructed to produce text in a different language, as illustrated in the upper example of <ref>, the foundation model often continues to decode in English. 




  
SFT 
SFT leverages labeled data pair (X_ins., Y) to empower models with the ability to follow instructions. This stage aims to maximize the probability of the expected answer Y conditioned on the input text X_ins., where X_ins. consists of the task instruction and task input.

    P_SFT(Y|X_ins.)= 
     ∏_j=1^T P(y_j|y_1:j-1,X_ins.)

SFT is crucial for aligning foundation models to perform task-specific instructions, effectively transforming a general-purpose LLM into an instruction-following assistant.
However, data quality, training costs, and the imbalance of multilingual data hinder the democratization of assistant LLM. As mentioned before, SFT may be harmful to pre-training knowledge. Thus, it is meaningful and important to understand the underlying mechanism of SFT-based alignment and propose a more efficient alignment method.



 §.§ Beneath the SFT-based Alignment



  
Prior Knowledge Hypothesis
It is worth noting that pre-training corpora also contain sequences that naturally express task-specific information, which imparts certain capabilities to the foundation LLMs. 
For example, the presence of semantically equivalent expressions in the pre-training text may enable LLM acquire machine translation ability during pre-training stage <cit.>.

Despite its extensive prior knowledge, the foundation LLM still struggles with complex cross-lingual generation tasks. Beyond existing studies, we provide more concrete insights into this issue by prompting foundation LLMs with various instructions <cit.>. Notably, only 31.8% of these prompts successfully elicit translation capability from the foundation LLMs[For detailed information, please refer to <ref>.].

This deficiency may stem from two main factors: First, the proportion of text with the aforementioned characteristics in the pre-training corpus X_PT is still relatively small, and most of it is far from resembling human instruction text X_ins..  
Consequently, the model is more likely to predict tokens suitable for completing formal texts than those required for task-specific instructions. As a result, the foundation LLM often fails to produce tokens y ∈ Y_1:T in the intended target language.
Secondly, the predominance of English in the pre-training data skews the token generation probabilities of foundation LLM. Given a cross-lingual context, the model favors predicting tokens in English, while the token probabilities for other languages remain comparatively low. For example, English data comprises up to 90% of the Llama2 pre-training data <cit.>, which may lead models to generate text with an English-centric bias. 

The above hypothesis might be reasonable when we revisit <ref> and <ref>. The probability P_LM(X_PT) of the next token prediction for the foundation model is conditioned on the distribution of the pre-training text X_PT. 
SFT narrows the probability space for token selection, adjusting the parameters to better align with the distribution, i.e., the probability P_SFT(y|X_ins.) is conditioned on the distribution of the instruction text X_ins..



  
Experimental Settings
To validate the aforementioned hypothesis, we selected the representative cross-lingual task of machine translation as our analytical testbed. The main research method involved quantifying the differences and similarities in the decision space and token selection behavior between the foundation LLM and the SFT-aligned LLM. 
For the model selection, we chose the foundation Llama2 7B model and conducted supervised fine-tuning on it using the Alpaca dataset[<https://github.com/tatsu-lab/stanford_alpaca>]<cit.>. 
The optimization was carried out using a cosine learning rate scheduler, with the maximum learning rate set to 2e-5 and a warmup ratio of 0.03. 
Training was performed on two Nvidia-H800 GPUs using LoRA parameter-efficient fine-tuning <cit.> technique, with a cumulative batch size of 64. Other hyper-parameters follow those of the original Alpaca settings.








  
A Prior Token Elicits Silent Majority
Inspired by the categorization of token shifts by <cit.>, we propose to quantify the agreement of token selection between foundation LLM θ_PT and SFT LLM θ_SFT. 
Given the same prefix input X̂, we aim to measure whether the next token selected by the SFT LLM, y_SFT, is among the top-K tokens, 𝐲_PT, with the highest probabilities in the decision space of the foundation LLM, which can be formally expressed as follows:


    y_SFT = _y ∈ V P(y|X̂;θ_SFT)
    𝐲_PT = { y | arg topK_y ∈ V P(y|X̂;θ_PT) }
    Aggrement_K = 1/L∑_l=1^L1_y_SFT∈𝐲_PT

where V is the vocabulary shared by two models, and L is the length of the dataset.

We compare the agreement of the token selection made by the models under the same prefix text X̂ in two different experimental setups. The first setup uses the instruction text as the prefix, i.e., X̂ = X_ins.; the second takes the first token decoded by the SFT model as a prior token, appending it to the original instruction prefix, i.e., X̂ = [X_ins., y_SFT^(1)]. 
For the SFT model, the second setup is equivalent to continuing its own decoding behavior, whereas for the foundation model, it becomes decoding with the addition of a prior token.

<ref> illustrates the agreement between the foundation model's predictions and those of the SFT model regarding the selection of the next token, given an identical text prefix. Across the entire translation data, it is observed that after incorporating merely one prior token, the foundation model exhibits a high degree of agreement with the SFT model in terms of token selection. This demonstrates that the alignment effect of SFT in cross-lingual generation tasks is also somewhat superficial.
Even in instances where the token with the highest probability differs between the two models, 90.8% of the tokens chosen by the SFT model are present within the “silent majority” in the decision space of the foundation model, specifically, among the top 20 most probable token choices.



  
Lens of Distribution
Instead of focusing on the coverage of token selection outcomes, we also observe the decision dynamics and similarities from the perspective of the overall probability distribution, with the data settings consistent with the previous setup.
First, as shown in <ref>, after adding a prior token, the probability of the next tokens chosen by both models have closely aligned distributions. 
The reason that the foundation model exhibits a high probability given the instruction text as a prefix lies in a preference for choosing to continue the instruction text rather than completing the cross-linguistic semantic transformation.
Additionally, we quantify the distribution disparities between the two models through the probability distribution of the vocabulary.
The disparity metrics used include Kullback-Leibler (KL) divergence, Jensen-Shannon (JS) divergence, and cross-entropy <cit.>. As depicted in <ref>, the disparity of decision space of the foundation model significantly decreases after adding the prior token, aligning more closely with the SFT model.

These findings indicate that such prior tokens serve a dual function: they not only steer the foundation model towards generating tokens pertinent to cross-lingual generation but also modulate the decision space to align more closely with the task-specific distribution.




§ PRETTY: PREFIX TEXT AS A YARN



 §.§ Motivation

The observations discussed earlier confirm that SFT effectively narrows the decision space of the foundation model during text generation that is conditioned on instruction text. 
The disparity in token selection between the foundation LLM and the SFT LLM, however, might not be reduced by a training-based transfer methodology. 
By appending a prior token into the instruction text, the choices of the next token between the two models tend to become largely consistent, and in the vast majority of cases, the tokens chosen by SFT model are also found within the high-probability candidate words of foundation model. 
These phenomena show that the alignment elicited by SFT is somewhat superficial in cross-lingual generation tasks and motivate us to propose a training-free alignment method by leveraging these prior tokens.



 §.§ Formulation

Upon revisiting <ref> and <ref>, the goal of proposing a training-free approach is to enable the conditional decoding probability of foundation model to approximate those of SFT model. 
Therefore, ideally, the selected prior tokens X_pri. = {x_pri.} may satisfy the following criteria:

    P(𝐲_PT|[X_ins., X_pri.];θ_PT) 
    ≈ P(𝐲_SFT|X_ins.;θ_SFT)

where 𝐲_PT and 𝐲_SFT represent the outputs of the foundation and the SFT models, respectively. It is important to note that a single prior token may not serve as an optimal solution due to its non-derivable characteristic. Hence, we extend our methodological approach to include appending multiple prior tokens, grouping them to form a prefix text.



 §.§ Construction of Prior Tokens

To ensure that the proposed method is applicable to a wide array of languages, we propose three construction strategies based on the availability of language resources, aiming to guarantee the universality of our approach.



  
SFT Prior represents an ideal scenario where the first few tokens generated by a SFT model are used as priors. This method is theoretically rational when the SFT model is derived from the same foundation model because it directly approximates <ref> by sampling x_pri.∼{y_SFT}. In practical applications, this might be suitable for high-resource languages due to the imbalanced language capabilities of other languages. 
Additionally, SFT could potentially degrade the knowledge and abilities that the foundation model has already acquired. In such cases, using prior tokens from the SFT model can contribute to generating better results. This situation will be discussed further in the subsequent section.



  
Refined Prior is more readily accessible for most languages and tasks. We can utilize the output tokens generated by a smaller model trained for specific downstream tasks and use them as prior tokens to achieve weak-to-strong generalization <cit.>.



  
Pseudo Prior
For extremely low-resource language pairs, where there is no labeled data for downstream tasks, both SFT and Refined priors are difficult to obtain. 
For cross-lingual tasks, we can create pseudo labels in target language as prior tokens. 
For instance, in machine translation tasks, we might use bilingual dictionaries to acquire pseudo prior tokens. 
However, the quality and accuracy of pseudo labels remain uncertain, and the extent of their impact on the generative performance of the foundation LLM is not yet clear. We will explore this problem further in the context of experimental results discussed later in the paper.



§ EXPERIMENTS

We examine the effectiveness of our proposed training-free alignment method on two distinct tasks: machine translation, cross-lingual summarization and non-English POS tagging. 
Machine translation serves as a prototypical cross-lingual generation task, entailing the transformation of a sequence from a source language to a target language <cit.>. 
As for cross-lingual summarization, it requires the model to generate a summary of an article in a different language <cit.>. 
Although POS tagging <cit.> primarily assesses the model's ability to understand monolingual text, we include it as multilingual experiments to show the universality of our methods.







 §.§ Experimental Settings
 


  
Data
We use Flores-101 <cit.>, CrossSum <cit.> as benchmarks for machine translation and cross-lingual summarization tasks, respectively.
For POS tagging tasks, we choose the POS test split from the XGLUE benchmark <cit.>, which is derived from the Universal Dependencies Treebank v2.5.
To investigate the performance across various resource languages, we carefully selected eight languages based on the pre-training data proportions disclosed in the Llama2 technical report <cit.>. 
These languages are French, German, Chinese, Russian, Ukrainian, Portuguese, Hindi and Arabic. Among these, the first four languages account for more than 0.1% of the pre-training data of Llama2, while Ukrainian and Portuguese fall below 0.1%, Hindi and Arabic is below 0.05%. For the Llama2 model, we can categorize these three types of languages into high-resource languages, low-resource languages, and extremely low-resource languages, respectively.



  
Models and Baselines 
The settings of Llama2 foundation model and the SFT model are consistent with those described in <ref>.
To further demonstrate the generality of our proposed method, we incorporated the Mistral-7B LLM family <cit.> into our experiments, covering both out-of-the-box SFT and foundation models.

In the machine translation task, the Llama2 foundation model does not tend to generate translations when given explicit translation instructions. While this is a normal phenomenon according to our previous discussion, to ensure a fair comparison, we also searched for a better prompts for the foundation model. This prompting approach is referred to as “Llama2-7B_Prompting” in subsequent sections. 
For POS tagging, we experimented with various instructions and selected one that consistently prompts both the foundation model and the SFT model to reliably generate classification results in text.
Although we report the zero-shot performance for the aforementioned tasks, we found that even out-of-the-box SFT models cannot produce stable output for cross-lingual summarization task. Hence, we prepend a constant demonstration before the input to also assess the effectiveness of our proposed method under the in-context learning paradigm <cit.>.



  
Sources of Prior Token
The sources of crafting prior tokens include:

    
  * SFT Prior: We took the first k tokens of output produced by SFT model as the prior tokens. For multiple SFT models, we select the model that demonstrates better performance. 
    
  * Refined Prior: We use downstream task models with smaller parameter sizes as the source of refined priors. For the different tasks, we utilized the distilled 600M variant of NLLB-200 translation model[<https://huggingface.co/facebook/nllb-200-distilled-600M>]<cit.>, mT5 cross-lingual summarization model[<https://hf.co/csebuetnlp/mT5_m2m_crossSum>] and the Unicoder-NLU model[<https://github.com/microsoft/Unicoder/>]<cit.>, respectively. 
    
  * Pseudo Prior: The pseudo prior is applied to two cross-lingual tasks since it can utilize cross-lingual language resources. 
    We create pseudo prior tokens for machine translation task by referencing dictionary [Please refer to <ref> for dictionary information.] entries.
    For cross-lingual summarization, we initially extract keywords from each passage using KeyBERT <cit.> and then perform word-by-word translation.
    However, not all initial sentence tokens will be covered by the dictionary. To handle such instances, a back-off strategy is implemented, where the target language equivalent of the first available dictionary token is used as the prior token.


For two cross-lingual task, the first k=2 tokens are chosen as the prior tokens. This helps to avoid inadequate guidance from single non-informative tokens like punctuation or numbers. In the case of the pseudo prior, due to the back-off strategy, only one token is used for fair comparison.
For POS tagging task, the strategy is more straightforward with only the first k=1 label considered as the prior token.













 §.§ Evaluation

To ensure the integrity of the output data from all models, we standardized the output by cleaning it in accordance with the specific output style of each model. Subsequently, we conducted a manual inspection to guarantee that only the required labels were retained.



  
Task-specific Metrics 
We use two metrics to evaluate the performance of translation quality: spBLEU[<https://github.com/mjpost/sacrebleu/>] <cit.> and COMET[<https://github.com/Unbabel/COMET>]<cit.>.
We employed the ROUGE <cit.> and LaSE <cit.> metrics for the evaluation of summarization quality. 

For the POS tagging task, we report both the precision score and F_1 score.



  
Relative Performance
We further compute the ratio of the performance scores of the foundation model to the scores of the SFT model with the application of different strategies.
This ratio serves as a metric for assessing the extent to which the foundation model approximates the SFT model's performance when different strategies are applied.



 §.§ Main Results



  
Machine Translation
As shown in <ref>, for the machine translation task, we use up to two prior tokens as decoding guidance, allowing the base model to achieve performance comparable to that of a model after SFT. 
Moreover, in some language pairs, the translation performance outperforms SFT model when guided by Refined Prior tokens from a smaller model.
For Llama2 model family, the prior tokens provided by the SFT model, although slightly less effective, still allow the foundation model to achieve 98% of the performance of SFT model. On the other hand, the use of pseudo labels derived from a dictionary exhibits the least effectiveness, yet this strategy still surpasses the results achieved through costly prompt engineering.



  
Cross-lingual Summarization
The results presented in Table <ref> indicate that the foundation model exhibited superior performance compared to the SFT model in this in-context learning scenario. For prior-guided decoding, the performance of the foundation model was degraded when using prefix tokens from the SFT model, and the small performance gap in this setting suggests that the alignment achieved by the SFT model is relatively “superficial”. Notably, the performance of Llama2 foundation model significantly improved when other priors were provided, even when using translated keywords as pseudo labels.



  
Non-English POS tagging
The performance results of POS tagging task are presented in <ref>. These results align with the insights gleaned from the machine translation task, specifically regarding the strategy of prior token construction. 
Notably, for POS tagging task, the performance of SFT model on most language pairs falls short of the foundation model, suggesting that SFT detrimentally affect the knowledge learned at the pre-training stage.
Encouragingly, when the foundation model empowered by auxiliary prior token surpasses the performance of SFT model as well as the prompting results of itself, highlighting the potential of our proposed method in mitigating the catastrophic forgetting problem associated with SFT. 




§ ANALYSIS AND DISCUSSION



 §.§ Quality of Prior Tokens

To investigate the quality of prior tokens from different sources and how they impact the final performance, we further analyze why the prior tokens given by the SFT model are less effective than those from external auxiliary models in POS tagging task. 
Unlike the machine translation task, the positional result for the POS task is definite, so we are able to verify whether it corresponds to a ground truth label. 
The results in <ref> confirm two points. First, even if the prior tokens provided by the SFT model are of low quality, the foundation model does not suffer from severe error propagation. Secondly, the final performance of proposed method is still associated with the quality of prior tokens. This suggests that prior tokens closely aligned with the ground truth can steer the foundation model towards a more accurate decision trajectory, thereby yielding superior performance.





 §.§ Choice of Prior Tokens

Based on the findings from the previous section, if incorrect labels used as prior tokens can still elicit the ability of foundation model, then could random prior tokens in the target language trigger cross-lingual generative capabilities? 
To investigate this, we attempted to use random tokens of different parts of speech as the prior tokens in the English-Chinese machine translation task.
For instance, “Modal Prior” refers to the use of randomly picked modal verb in Chinese as the initial token.
 
The results shown in <ref> indicate that the model could not be aligned to a better decision trajectory by these random prior tokens, whether they were function words or tokens with actual meaning. This supports the validity of our proposed methods for constructing prior tokens and also supplements previous findings. From this, we can summarize some rules about prior tokens: they can be of low quality but should not be completely unrelated to the target sequence.





 §.§ Number of Prior Tokens

Figure <ref> depicts the relationship between the number of preceding tokens provided and the resulting changes in translation performance. It becomes apparent that performance generally improves with the addition of more tokens. 
Additionally, we note that introducing two prior tokens appears to be a performance inflection point, which may be due to instances where the initial token is a punctuation mark or a number.





§ CONCLUSIONS

In this paper, we investigate and analyze the decision-making discrepancies between the foundation model and the SFT model within cross-lingual generation contexts. Drawing from our analysis, we introduce a novel cross-lingual alignment method that requires no additional training and is resource-efficient. The proposed method aligns the foundation LLM to perform comparably with the SFT model solely by utilizing prefix text as priors during generation. In the future, we aim to broaden our research to encompass additional alignment scenarios, such as those involving reinforcement learning from human feedback.



§ LIMITATIONS

The primary limitations of our study stem from the scope of model validation. Our research is limited to 7B models. Future endeavors should aim to extend the validation to a broader scope of models and incorporate various parameter scales to support the universality of our findings.
Furthermore, the availability of language resources is still a practical problem, particularly for low-resource languages where access to Prior Token and Refined Token sources is limited. Despite these challenges, our experimental results indicate that Pseudo Prior tokens still exhibits promising potential. 
It is important to note, however, that the development of pseudo tags may require a dedicated investigation into the linguistic rules specific to each downstream task. This process is inherently time-intensive and resource-demanding.



§ ACKNOWLEDGEMENTS


This work was supported in part by the Science and Technology Development Fund, Macau SAR (Grant Nos. FDCT/0070/2022/AMJ, FDCT/060/2022/AFJ), Ministry of Science and Technology of China (Grant No. 2022YFE0204900), National Natural Science Foundation of China (Grant No. 62261160648), the Multi-year Research Grant from the University of Macau (Grant No. MYRG-GRG2023-00006-FST-UMDF), and Tencent AI Lab Rhino-Bird Gift Fund (Grant No. EF2023-00151-FST). This work was performed in part at SICC which is supported by SKL-IOTSC, and HPCC supported by ICTO of the University of Macau.










§ LATEST BENCHMARK

The composition of the pre-training corpus for Llama2 is indeed unclear as Meta AI has not disclosed specifics in their technical report <cit.>. Consequently, we cannot confirm the inclusion of Flores-101 datasets in the pre-training phase.
To mitigate this uncertainty, we conducted additional experiments using the WMT23 machine translation benchmark. Since Llama2 was released on July 18, 2023, and the WMT23 News Shared Task submission[<https://www2.statmt.org/wmt23/translation-task.html>] deadline was July 20, 2023, with the reference translations released on October 10, 2023, it is highly unlikely that WMT23 data influenced Llama2's pre-training. 

For these experiments, we used conventional BLEU as our evaluation metric because the Flores-101 benchmark introduces a specialized BLEU score variant known as spBLEU, which diverges from the conventional BLEU by adopting the sentencepiece model <cit.> for tokenization. 
As shown in <ref>, the experimental results indicate a consistent trend and support similar conclusions to those obtained from previous benchmarks.



§ DETAILED SETUP



 §.§ Llama2 Alpaca

To avoid the need of fine-tuning for Llama2 family, we initially considered using Llama2-Chat. However, it is important to note that Llama2-Chat is a version that has been fine-tuned using both SFT and Reinforcement Learning with Human Feedback (RLHF), as it has been aligned with human preferences for helpfulness and safety. In practice, we observed that the Llama2-Chat model sometimes overly prioritizes safety, leading to instances where it refrains from generating responses due to its strict safety protocols. Consequently, we opted for a SFT model that was fine-tuned using Alpaca dataset to ensure more consistent outputs and used it as a source of obtaining SFT priors.


  
Hyper-parameters 

The hyper-parameter settings are presented in Table <ref>. This experiment was conducted using the DeepZero 2.









 §.§ Prompting Strategies
 


  
Machine Translation

    
  * Normal: Translate the following sentences from {Source Language} to {Target Language}. {Source Text}
    
  * Prompting: {Source   Language}: {Source Text} \n{Target Language} Translation:



  
Cross-lingual Summary

    
  * Please write a concise summary of the text, return your responses with one line that cover the key points of the text.\n Input: {Source Text} \n Output: {Target Language} Summary:



  
POS tagging

    
  * Please provide the POS tags for each word in the input sentence. The input will be a list of words from the sentence. The output format should be a JSON array, where each key-value pair includes a word from the input list and its corresponding POS tag from the set of labels: [{PoS Tags}]\nNote: Your response should include only a JSON array, presented in the order that the words appear in the input   sentence. Each key-value pair should contain the word and its POS label.





 §.§ Success Rate of Vanilla Prompting
 
We randomly selected 500 sentences from the Flores-101 benchmark for four translation directions and randomly used ten diverse prompts for the corresponding translations. In addition to prompts proposed by <cit.>, the used prompts are as follows:

    
  * Translate the following sentences from {Source Language} to {Target Language}.
    
  * Can you help me translate this sentence from {Source Language} to {Target Language}?
    
  * What is the translation of sentence in {Target Language}? {Source Text}
    
  * Infill in {Target Language}: {Source Text}
    
  * Please provide machine translation text for me.\n{Source Language}: {Source Text}\n {Target Language}:

To determine whether a translation was generated (regardless of its translation quality), we employed automated tools[<https://github.com/Mimino666/langdetect>] and then checked the results by human. We then calculated the success rate of translation requests. The results, which will be detailed in table below, indicate that foundation LLMs exhibit considerable instability in generating the target language when presented with different prompts. In most cases, the LLMs failed to produce the desired language output.





 §.§ Pseudo Labels

We collected lexical data from various sources, including Wiktionary[<https://www.wiktionary.org/>], MUSE[<https://github.com/facebookresearch/MUSE#ground-truth-bilingual-dictionaries>]<cit.>, and PanLex[<https://panlex.org/source-list/>], and merge their entries into one dictionary. <ref> presents the detailed information of compiled dictionaries.


