Language Imbalance Driven Rewarding for Multilingual Self-improving
Large Language Model;Self-Improving;Multilinguality
We would like to emphasize that the focus of our method is multilingual self-rewarding that self-improves the general multilingual capability without external resources. In `Lines 53-55`, we explicitly mention that our work is primarily inspired by Self-rewarding (Yuan et al., ICML 2024). Table r1 provides a detailed comparison, highlighting the potential similarities and differences between our work, Self-rewarding, and MAPO (She et al., ACL 2024). 


Table r1: The relationship between our approach, Self-rewarding, and MAPO.
|                     | Ours                                                             | Self-rewarding              | MAPO                                                 |
|---------------------|------------------------------------------------------------------|-----------------------------|------------------------------------------------------|
| Task                | **Instruction following (mainly)** and _math reasoning (additionally)_ | **Instruction following**       | _Math reasoning_                                       |
| Training method     | **Iterative DPO**                                              | **Iterative DPO**               | **Iterative DPO** or PPO                               |
| Reward signal (core contribution)       | Intrinsic language imbalance (prior)                                    | LLM-as-a-Judge (self-judge) | Semantic consistency (evaluated by external MT model)         |
| Preference data form     | **Preference pairs**                                                 | **Preference pairs**            | Preference score ('alignment' score) for each sample |
| External model/data | **None**                                                             | **None**                        | Off-the-shelf MT model                               |
| Multilingual    | _Yes_                                                              | No                          | _Yes_                                                  |

As shown in `Table r1`, our approach bears more resemblance to Self-rewarding (highlighted in **bold**), including the task we focused on, the training method, preference data form, and the self-improving paradigm. Therefore, we mainly contextualized our approach with LLM self-improving methods in related works. In contrast, the only similarity between our work and MAPO is that we all experimented on multilingual mathematical reasoning task (the _italicized_ text). 

Regarding the reviewer’s comment on the similarity between our approach and MAPO, particularly concerning the use of MT for building preference data and iterative DPO for optimization, we believe there are key differences that set our work apart. 

1. **MT utilization**: Our work only leverages the **LLM itself** to translate responses across languages, while MAPO employs an **external MT model** as a reward model.
2. **Preference pair construction**: 

    a) In our work, preference pairs are constructed based on **language imbalance**—specifically, the prior that dominant languages generally outperform non-dominant ones. This contrasts with MAPO, which does not directly construct preference pairs. Instead, MAPO first employs an MT model to assess the **consistency between reasoning paths** generated in different languages, where the MT model functions as a reward model. The reward score derived from this consistency assessment is then used to construct preference pairs. 

    b) Our approach could construct preference pairs for **both dominate and non-dominate languages**, while MAPO can only derive reward signal for **non-dominate ones**.
4. **Iterative DPO training**: As shown in `Table r1`, we draw inspiration from the Self-rewarding method, which constructs preference pairs without external supervision and **only adopts iterative DPO training**. MAPO, on the other hand, **employs both PPO and DPO training** in their experiment, as they can obtain a reward score ('alignment' score) for each response using the external MT model.

In conclusion, we believe the similarity in `Table r1` and the distinctions listed above make it clear that **our approach is directly inspired by Self-rewarding instead of MAPO**. 

We did not include MAPO as a major related work, since it is designed for only reasoning tasks rather than general ones such as instruction following tasks. This is because off-the-shelf translators fail to assess consistency between LLM responses due to their limited context size. Besides, it does not completely obey the self-improving paradigm, due to the use of the external translator. 

To clear up the misunderstandings, we will include a detailed discussion in the revised version on the relationship between our approach, Self-rewarding, and MAPO, emphasizing their major differences.
 
We hope the evidence provided above could address your concerns. We would greatly appreciate it if you could kindly reconsider our work with these distinctions in mind.
Clarification on Ethics Concerns
Yes
As I mentioned in the weaknesses, the paper present a method that use translation to build multilingual preferences, and use DPO to optimize the model, which is quite similar to the one proposed in She et al., ACL 2024. The authors obviously knew the referenced paper, but provides no explanation in the relation between the two. I don't think this is inappropriate. In case I was personally biased, I raise the problem to the ethics committee for further judgement.
Yes, Research integrity issues (e.g., plagiarism, dual submission)
I am not sure why languages such as es and fr only have a reward accuracy of 60%. It seems that they are well represented in the training data. It would be helpful if further explanation or analysis are provided.
The method is effective in improving the performance of general instruction following and mathematical reasoning.

The paper also presents analysis for the effectiveness and accuracy of the preference pairs, which serves as a nice support for the method.
This paper presents a method for bridging the language abilities for dominant languages and non-dominant languages. The method use the translation between the generation results in the dominant language and non-dominant language to build preference pairs, and conduct an iterative DPO optimization. The experiment shows improvement over the original model.
The paper is not the first attempt to improve the multilingual ability of LLMs by cross-lingual optimizations. In one of the cited paper, She et al., ACL 2024, where experiments are conducted in optimizing the preference with DPO and an off-the-shell translator.

I find the proposed method quite similar to the above one, but in this paper there is no clear indication of the potential relations.
Yes
No ethics review needed.
1. Table 2 row 2 "Self Translation" is confusing me: which language is translated into English (8.03)?
2. The structure of section 5 looks weird. Why would arithmetic reasoning be a separate section/experiment?
3. Line 143, Equation (7), the length normalization term $|y_{win}|$ is inside $log$. Is this correct?
- I think the idea of exploiting the gap in an LLM's inherent language capability for self-improvement is interesting and intuitive.

- The paper has extensive experiments across open-ended and close-ended benchmarks. Results are consistent and in favour of the proposed method. It is clear that through iterative data synthesis and training, models can progressively improve in both dominant languages (`dl`) and non-dominant languages (`nl`).
This work proposes to create a preference dataset using a multilingual LLM by exploiting the quality difference in responses in dominant languages (`dl`) and non-dominant languages (`nl`). The paper makes two assumptions that for the same instruction, 1) an `nl` response after being translated to `dl` is still worse than the original `dl` response, and similarly, 2) a `dl` response after being translated to `nl` is still better than the original `nl` response. The authors then provided empirical evidence for these assumptions. Iterative DPO training was done using the synthesized data and the trained models were tested on multilingual AlpacaEval, multilingual MT-bench, and some structured multilingual tasks. Results are shown to be in favour of the proposed method that models can iteratively improve by using data synthesized from itself.
1. Several factors in asserting the assumptions could not be carefully controlled. Table 1 line 180: GPT4-as-a-judge is used to confirm the quality of responses in different languages, however, there is no guarantee that the scores for different languages are on the same scale and that GPT4 is able to judge those languages using the same "standard". This could apply to Table 2 and Table 3 too.

2. I find the motivation of creating (`nl`, `dl->nl`) preference data reasonable. However, I did not find creating the response pair (`dl`, `nl->dl`) intuitive. Since the model already could generate `dl` well (higher quality), why would having a lower-quality sample benefit alignment, especially as your Finding 2 that "the dominant language also benefits [...] a phenomenon that has rarely been observed"? Do you have any explanations for this?

3. I think that all multilingual benchmarks used in this work, both open-ended and structure tests, are machine-translated from English to other languages. Since the preference data is also constructed using self-translation e.g. (`nl`, `dl->nl`), it might be possible that some kind of "translationese bias" is exploited as opposed to actually improve multilingual capability.
Yes
No ethics review needed.
1. It would be better to use more advanced LLMs like GPT-4o as the judge due to its stonger multilingual capabilities. 
2. Does the approach for weaker models like Llama-2-7B and lower-resource langauges? If it works, this method would be more impactful.
1. This work suggests a promising direction for self-improving multilingual LLMs by leveraging intrinsic language imbalance.
2. Experimental results demonstrate significant improvements over baseline models on multilingual benchmarks.
3. The method is clearly explained, which is easy to understand.
This paper leverage the langauge imbalance of LLMs to bootstrap the multilingual capabilities of LLM in a self-improving manner. The proposed approach involves training through Iterative Direct Preference Optimization (DPO), which leverages preference data generated by the model itself. By translating prompts and responses between dominant and non-dominant languages, the model assigns preferences, with responses in dominant languages generally preferred. The system iteratively fine-tunes the model on these preferences, improving multilingual performance without human-annotated data. Experiments on the Meta-Llama-3-8B-Instruct model show substantial performance improvements across both instruction-following and arithmetic reasoning tasks, with enhancements across all evaluated languages.
1. The evaluation heavily relies on GPT-4 to assess response quality, but its reliability as a multilingual judge is uncertain, which was discuessed in [1]. The evaluation in Table 1 across various languages may be biased because it's unclear if GPT-4 rates identical responses equally in different languages. A fairer method would be to evaluate responses in the same language, including both directly generated and translated ones, to ensure scores are more comparable. 
2. The method depends on LLMs to self-translate responses, but the quality may be inferior to established systems like Google Translate and NLLB, particularly for low-resource languages, as discuessed in [2]. To enhance the analysis, it would be beneficial to assess the method using low-resource languages such as Swahili, Thai, and Bengali, as well as comparing it with more translation systems like Google Translate.

[1] Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?
[2] Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models.
This paper proposes Language Imbalance Driven Rewarding, which leverages the inherent imbalance in LLMs as a reward signal to bootstrap LLMs’ multilingual capabilities in a self-improving manner.
@inproceedings{
anonymous2024language,
title={Language Imbalance Driven Rewarding for Multilingual Self-improving},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Kak2ZH5Itp},
note={under review}
}
Large Language Models (LLMs) have achieved state-of-the-art performance across numerous tasks. However, these advancements have predominantly benefited "first-class" languages such as English and Chinese, leaving many other languages underrepresented. This imbalance, while limiting broader applications, generates a natural preference ranking between languages, offering an opportunity to bootstrap the multilingual capabilities of LLM in a self-improving manner. Thus, we propose $\textit{Language Imbalance Driven Rewarding}$, where the inherent imbalance between dominant and non-dominant languages within LLMs is leveraged as a reward signal. Iterative DPO training demonstrates that this approach not only enhances LLM performance in non-dominant languages but also improves the dominant language's capacity, thereby yielding an iterative reward signal. Fine-tuning Meta-Llama-3-8B-Instruct over two iterations of this approach results in continuous improvements in multilingual performance across instruction-following and arithmetic reasoning tasks, evidenced by an average improvement of 7.46\% win rate on the X-AlpacaEval leaderboard and 13.9\% accuracy on the MGSM benchmark. This work serves as an initial exploration, paving the way for multilingual self-improvement of LLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Model Self-Improving Multilinguality
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ef955ff2d424569a56952bf2dc7015c0afa359e4.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Language Imbalance Driven Rewarding for Multilingual Self-improving
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission