Enhancing Multilingual Reasoning in LLMs: Insights from Cross-Linguistic Correlations and Optimal Data Proportions
Large Language Models;Multilingual Reasoning;Fine-Tuning
Thank you for your insightful comments, which have greatly helped us improve our work. We have polished our paper according to your suggestions. Here are our responses to your concerns:

1. Regarding W1 - Why is language alignment only studied within mathematical reasoning?
Our primary objective was to enhance LLMs' mathematical reasoning capabilities across multiple languages. We first investigated the relationship between multilingual alignment and mathematical reasoning. After achieving success in mathematical reasoning, we applied this approach (using specific multilingual training data proportions) to other reasoning scenarios and found similarly positive results. We acknowledge that our original explanation may have caused confusion and have revised it in the new version.

2. Regarding W2 - Criteria for selecting 10 or 25 languages:
a) Selection of 10 languages: These languages are commonly used in related works ( [1][2][3] and so on. ) to evaluate LLMs' multilingual mathematical reasoning capabilities. This selection enables both representation and comparison with existing research.

b) Extension to 25 languages: To further validate our method's effectiveness, we expanded beyond the conventional 10 languages to include 25 languages, covering both commonly used and low-resource languages. The selection referenced historical WMT datasets for better representation. We have added this explanation to the paper for clarity.

3. Regarding W3 - Basis for language grouping:
Our grouping followed these principles:
- English forms one group due to its dominance in training data and strongest reasoning capabilities
- German, French, and Russian form another group based on their demonstrated positive correlation with English alignment and mathematical reasoning abilities
- The remaining languages form the third group
We have clarified this explanation in the revised paper.

4. Regarding W4:
Our approach enhances reasoning capabilities across all languages (including low-resource ones) by increasing training data for high-resource languages (de, fr, ru) in fine-tuning datasets. We chose high-resource languages for alignment study because of their abundant, readily available data, which helps investigate how to generalize English reasoning capabilities to other languages. We have revised our explanation to make this clearer.

5. Regarding W5:
Given the scarcity and high translation costs of low-resource language data, coupled with LLMs' relatively poor reasoning performance in low-resource contexts, efficient utilization of existing low-resource data is crucial. We have added relevant citations and clarified this explanation in the paper.

6. Regarding W6 - Generalization to other benchmarks:
Your concern is valid. We addressed this by:
- Using both MGSM and MSVAMP datasets for evaluation (Figure 9)
- Extending our method to Python code reasoning
- These are the primary benchmarks in the field, also used by related works ([1][2][3] and so on ).
- We will open-source our code, datasets, and models for community feedback

Regarding Q1 - Missing references:
Thank you for pointing this out.  We have added the relevant citations in Section 1 to make our arguments more rigorous and convincing.

Regarding Q2 - Data format reference:
Thank you for pointing this out. The data format has been validated in previous research, and we have added the missing reference to Figure 2.

Regarding Q3 - Language extension motivation:
We have clarified that the extension from 10 to 25 languages was aimed at further validating our method's effectiveness. While the initial 10 languages were representative, expanding to 25 languages allowed us to test our approach more comprehensively across both high-resource and low-resource languages.

Regarding Q4 - Language names:
We appreciate this helpful suggestion and have added a comprehensive table in the appendix that maps language codes to their full names.

Regarding Q5 - Figure captions:
Thank you for pointing this out. We have updated all figures with complete and descriptive captions as suggested.

### Conclusion
In conclusion, we sincerely appreciate your valuable feedback, which has helped us significantly improve our paper. We have implemented your suggestions and hope our revisions and responses provide a clearer understanding of our work. Your recognition is important to us, and we earnestly hope to receive a more favorable evaluation based on these modifications. We welcome further discussion and are happy to address any remaining concerns. Thank you!  



[1] The Power of Question Alignment in Multilingual Reasoning: Broadened Scope and Deepened Insights

[2] Question Translation Training for Better Multilingual Reasoning

[3] MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages
Responses to Reviews and Manuscript Revisions
Yes
No ethics review needed.
Q1: Many references are missing from the claims presented in section 1/ introduction

Q2: Figure 2: Is this data format known for the literature about its performance? Where is the reference?

Q3: Line 076: The motivation for the language extension from 10 to 25 is not clearly explained. What does it mean that the authors extended the language coverage to 25 languages?

Q4: It might be better to provide the full language names in the tables instead of the language codes since, depending on the implementation and the taxonomy used, the codes might be different. 

Q5: Please provide captions on all the figures.
There is significant interest in optimizing the pre-training or fine-tuning mixtures for multilingual LLMs, and this work aims to provide valuable insights into this area.
This paper investigates how varying language proportions in multilingual reasoning datasets impact the fine-tuning performance of large language models (LLMs). The authors aim to demonstrate that carefully balancing language data can enhance LLM performance, achieving state-of-the-art results in multilingual mathematical reasoning and problem-solving while reducing data and translation costs.
W0: The overall writing of the paper lacks coherence, making it difficult to follow the experimental phases. The fragmented presentation of the experiment phases obscures essential aspects of the study, such as training characteristics, the mixture used for fine-tuning, the evaluation tasks, metrics, and analysis. Additionally, several figures, such as Figure 3, lack complete captions, contributing to the confusion. There is also repetition in section headers (e.g., "Phase 2" in Lines 269 and 320), which affects readability and structure.


There are a lot of points and claims that lock in motivation and support:
- W1: Why is language alignment only studied within the context of mathematical reasoning? The rationale for limiting the focus to this particular setting is unclear.
- W2: What criteria were used to select the 10 or 25 languages? How are "high resource" languages defined in this context? Is the selection based on a specific pretraining dataset, and is there a reference for this?
- W3: The grouping of languages appears arbitrary. What was the basis for forming these specific language groups?
- W4: In Line 189, why did the authors choose only high-resource languages for language alignment when, as stated in Line 35, the problem primarily impacts low-resource languages? This choice seems to contradict the motivation expressed in the introduction.
- W5: In Line 45, the paper claims, "The key is to efficiently leverage a small amount of low-resource language data to broadly enhance the multilingual reasoning capabilities of LLMs." Is there any literature or existing research that supports this assertion?

W6: It appears that the evaluation was conducted solely on the MGSM benchmark. This raises concerns about overfitting the optimal mixture to this specific benchmark. How can the proposed method for creating an "optimal mixture" be generalized to other benchmarks?
Yes
No ethics review needed.
1. How do you plan to do quality control for translations in languages where direct verification might be challenging?
I am asking this question because some low-resource languages may not generate accurate translations in English or vice-versa. It would be interesting to know the author's view on this.
The paper shows a unique way to improve multilingual reasoning in LLMs by optimizing the dataset language proportions and that is highly relevant for AI applications. This work contributes a systematic approach to identifying optimal language data ratios. Also, the paper shows extensive methodology and use of Gaussian Process Regression for language group optimization. The outcome/resultant datasets from this work (HighMath-350k and HighCode-350k) are specifically for multilingual reasoning tasks and that will be helpful for future research in this field.
The paper shows a way to enhance multilingual reasoning in LLMs by optimizing the proportion of language data in multilingual reasoning datasets. It addresses challenges in fine-tuning LLMs for low-resource languages, which often lack sufficient training data, by investigating the impact of language proportions on model performance. The paper shows optimal language ratios and volume of data for fine-tuning. The paper also present 'HighMath-350k' and 'HighCode-350k' datasets for multilingual mathematical reasoning and Python-based problem-solving.
There are 25 languages in the phase3 of this study. But in phase 1 and 2, they used a smaller subset. How does this difference may affect the generalizability of findings to low-resource languages is not covered. Also, the datasets generated in this study are based on Non-English to English translation task. However, the study does not cover how to safeguard against translation inaccuracies in low-resource languages.
Yes
No ethics review needed.
In line 252, Figure 3 depicts the variations in translation performance. Which LLM do the results in the figure pertain to? Is the correlation consistent across different LLMs?

In lines 256 to 259, Figure 4 assesses the model's performance on the MGSM dataset after fine-tuning with various non-English to English translation pairs. However, does the author take into account that fine-tuning with WMT datasets might lead to a decline in mathematical reasoning performance?

In lines 368 to 370, when extending 10 languages to 25 languages, how to get the ratio of en:de:ru:fr:others = 24:8:8:8:12?
The paper addresses a significant gap in the literature regarding the optimal balance of language proportions in multilingual reasoning datasets for LLMs.

The research is methodologically sound, with a clear three-phase approach and over 600 groups of experiments providing a robust basis for the findings.
This paper presents a systematic study on the impact of language data proportions in multilingual reasoning datasets on the fine-tuning performance of LLMs. The authors claim to have identified optimal language distributions and data volumes for fine-tuning, leading to state-of-the-art performance in multilingual mathematical reasoning and solving mathematical problems using Python code. The study also aims to reduce data volume requirements and translation costs compared to existing methods.
The paper outlines a three-phase methodology to minimize the search space to a manageable size. A crucial step involves categorizing languages based on their alignment with English. It raises questions about the general applicability of the findings. If a LLM is trained primarily based on non-English datasets, does it still work? 

As stated in the abstract, the paper asserts that it achieves state-of-the-art performance in both multilingual mathematical reasoning and in solving mathematical problems using Python code. However, the experimental section lacks empirical comparisons with existing benchmarks that utilize Python code.

Lastly, the presentation requires further enhancement. The font size of the figures is too small, and it would be beneficial to include descriptions for figures, such as Figure 3 and Figure 7.
Yes
No ethics review needed.
1. what was the reason that English to Non-English translation is not explored?

2. (line 208) what are the two examples used in the prompt? 

3. (line 215) what's the percentage of discarded data entries? 

4. does the pattern / ratio found in this paper single-purposely apply to math reasoning dataset? could it generalize to other multilingual tasks? or should researchers and developers follow the 3 phases and look for a new ratio?

5. (table 1) why is only llama3 investigated in phase 3?

6. (figure 3) the reasoning performance dropped after finetuning (like Swahili), and the dropped amount is close to the improvements. how should we understand this phenomenon?   

7. can you explain the reason why model llama2 was not shown in Table1 but used in phase 3 in the experiment?
The paper is well written and nicely presented with comprehensive tables and figures.

The topic discussed is original and significant, especially during the finetuning stage; that is equal amount of data entries in different languages may not result in the optimal performance. 

Authors made it clear that via experiments; by strategically distributing the amount of data across multiple languages, the multilingual reasoning ability can be acquired at much cost.

The experiment covers 25 languages and most performant open source LLMs.

Authors created two multilingual datasets, namely HighMath-350k and HighCode-350k.

The idea, "given a fixed amount of data volume, how to achieve the optimal performance by including the right amount of data for each aspect", can be borrowed at different stages of the training process, which will lower the cost of training as well as the GPU hours needed.
LLM gains reasoning ability across multiple languages after finetuning. The typical approach is to use the same amount of data entries in each language during finetuning. This paper points out that it is more efficient and effective when the proportion of each language is different. Thru a 3-phase systematic approach, authors found the optimal balance in terms of the proportions and achieved the SOTA performance in math reasoning and python code reasoning. It effectively reduces the data volume and translation cost needed during finetuning for multilingual reasoning.
There are some minor grammar issues, such as no space after a comma or a period. For example, line 221 "selection,in ..." and line 231 "system.In ..."

Another minor layout issue is that the last two figures split the conclusion onto two pages.

Figure 3 and figure 7 might need some description briefly explaining the purpose for improved readability. 

Figure 3 depicts three scores (COMET, BLEURT, BLEU), but there is no elaboration in the context about them.

It is unclear why the authors only adopted finetuning from non-English to English translation pairs and omitted bi-directional pairs in phase 1.

Finetuning configuration is not properly documented.

The translation of dataset is solely based on the model DeepSeek-Chat-v2-236B. Even though authors adopted some methodology to ensure the quality of translation, like preserving formulas and using Arabic numerals, the validation of the translated content can be improved. For example, a subset sampled from the translation can be further validated by native speakers for coherency and consistency.
@inproceedings{
anonymous2024enhancing,
title={Enhancing Multilingual Reasoning in {LLM}s: Insights from Cross-Linguistic Correlations and Optimal Data Proportions},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=S6cBH99BhB},
note={under review}
}
Large language models (LLMs) typically rely on fine-tuning to enhance their reasoning capabilities across various languages. However, limited research has been conducted on the optimal balance of language proportions within multilingual reasoning datasets. To fill this gap, we performed a systematic study to examine how different proportions of language data in multilingual reasoning datasets influence fine-tuning performance. Our study revealed a clear relationship between language proportions in datasets and the fine-tuning performance of LLMs. By fine-tuning multiple LLMs using the appropriate language distributions and data volumes identified in our study, we achieved state-of-the-art performance in both multilingual mathematical reasoning and solving mathematical problems using Python code. Furthermore, our approach significantly reduced data volume requirements and translation costs compared to existing methods, providing a valuable reference for future research.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models Multilingual Reasoning Fine-Tuning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8bf10b5f5c7faa98695c40398e49d69c96cec244.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Enhancing Multilingual Reasoning in LLMs: Insights from Cross-Linguistic Correlations and Optimal Data Proportions
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission