OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
Math Reasoning;Synthetic Data
Yes
No ethics review needed.
In Table 4, the authors compare with Qwen 2.5, Mathtral, NuminaMath and DS-Coder. I understand these are strong open-source baselines so comparing with them is sufficient to see the signal, but I would still encourage the authors include more OSS/ proprietary  models for further comparison (even other models are much weaker or stronger).
I think this paper is in general a high-signal paper. Specifically:

- **Well-written with high-density of details**: this paper is well-written. The flow is a naturally interleaved summary, detail, and explanations. Many important details (size of model, size of dataset, specific accuracy numbers .etc) are packed in the writing with clear explanations of the motivation.
- **Well-execution**: the experiments in this paper is clear. I particularly appreciate the effort and discussion about decontamination, which essentially strengthens this paper quite a bit.
- **Sufficient signal from experiments**: the model developed in this paper is not top 1 in the leaderboard (but understandable because Qwen uses more resources). But the signal sent by the experiments (tolerate to imperfect annotation, importance of question diversity .etc) is clear.
This paper introduces OpenMathInstruct-2, an SFT recipe for improving the math reasoning of language models. With extensive engineering efforts and experiments, the authors show the importance of solution formats, the need of a strong teacher, question diversity, and the robustness to imperfect solutions. Putting everything together, the authors show that OpenMathInstruct-2 improves Llama 3.1 and performs strongly compared to other open-source models.
There are two conclusions draw by the authors that I am not so sure: 

- **Format of the data**. In general I agree that excessively verbose CoT may not be good for either performance or user preference, but it is not very clear how exactly the CoT proposed in this work contribute to the improved accuracy of reasoning. The higher numbers reported by the author is a good observation but not the mechanism behind. In addition, existing successful models like O1, in many cases, also offer verbose CoT, yet its performance is remarkable. In general, I tend to believe the level of detail that should be included in the CoT may still need further investigation, and I do see mixed signals favor or not favor verbose CoT.
- **Limited domain**: the authors still use GSM8K and MATH as the seed data — they do generate new questions but the seed is still GSM8K and MATH. The author test the performance on more challenging tasks (AMC and AIME), which is a nice practice. But I do feel that the field is moving forward to wider domains (not just math but also other STEM) and higher level (not just high-school but college). For this work I think it is sufficient to just use GSM8K and MATH, but I would suggest the authors to move forward in their future endeavors — only focusing on these two datasets would limit the authors from making higher impact.
Yes
No ethics review needed.
1. What do you think about a self-improving framework, such as using an 8B Llama model to generate synthetic data and fine-tune it, then repeating the process for 2-3 rounds?
2. What do you think is the reason for the inadequate performance of on-policy data?
1. Open-Source Dataset: This paper released the code, the finetuned models, and the OpenMathInstruct-2 dataset under a commercially permissive license.
2. Extensive Experiments: Not only main experiments, authors did a lot ablation study about: 1. choice of data generation model, 2. robustness of SFT, 3. impact of question diversity. The results are interesting and solid.
This paper introduces OpenMathInstruct-2, a large open-source dataset for improving mathematical reasoning in language models. The authors propose a more concise Chain-of-Thought (CoT) format, which enhances performance compared to existing methods. Through extensive experiments, they demonstrate that using a strong teacher model for data generation is more effective than on-policy data from weaker models. The study also highlights the robustness of models to low-quality data and emphasizes the importance of question diversity in fine-tuning. Overall, the paper contributes to open-source resources and offers valuable insights into optimizing mathematical reasoning in language models.
This is a solid work; I don't think there are any obvious weaknesses.
Yes
No ethics review needed.
- The authors acknowledge but don't fully investigate why their OpenMath2-Llama3.1-70B model shows less dramatic improvements compared to their 8B model. They hypothesize that "our data blend or solution format might be more suited for weaker models" but don't test this hypothesis.

- Have you tried other sizes of teacher models?
* **Ablation Studies & Methodology** - The paper excels in its systematic investigation of dataset design choices. The ablation studies on solution format, teacher model quality, and robustness to low-quality solutions are thorough and well-controlled.

* **Open Science & Reproducibility** - The authors make significant contributions to open-source development by releasing their dataset, code, models, and detailed methodology. Their comprehensive documentation of prompts (in appendices D.1-D.5) and data generation pipelines enable full reproducibility.

* **Math Synthetic Data Insights** - The paper provides several interesting findings that advance our understanding of LLM synthetic data generation in math reasoning. For instance, the discovery that shorter solution formats can outperform more verbose ones (OpenMath CoT vs Llama CoT) challenges conventional wisdom about detailed explanations. Similarly, their finding that SFT is robust to up to 20% incorrect solutions provides valuable guidance for dataset creation, suggesting that aggressive filtering may be unnecessary.
The paper introduces OpenMathInstruct-2, a large open-source dataset for mathematical reasoning containing 14M question-solution pairs (approximately 600K unique questions), making it nearly eight times larger than previous open-source datasets. The authors conducted systematic experiments to understand how different dataset design choices affect model performance, finding that:

1. Solution format matters - excessive verbosity can hurt performance. 
2. Teacher model quality is important
3. SFT is robust to low-quality solutions
4. Question diversity is crucial.

Using these insights, they created OpenMathInstruct-2 and demonstrated its effectiveness through model fine-tuning. Their OpenMath2-Llama3.1-8B model achieved 67.8% accuracy on MATH, outperforming Llama3.1-8B-Instruct by 15.9%. The authors also introduced a thorough LLM-based decontamination pipeline to ensure test set integrity. The paper concludes by releasing all code, models, and the dataset under a permissive license to support open-source development in mathematical reasoning.
**Insufficient Exploration of Question Generation Quality** - While the paper presents a thorough decontamination pipeline for synthetic questions, it lacks a detailed analysis of the quality and diversity distribution of generated questions. The authors don't evaluate whether their synthetic questions maintain: 1) the same quality as the original questions; 2) the same difficulty level as the original dataset 3) the same diversity (or more diverse) compared with the seed data. This might be crucial. The authors' ablation studies also show that even though the model is error-tolerant to wrong answers to some extent (e.g., 20%), the accuracy would downgrade more if there were too many error examples in the training. It would be necessary to examine the quality of the 14M generated samples. 


**Potential Knowledge Distillation Bottleneck** - There's a fundamental question about the approach. Though the performance is promising on a small student model (i.e., Llama-3.1-8B) with the 14M synthetic data, it is unclear whether improvement comes from primarily distilling the teacher's knowledge or improving genuine mathematical reasoning capabilities. The diminishing returns with larger student models (70B vs 8B) are consistent with distillation behavior, where the student model's capacity approaches needed to mimic the teacher's surface patterns rather than learning deeper reasoning. This limitation could be particularly important for scaling mathematical reasoning capabilities beyond the current teacher model's abilities. I'd suggest either of the experiments:
- Further fine-tune the teacher model (340B) on the 14M data (which might be costly)
- Generate a relatively larger-scale training set (>1M) using an 8B model and fine-tune the model on the generated data. 

**Incomplete Investigation of Teacher Model Selection** - The authors demonstrate that Llama3.1-405B-Instruct outperforms Llama3.1-8B-Base as a teacher model, but they don't explore the space between these two extremes. Would a 70B model be sufficient for a teacher? Is there a clear relationship between teacher model size and generated solution quality? Given the computational cost implications of using a 405B parameter model for data generation, understanding the minimum viable teacher model size would be practically valuable.
Yes
NA
No ethics review needed.
How do authors find or select the OpenMath CoT format? The findings that OpenMath CoT is short but performs better is interesting. I think better COT format exploration is an interesting direction.

In section 3.1, aurthors discussed the LLM decontamination. Is the SFT data is processed the decontamnation? Have authors conducted experiments comparing the performance of with and without the decontamination?
The paper introduces a large-scale math reasoning dataset, which would be highly beneficial if released.

It is well-written and easy to follow.

The authors offer valuable insights into data curation and supervised fine-tuning, discussing aspects like the format of chain-of-thought and the diversity of questions. Some insights are interesting and can be explored deeper.
This paper curate a large scale (14 M qa-pairs) and high-quality math reasoning SFT datasets - OpenMathInstruct-2. At the same time, authors find some insights:

Verbose solutions is detrimental to SFT performance. Data generated by strong model is better. SFT allow some low-quality solutions. Question diveristy is crucial.

SFT the Llama-3.1-8B-Base on OpenMathInstruct-2 dataset achieve 67.8% performance, outperforming Llama3.1-8B-Instuct by 15.9%
The process of dataset generation is not novel. The main advantage is the larger scale.

Some findings like data generated by strong teacher is better, question diversity matters are very natural.
We create a massive, high-quality math instruction data to support open-source efforts on math reasoning.
@inproceedings{
anonymous2024openmathinstruct,
title={OpenMathInstruct-2: Accelerating {AI} for Math with Massive Open-Source Instruction Data},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=mTCbq2QssD},
note={under review}
}
Mathematical reasoning continues to be a critical challenge in large language model (LLM) development with significant interest. However, most of the cutting-edge progress in mathematical reasoning with LLMs has become closed-source due to lack of access to training data. This lack of data access limits researchers from understanding the impact of different choices for synthesizing and utilizing the data. With the goal of creating a high-quality finetuning (SFT) dataset for math reasoning, we conduct careful ablation experiments on data synthesis using the recently released Llama3.1 family of models. Our experiments show that: (a) solution format matters, with excessively verbose solutions proving detrimental to SFT performance, (b) data generated by a strong teacher outperforms on-policy data generated by a weak student model, (c) SFT is robust to low-quality solutions, allowing for imprecise data filtering, and (d) question diversity is crucial for achieving data scaling gains. Based on these insights, we create the OpenMathInstruct-2 dataset which consists of 14M question-solution pairs (≈ 600K unique questions), making it nearly eight times larger than the previous largest open-source math reasoning dataset. Finetuning the Llama-3.1-8B-Base using OpenMathInstruct-2 outperforms Llama3.1-8B-Instruct on MATH by an absolute 15.9% (51.9% → 67.8%). Finally, to accelerate the open-source efforts, we release the code, the finetuned models, and the OpenMathInstruct-2 dataset under a commercially permissive license.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Math Reasoning Synthetic Data
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9e1869ca39e43554e4bcdd3bee5c9f6466c718bd.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission