CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
Preference Model Pretraining
Yes
No ethics review needed.
- Is Majority voting@K comparable with reranking top K with a RM in Figure 3?
- The proposed method is highly scalable, allowing it to create 28 million preferred and rejected response pairs.

- The reward model can improve two different reasoning tasks (math reasoning and logic reasoning).
The paper presents CodePMP, a novel and scalable preference model pretraining (PMP) pipeline aimed at enhancing the reasoning capabilities of Large Language Models (LLMs). 

CodePMP builds its dataset in two stages: 1) generating instructions from GitHub files using a CodeLLM, and 2) generating two responses for each instruction from a strong CodeLLM and a weak CodeLLM, which are the preferred and rejected response pairs. 28 million such response pairs are constructed.

After pre-training an LLM on these 28 million pairs, CodePMP improves RM finetuning efficiency. The resulting reward model built upon CodePMP can then be leveraged to improve performance on reasoning tasks, including solving mathematical and logical problems.
- CodePMP creates preference pairs for coding tasks, but coding tasks are not evaluated in experiments. 

- The CodePMP data is constructed by deepseek-coder-instruct. It would be interesting to see whether CodePMP can further improve deepseek-coder-instruct on coding tasks.

- The reward model is initialized using Qwen2 models (Qwen2-1.5B and Qwen2-7B), which are more capable in math reasoning than the math generator MetaMath-Mistral-7B in Section 4.1.3. A more meaningful setting would be to examine whether the Qwen2 based reward model can further improve the performance of Qwen2 or Qwen2-Math on math reasoning tasks.

- Majority voting should be included as a baseline in Figure 3.

- Further analysis is needed to understand why the coding-based CodePMP contributes to improvements in math and logical reasoning.
Yes
No ethics review needed.
1. Figure 4 and 5 are too small to watch.
2. Why CodePMP can help reward modeling in mathematical tasks? Please give more analysis in the paper.
3. Reference formatting issues, such as line 429.
4. Would you consider using a larger reward model to verify the scalability?
1. This work proposes an interesting idea of preference pre-training.
2. This method is automated, reducing the dependency on manually annotated preference data.
3. With best-of-N strategy, CodePMP could improve LLM's reasoning performance.
The paper presents codepmp, a scalable pretraining method designed to improve the reward model by leveraging synthesized code-preference pairs. CodePMP collects a vast repository of publicly available source code from GitHub and leverages a strong and a weak model to generate synthetic preference pairs to pretrain a reward model. Utilizing this pretrained reward model, LLM's ability to solve reasoning tasks is largely improved by best of N sampling.
1. Unfortunately, the effectiveness of the RM was not fully verified, e.g. by using the RM for RFT or PPO training.
2. It remains unclear whether and how code training could help reasoning tasks in natural language. It would be great if the authors could have explored more on relationship between coding and reasoning tasks for model training.
3. No mention of whether the dataset will be open sourced.
4. The models used in data construction are limited. It would be helpful to verify the generalization with more number of generators, at least the ones used to construct the training data.
5. Evalution tasks are limited. Regular coding benchmarks like humaneval and mbpp should be relevant.
Yes
No ethics review needed.
1. In the Data Construction, a description summarizer is used to generate prompts that describe the code. There is no any detail on this part. And what is quality of constructed data pair? There is no any related analysis in this work. It is not a human-annotated dataset. So it is important to show the data quality.  

2. In Section 4.2.2, how the different number of N candidate responses are generated? Is Qwen2 model used to sample these candidate response? There is no details about this.

3. In Figure 4, the author claims that: RM with Code PMP initialization consistently outperform. However, it seems not right. Figure 4 (g) shown a counter part case. RM with Code PMP initialization is consistently  worse than RM without Code PMP initialization. Please check in detail.

4. As shown in Figure 5, increasing the number of code-preference pairs consistently improve accuracy. Look like there is still some improvement space. Do you think more preference data can be obtained without too much work?

Other questions: is the PMP data planed to be released? how many code-preference pairs? (only file number and token number are shown, e.g, Python files, with 20 million files and 13.1 billion tokens,)
1. A pipeline that create synthesize code-preference pairs is introduce in this work. It can help to solve the scarcity of high-quality preference data if it is working well.

2. Large improvements are achieved on several reasoning tasks (GSM8K, MATH, ReClor, LogiQA2.0) with these synthesize code-preference pairs data.

3. Some details of CodePMP are shown. These can be helpful to the community.
This work introduce CodePMP, a pipeline that create synthesize code-preference pairs in order to have more high quality preference data. The authors show that a better reward model can be obtained with these preference pair data. And it also shows the improvements in several reasoning tasks (GSM8K, MATH, ReClor, LogiQA2.0) with these synthesize code-preference pairs data.
1. The experiments details are missing or confused. It is better to clarify in the next version. Please check the following Questions section for more details. 

2. In the experiments, only two Qwen2 models are used for the evaluation. Other model family results can be used for the verification of the methods.
Yes
No ethics review needed.
Given that CodePMP relies heavily on code data, how well do you expect it to generalize to reasoning tasks that do not have a structured, logical basis like code (e.g., common-sense reasoning, social interactions, or natural language inference)? Have you explored tasks beyond mathematical and logical reasoning?
**1. Efficiency in Reward Model Fine-Tuning**


CodePMP significantly enhances the sample efficiency of reward model (RM) fine-tuning, as the experiments indicate that fewer samples are required to achieve high accuracy. The paper shows that RMs initialized with CodePMP outperform directly fine-tuned models, even with smaller amounts of data. This efficiency makes the method appealing for real-world applications where data collection can be expensive or time-consuming.

**2. Scalable Data Generation**


One of the major strengths of CodePMP is its ability to generate large-scale preference data using code-preference pairs. By synthesizing data from publicly available source code on platforms like GitHub, the authors address the scarcity of high-quality preference data. This reduces the reliance on costly human annotation, making the pretraining process significantly more scalable and potentially more cost-effective than traditional methods relying solely on human feedback.

**3. Leveraging Code for Reasoning Tasks**

The paper introduces an innovative use of GitHub-sourced code to enhance reasoning in LLMs. By leveraging the logical structure of code, the authors create code-preference pairs, where stronger models generate "accepted" responses and weaker models generate "rejected" ones. This approach capitalizes on the abundance of publicly available code on GitHub, using it to simulate reasoning tasks. The structured nature of code makes it an ideal dataset for pretraining models to improve problem-solving and logical reasoning capabilities, aligning with recent research showing that training on code boosts reasoning performance in LLMs.
This paper presents CodePMP, a novel pipeline for preference model pretraining aimed at enhancing the reasoning capabilities of large language models (LLMs). The primary challenge addressed is the scarcity of high-quality preference data, which is crucial for reinforcement learning from human feedback (RLHF) but costly to annotate. The authors propose a solution by leveraging code-preference pairs synthesized from publicly available high-quality source code, such as from GitHub, to improve reward model finetuning.

The CodePMP pipeline generates millions of code-preference pairs by using two different code models (a stronger model and a weaker model) to create a “chosen” and “rejected” code response for a given code prompt. These pairs are then used to pretrain the preference models, leading to better sample efficiency and improved reasoning performance in downstream tasks like mathematical reasoning (GSM8K, MATH) and logical reasoning (ReClor, LogiQA2.0).
**1. Lack of Examples** 

The paper lacks clear and detailed examples that illustrate how CodePMP actually solves reasoning tasks using code-preference pairs. While the overall methodology is explained, it's hard for the reader to grasp how reasoning is specifically improved through the use of code data. The absence of step-by-step examples of how the code-preference pairs are used to train the reward model leaves an important gap. For instance, showing a few real examples of code, its corresponding “chosen” and “rejected” responses, and how these responses contribute to solving a reasoning task would be highly valuable. The appendix, while providing some additional technical details, does not adequately fill this gap. 


**2. Limited Baselines**

The paper’s comparison to **alternative methods** is quite limited. It focuses primarily on showing the improvements over direct RM fine-tuning without pretraining, but fails to benchmark CodePMP against other advanced reasoning techniques such as:

Chain-of-Thought (CoT) prompting, which has been shown to significantly improve reasoning performance by breaking down tasks into intermediate steps.
Search-based methods like Monte Carlo Tree Search (MCTS), which have been effective in enhancing reasoning capabilities in language models through external computation and verification steps.
A reader would expect a thorough comparison of CodePMP against these methods, especially since these are widely used in the current state-of-the-art LLM systems. Additionally, the paper should clarify:

How does CodePMP stack up against CoT and search-based approaches in the same experimental setup?
What are the strengths and weaknesses of CodePMP relative to these techniques?

**3. Challenges in Maintaining Data Quality with Evolving LLM Capabilities**

A key limitation of the CodePMP methodology is its reliance on two distinct models—one to generate "accepted" (high-quality) responses and another to produce "rejected" (low-quality) responses. While this setup works for synthesizing large quantities of training data, there is a valid concern that as LLMs improve, the gap between the two models could narrow, leading to mismatches in the data quality. For example, there is a risk of **data distribution drift:** As the strong model improves, the generated "accepted" responses may no longer reflect the kind of reasoning tasks the preference model was initially trained on, leading to a mismatch between the training data and the actual data seen in practice. The system might overfit to synthetic training data that does not generalize well to the actual tasks faced by stronger LLMs.
@inproceedings{
anonymous2024codepmp,
title={Code{PMP}: Scalable Preference Model Pretraining for Large Language Model Reasoning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=tpVQHb4pea},
note={under review}
}
Large language models (LLMs) have achieved notable advancements in natural language understanding and generation, driven by scalable pretraining and advanced finetuning techniques. However, improving reasoning abilities in LLMs, particularly through reinforcement learning from human feedback (RLHF), remains a challenge due to the scarcity of high-quality preference data, which is often labor-intensive to annotate and essential for reward model (RM) finetuning. To alleviate this issue, we introduce CodePMP, a scalable preference model pretraining (PMP) pipeline that leverages vast amounts of code-preference pairs synthesized from publicly available, high-quality source code. CodePMP improves the sample efficiency of RM finetuning by sufficiently pre-training preference models on synthesized code-preference pairs. In addition to validating CodePMP on widely used mathematical reasoning tasks (GSM8K, MATH), we also demonstrate its effectiveness on logical reasoning benchmarks (ReClor, LogiQA). The results consistently indicate that CodePMP significantly improves the reasoning performance of large language models (LLMs). Furthermore, our findings underscore the critical role of scalable preference model pretraining (PMP) in achieving efficient reward modeling.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Preference Model Pretraining
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f65f3bb1a52e0cce0bcffc96907422164882851e.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission