Evaluating Robustness of Reward Models for Mathematical Reasoning
mathematical reasoning;RLHF;reward models;reward overoptimization;language models;benchmark
Yes
No ethics review needed.
* RewardMath is based on the dataset MATH500, where does the dataset MATH500 come from? Is MATH500 prior work (and if yes the citation is missing) or is this a contribution of the paper (in this case it should be made clear).
* Does MATH500 address the incorrect annotation problem found in PRM800K? 
* Can authors also show evaluations and ablations on gsm8k [1] and MATH [2] which are the most common eval tasks for LLM reasoning capabilities?
*  Authors identify that in RewardBench the accepted response often has less steps than rejected ones, which could give a chance for models to reward hack (i.e. reward relies on the number of steps instead of the actual response quality). Did the authors ablate this? I.e.  Does the reward-hacking model predict lower reward if we make the accepted response in RewardMath longer? And vice versa, does it predict higher reward if we make the rejected response shorter? 
* Regarding RewardMath giving more than one rejected responses: If one is trying to do preference learning using a llama model as the base model, is it important for the reward model to know the rejected response generated by a non-llama model should be worse than the accepted response? I.e. the distribution could be very different that it never encounters it during preference learning. I.e. for Figure 4, if we use a llama model as the policy, does one-to-many RewardMath still do better than one-to-one RewardMath chosen & Llama rejection?
* Does reward-model free alignment methods like DPO also suffer from reward model overfitting problem?  What is the advantage of using reward models over reward model-free methods for reasoning tasks? 
* Does the benchmark evaluate cases where both the rejected and chosen response arrive at the same answer, but the rejected answer has the wrong steps? I.e. this is common for truth or false questions. 


[1] Training Verifiers to Solve Math Word Problems

[2]  Measuring mathematical problem solving with the math dataset
* The topic on how to Improve LLM reasoning capabilities has recently gained a lot of attention. This paper focuses on having good benchmarks for evaluating these efforts, and this could be very impactful if done correctly. 

* Authors identify flaws of existing benchmarks and make good efforts to fix them. 

* Paper has good results, specifically Figure 4 is very cool showing RewardMath has stronger correlation with downstream tasks.
Authors aims to design a better benchmark for evaluating reward models in reasoning tasks. Authors identify problems with the previous benchmark RewardBench and proposes RewardMath:
* RewardBench is based on PRM800K which contains many wrong annotations, RewardMath instead is based on MATH500.
* RewardBench uses pairs of (human annotated example, LLM annotated incorrect example).  RewardMath includes more than one incorrect example.
* RewardBench’s accepted and rejected examples have different number of steps, this could be a spurious correlation that leads to reward hacking. RewardMath fixes this.
* RewardBench’s PRM evals uses product instead of mean which biases shorter responses, authors fix this by using mean instead. 

To demonstrate the improvements of rewardmath 1) authors compare performances of different reward models on both RewardBench and RewardMath, and show rewardmath has higher correlation with downstream evals 2) authors show rewardmath exhibits the common trend of larger dataset -> better performance while rewardbench does not.
See questions I have below
Yes
No ethics review needed.
- As the proxy reward model trained from synthetic data shows far from optimal performance in Table 3 (only around 13% for RewardMATH and 69% for RewardBench), can you consider using better proxy RMs? The increase in value from 12.68 to 13.51 is not very convincing to me that this is a strong trend.

- In Figure 6, the gold reward (or even the oracle reward) does not drop for most cases even with the maximum KL distance considered. If a larger N is considered for BoN sampling, will the graph drop down as in Gao et al [2]? For a larger N is RewardMATH still successful in detecting more robust reward models regarding the overoptimization problem?
- The paper provides clear and sufficient empirical evidence that their RewardMATH benchmark is more reliable than the math subset of RewardBench [1]. The empirical results are also clear as LLM policy using BoN on high-scored reward models on RewardBench shows little to no correlation with the performance increase of Math benchmarks (r-square = 0-0.1), while RewardMATH shows a much stronger correlation 
(r-square = 0.6-0.8) in Figure 3.
- The authors have evaluated diverse reward models on RewardMATH, including LLMs (generative reward models), classifier-based reward models, and process reward models.
- The paper considers the problem of over-optimization using a synthetic setup of gold RMs and proxy RMs.  

[1] https://arxiv.org/abs/2403.13787
This paper proposes RewardMATH, a reward model evaluation benchmark focused on the math domain. This benchmark adopts a one-to-many comparison to evaluate reward models robustly. Specifically, it provides 10 responses for each prompt where only 1 response is the answer. The evaluated reward model is considered accurate only when the answer response is given the highest score among all 10 responses. 
The authors also empirically provide sufficient evidence that the RewardMATH benchmark provides reliable estimates for reward models that give policies optimized using BoN are indeed robust in math benchmarks.
- The work would be more interesting if the authors showed any other domains (such as coding or text summarisation or maybe safety) reward model benchmark can be improved by the framework proposed here (by adopting multiple responses and using diverse LLMs to generate outputs). Any initial or limited experiments would be helpful. 

- The lack of PPO (or DPO) usage for policy fine-tuning in experiments seems like a major weakness. The main contribution of this paper is using policy fine-tuning methods to verify if the RewardMATH benchmark scores correlate with the signals it provides during policy fine-tuning. I agree with this approach and am impressed by the number of experiments conducted to verify this using mainly Best-of-N sampling. However, Best-of-N sampling is an inference time method to generate better model outputs using reward models, whereas PPO (or possibly DPO) is the main fine-tuning method researchers use. Although Figure 5 does show a PPO experiment under a synthetic setup, the number of checkpoints or whether the dots follow the findings from Gao et al [2] is not clear to me. Without any solid PPO results, Best of N sampling seems not enough to verify the benchmark's capability of measuring the robustness of reward models. The work will be much more convincing if the authors show more PPO-trained policy evaluations. Or at least, it will be helpful if the author provides more context as to why PPO is hard to train in their non-synthetic setup. Also, I suspect high-scoring reward models on RewardMATH have the ability to find the best response from multiple responses, and Best-of-N adopts a very similar way as it picks the response with the highest reward, resulting in a high correlation of results. Whether this ability will generalize even on PPO setups is not clear to me at this point. 

- Experiment results in Figure 6 compare diverse RMs on both RewardBench and RewardMATH benchmarks with gold or oracle rewards. It would be nice if the authors not only provided the numbers but also a statistical analysis (such as Kendell's tau) that measures the agreement between RewardMATH(or RewardBench) and oracle (or gold) reward scores in Figure 6. 

[2] https://arxiv.org/abs/2210.10760
Yes
No ethics review needed.
1. [Clarification] Are the prompts used to evaluate the LLM judge on REWARDMATH the same as the prompt used to evaluate the LLM judge on the Reward Bench? Different prompting strategy (eg. difference system prompt) raises concerns regarding fair comparison between the two benchmarks.
2. [Clarification] What is MATH500? The author did not mention the details behind this dataset which they used for their benchmark. Were there any steps taken to ensure the dataset is not contaminated with the models being evaluated? If the dataset is used during training on any of the evaluated RMs, the benchmark’s reliability will be undermined. 
3. [Clarification] What was the motivation behind different parts of the Synthetic Data experiment? What was the reasoning behind using the MetaMATH dataset? Why was only 80K out of the 155K data points augmented from MATH used for training? 
4. The authors did not mention how the incorrect solutions are ensured to be actually incorrect. Were there steps taken to validate the said incorrect solutions are indeed incorrect?
1. **Thoroughness**: The paper presents detailed implementations, including training hyperparameters and experimental protocols. This ensures that other researchers can accurately reproduce the experiments and validate the findings. 
2. **Relevance**: This work addresses a critical gap in the field by focusing on reward model evaluation, a crucial area of research that has significant implications for the development of more reliable AI systems.
3. **Motivation**: The paper presents a compelling critique of the existing Reward Bench evaluation metric, establishing a strong foundation for their work. The authors make a persuasive case for developing benchmarks that minimize over-optimization risks, backing their arguments with experimental evidence. This dual focus on improving metrics while addressing practical concerns demonstrates clear motivation for the research.
The paper proposes a new benchmark, REWARDMATH, to improve the robustness evaluation of reward models in mathematical reasoning tasks. It highlights limitations in the existing RewardBench benchmark, which relies on single comparisons between chosen and rejected solutions, potentially leading to reward hacking and misjudgments of model robustness. REWARDMATH addresses this by using one-to-many comparisons with multiple incorrect solutions to better capture robustness and reduce the risk of reward over-optimization. Experimental results indicate that scores on REWARDMATH strongly correlate with policy optimization outcomes and provide a more reliable measure of reward model robustness compared to RewardBench. This benchmark aims to enhance the development and reliability of RLHF systems, with the findings underscoring the potential of REWARDMATH to serve as a trustworthy evaluation tool in this domain​.
1. **Clarity**: The paper is generally well written, however, it has some clarity issues, especially in section 5, which is hard to follow. Clarification questions are asked in the question section, marked with [Clarification]. The authors should address those questions. 
2. **Benchmark Biases**: The paper has several biases, raising concerns on the claimed robustness and reliability. Examples and comments below:

> Line 206: Hence, we first convert the human-annotated solutions from MATH500 into step-by-step machine-generated solutions. We prompt GPT-4, using 4 carefully crafted exemplars for each math subject as part of the prompt.

All correct solutions in the benchmark are generated via GPT-4, raising concerns regarding biases towards GPT-series models. Even though the authors manually inspect the solutions, the solutions were still mainly generated using GPT-4. Notably, the authors observe LLM judges from the GPT-4 Series to perform significantly higher than other models (Line 286), which is likely due to this oversight (since it is known LLM judges tend to bias their own response, eg. GPT-4 family judge favors responses from GPT-4 family). The authors should use a diverse set of LLMs to curate the correct solutions to avoid potential biases.

> Line 805: Secondly, we instruct GPT-4-0125-preview to select a specific step from the correct solution, transform it into an erroneous step, and then prompt again to continue generating the solutions from the erroneous step.

Similar to the previous point, employing GPT-4-0125-preview as editor to insert errors into other LLMs’ answers may introduce biases. Additional validation is needed to ensure the benchmark does not exhibit any bias towards GPT family models. 

> Line 402: We assume Internlm2-7B-reward, which performs well on both RewardBench and REWARDMATH, as the gold RM.

The use of Internlm2-7b-reward as the gold standard lacks sufficient justification and raises several concerns about experimental validity. The author relies primarily on performance metrics from RewardBench and REWARDMATH, but this approach is problematic for multiple reasons. First, the authors themselves criticized RewardBench for containing incorrect ground truth data and failing to adequately assess reward models. Second, using REWARDMATH as a benchmark is circular since it's the very dataset being studied. High scores on these benchmarks alone don't necessarily indicate that a reward model can reliably approximate human preferences. To establish Internlm2-7b-reward as a legitimate gold standard, the author should conduct additional validation studies specifically demonstrating its ability to align with human judgment on mathematical tasks.

> Line 426: We find that proxy reward models trained on smaller datasets reach peak rewards at lower KL divergences, indicating faster over-optimization.

The author assumes KL divergence adequately captures optimization degree without proper justification. KL may not account for other important aspects of policy change. Further study will strengthen the experimental results. 

3. **Comprehensiveness**: This paper's scope is notably narrow, focusing solely on evaluating reward models' performance on mathematical problems. While it attempts to address limitations in a small subset of the Reward Bench dataset, its improvements remain constrained. The study primarily concentrates on reward over-optimization, overlooking other potential vulnerabilities in reward model benchmarking. Additionally, the benchmark's methodology of comparing one correct solution against multiple incorrect ones limits its thoroughness. Furthermore, the author's assumption that MATH500 adequately represents mathematical reasoning tasks may be oversimplified. These limitations collectively suggest a need for a more comprehensive approach to reward model evaluation.
Yes
No ethics review needed.
Why do the authors choose 10 generations with 9 incorrect and 1 correct answer in the k-way comparisons? How does the choice of k and numbers of correct and incorrect answers affect the resulting correlations and reward overoptimization? The relationship between reward overoptimization and the proposed benchmark needs more rigorous analysis.
The paper provides a good correlation study between the proposed benchmark and best of N downstream performance in datasets like MATH and Gaokao etc. The proposed one-to-many comparison seems to be on the right direction for better correlation with downstream performance.
The paper introduces REWARDMATH, a benchmark for evaluating reward models in mathematical reasoning tasks, arguing that it provides a more reliable evaluation than the existing RewardBench by using one-to-many comparisons instead of pairwise comparisons. The authors validate their approach by showing correlation between benchmark performance and best-of-n sampling results.
I have the following major concerns:

1. Technical Contribution & Novelty: The primary contribution seems to be replacing pairwise comparisons in reward bench with best of N comparisons, which is an incremental modification rather than a substantial methodological advance. I do not think the change only is sufficient for a publication in top machine learning conference. The correlation between N-way comparison performance and best-of-N sampling results is somewhat expected and doesn't provide deep insights into reward model behavior. 


2. Unclear Definition of Robustness: The paper uses "robustness" throughout but never provides a precise definition. The authors seem to equate robustness with correlation to best-of-n sampling results, but this is circular reasoning since the benchmark itself uses similar methodology. There's no clear framework for what properties a "robust" reward model should have beyond empirical correlation with certain metrics. 


3. Limited Experimental Validation: The paper relies heavily on correlation with best-of-n sampling as validation, but doesn't explore other important aspects of reward model behavior. To make the paper deeper and broader, it would be great if the authors could try comparing the correlation of different real down-stream fine-tuning techniques like BoN SFT, DPO, PPO etc. and see whether how RewardBench and RewardMath correlate with downstream performance there. It would also be interesting to see if such observation extends to other domain like coding, and perhaps even open-ended generations without ground truth.
Yes
No ethics review needed.
Q. How much of the relatively poor results for RewardBench is due to the noisy annotations inherited from PRM800K, as mentioned in Sec. 3.1? In other words, could simply fixing these annotations significantly change the comparison?
1. The paper identifies a particular limitation of RewardBench, a popular benchmark for evaluating reward models, in assessing mathematical reasoning and introduces a new benchmark that addresses this issue by including one-to-many comparison data.
2. The authors provide extensive experiments and analyses across different reward model types, including both proprietary and open models, and assess various performance metrics.
The paper addresses a specific limitation of RewardBench, a widely used benchmark for reward model evaluation, in its assessment of mathematical reasoning capabilities. To this end, the authors introduce RewardMATH, a new benchmark that employs one-to-many comparisons of chosen and rejected responses to mathematical questions to enhance evaluation robustness. Experiments show that RewardMATH correlates well with policy performance and is more effective at identifying potential reward overoptimization and the reliability of reward signals.
1. The benchmark comparison primarily involves RewardBench, which is designed to evaluate reward models more holistically across various domains. However, is the comparison in terms of mathematical reasoning appropriate, given that RewardMATH is specifically designed for this purpose? If RewardBench is indeed the most comprehensive eval set even for mathematical reasoning tasks prior to RewardMATH, it would be helpful to clarify this point.
2. The benchmark appears to lack cases where reward models must distinguish between correct solutions of varying quality, such as those missing reasoning steps. It is also unclear whether 500 samples is sufficient to cover diverse mathematical reasoning tasks.
3. Tables 1 and 2 report performance comparisons of various LLMs on RewardBench and RewardMATH. The results seem to merely suggest that the two benchmarks differ significantly. Can we conclude from these results that "high scores on RewardBench do not guarantee **robustness** in reward models"?
We propose a design for a reliable benchmark for reward models and validate our design using the results of optimized policies and through the lens of reward overoptimization.
@inproceedings{
anonymous2024evaluating,
title={Evaluating Robustness of Reward Models for Mathematical Reasoning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=0er6aOyXUD},
note={under review}
}
Reward models are key in reinforcement learning from human feedback (RLHF) systems, aligning the model behavior with human preferences.
Particularly in the math domain, there have been plenty of studies using reward models to align policies for improving reasoning capabilities.
Recently, as the importance of reward models has been emphasized, RewardBench is proposed to understand their behavior.
However, we figure out that the math subset of RewardBench has different representations between chosen and rejected completions, and relies on a single comparison, which may lead to unreliable results as it only see an isolated case.
Therefore, it fails to accurately present the robustness of reward models, leading to a misunderstanding of its performance and potentially resulting in reward hacking.
In this work, we introduce a new design for reliable evaluation of reward models, and to validate this, we construct RewardMATH, a benchmark that effectively represents the robustness of reward models in mathematical reasoning tasks.
We demonstrate that the scores on RewardMATH strongly correlate with the results of optimized policy and effectively estimate reward overoptimization, whereas the existing benchmark shows almost no correlation.
The results underscore the potential of our design to enhance the reliability of evaluation, and represent the robustness of reward model.
We make our code and data publicly available.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
mathematical reasoning RLHF reward models reward overoptimization language models benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1d4b478bfde9d6708c1e37c9fa5ce32c5f4d1a17.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/f881b604bd9898f154a5ed11cc10b7f855da77d6.zip
Evaluating Robustness of Reward Models for Mathematical Reasoning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission