RMB: Comprehensively benchmarking reward models in LLM alignment
LLM Alignment;reward model;evaluation
Yes
No ethics review needed.
Please refer to the weakness, I'm happy to modify my rate based on the response of the authors and refer to other reviewers' comments.
1. The paper is written well and is easy to understand.
2. The studied problem is significant..
3. The results seem to outperform the SOTA datasets of the reward model evaluation.
The paper presents RMB, a benchmark evaluating reward models (RMs) across 49 real-world scenarios, improving alignment assessments for LLMs. RMB reveals generalization issues in current RMs and explores factors affecting generative RM effectiveness.
1. The paper currently includes some discussion related to benchmark comparisons in Section 5.2, particularly with RewardBench. However, a more explicit comparison of the features and approaches of existing benchmarks early in the paper would better highlight the novelty of this work. Rather than relying on experimental results to convey superior performance, detailing how our model’s capabilities differ from those of previous benchmarks would strengthen the paper's contribution.
2.  In the conclusions about evaluation results, statements like “It is hard for an RM to be both competitive in judging helpfulness and harmlessness” and “Generative models show great promise in reward modeling” could be enhanced by including an explanation of the underlying reasons for these findings. Adding these insights may help the community engage more deeply with the work, offering valuable perspectives that could inspire further exploration beyond the results alone.
3.
Yes
No ethics review needed.
- Were any alternatives to GPT-4 scoring considered? How might the benchmark differ if using other models (or ensembles) or human annotators? (I understand human annotation is costly) 

- Did you observe any patterns when best-of-N evaluation diverged significantly from pairwise evaluation?

- What recommendations would you make for effectively using RMB during the reward model development process? Is it simply meant to be used as a validation set? 

- What tradeoffs between helpfulness and harmlessness exist? Why do you think they seem to be mutually exclusive? 

- Do you see any negative impacts or possible misuses of the benchmark?
RMB presents many strengths, especially against its main predecessor, RewardBench. Primarily, the dataset size is much larger than any similar datasets used to evaluate reward models. The paper does thorough investigation of best-of-N evaluation, which related papers had proposed as a future direction of research. The authors do a good job of categorization and subcategorization, to ensure broad coverage across different tasks. The benchmark shows good progress towards a useful proxy for downstream model performance. Further, the paper is clearly written and has good formatting.
The paper proposes RMB (Reward Model Benchmark) which is a benchmark for evaluation of reward models used in LLM alignment. It consists of 49 real–world scenarios between the two proposed categories “helpfulness” and “harmlessness”. They use two methods of evaluation: pairwise comparison and best-of-N evaluation, which tests the models’ ability to choose the best response from multiple candidates. Another key finding is that generative models show strong performance, often outperforming discriminative models. Further, they show positive correlation between benchmark performance and downstream alignment tasks.
There are a couple of weaknesses in the paper. The human annotator sample size is small relative to the size of the dataset. The correlation analysis focuses mainly on the best-of-N sampling rather than RLHF, which is acknowledged in the paper as a limitation. While mentioning that it is hard for a model to be both competitive in judging helpfulness and harmlessness, the authors don’t deeply explore the trade-offs between the two metrics. The authors used various models in the categorization and filtering steps, but the scoring was only done by GPT4, which might introduce bias. There are also a few styling errors and typos throughout the paper that would need to be cleaned up prior to approval.
Yes
No ethics review needed.
See Weakness.
- The paper is well-structured and clearly written, explaining the methodology and results effectively.
- RMB effectively addresses the limitations of developing reward models that align with the objective, whether helpfulness or harmlessness. Correlation evaluations show that the reward model's performance on RMB can reflect the performance of the downstream aligned model more accurately. This would enable researchers to evaluate and iterate on reward models more efficiently before training a more computationally challenging model alignment.
- The paper provides solid empirical evidence supporting the effectiveness of RMB. The experiments on pairwise and best-of-N accuracy cover different generative and discriminative reward models and provide a good insight into their usefulness as judges.
The paper proposes RMB, a comprehensive benchmark for evaluating reward models (RMs) in LLM alignment. The dataset contains comprehensive real-world scenarios on helpfulness and harmfulness and responses generated by different LLMs. The paper validates the benchmark's effectiveness by showing a correlation with downstream alignment performance.
- The reliance on LLM-generated response in the dataset curation may pose potential long-term limitations. As reward models are often used to align newer LLMs, using responses from current LLMs to evaluate them could create a circular dependency. In other words, this benchmark may not be able to differentiate reward models that could guide LLMs that are beyond current capabilities. In the next iterations, it would be nice to incorporate human-generated responses to ensure more diverse sources of responses.
- The correlation evaluation is leveraging best-of-N as an alignment, which is not the only way people are using the reward models. Would it be possible to gather a set of reward models and aligned model pairs that leverage the reward model during training and see the correlation there?
- nit: Mu et al.. citation is missing a year (L229)
@inproceedings{
anonymous2024rmb,
title={{RMB}: Comprehensively benchmarking reward models in {LLM} alignment},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=kmgrlG9TR0},
note={under review}
}
Reward models (RMs) guide the alignment of large language models (LLMs), steering them toward behaviors preferred by humans. Evaluating RMs is the key to better aligning LLMs. However, the current evaluation of RMs may not directly correspond to their alignment performance due to the limited distribution of evaluation data and evaluation methods that are not closely related to alignment objectives. To address these limitations, we propose RMB, a comprehensive RM benchmark that covers over 49 real-world scenarios and includes both pairwise and Best-of-N (BoN) evaluations to better reflect the effectiveness of RMs in guiding alignment optimization.
We demonstrate a positive correlation between our benchmark and the downstream alignment task performance. Based on our benchmark, we conduct extensive analysis on the state-of-the-art RMs, revealing their generalization defects that were not discovered by previous benchmarks, and highlighting the potential of generative RMs.  Furthermore, we delve into open questions in reward models, specifically examining the effectiveness of majority voting for the evaluation of reward models and analyzing the impact factors of generative RMs, including the influence of evaluation criteria and instructing methods. We will release our evaluation code and datasets upon publication.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM Alignment reward model evaluation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/58b90d568446bb9bd4bdd3f9e4025196dceba857.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
RMB: Comprehensively benchmarking reward models in LLM alignment
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission