Generative Reward Models
RLHF;reward model;LLM Judge;reasoning
Yes
No ethics review needed.
* Can you provide experimental results showing the Best-of-N performance with your reward model? This would help understand how the improvements in preference modeling translate to actual policy performance.
* Given the computational requirements of majority voting, how would you recommend implementing this approach practically in PPO? What modifications or optimizations might be needed?
* Could you elaborate on the comparisons with similar existing methods in the literature? Specifically, how does your approach differ from and improve upon previous work like PairRM [1] and Self-Rewarding LMs [2]?
* The paper successfully demonstrates that majority voting (self-consistency) at test time can substantially improve preference modeling accuracy. The results show consistent improvements across different datasets, providing a practical way to enhance preference modeling through additional compute at inference time.
* The evaluation for reward modeling covers both in-domain (UltraFeedback) and out-of-domain (RewardBench) datasets. The experimental results demonstrate robust performance improvements, particularly in OOD scenarios where traditional approaches often struggle.
* The paper provides strong empirical evidence that DPO training works effectively with CoT-GenRM. The experiments show that this training approach leads to better generalization compared to other training methods.
* The paper proposes GenRM (no CoT), which is similar to PairRM [1], focusing on direct pairwise preference prediction without the Bradley-Terry assumption. The core technical contribution is mainly about removing the explicit reward modeling design.
* Introduces CoT-GenRM (similar to Self-Rewarding LMs [2]) with two training approaches: bootstrapping and rationalization. Through experimental analysis, the authors found that while rationalization works well in-domain, it doesn't generalize effectively to OOD domains. The paper demonstrates this through comprehensive evaluations.
* A key advantage demonstrated is that self-consistency can be used as test-time compute scaling to improve preference modeling accuracy. The experimental results show that majority voting with 32 samples can significantly boost model performance across different evaluation settings.

[1] PairRM: https://arxiv.org/abs/2306.02561
[2] Self-Rewarding LMs: https://arxiv.org/abs/2401.10020
* The evaluation is limited in demonstrating what the preference accuracy improvements mean for actual policy performance. While the paper shows improvements in preference modeling accuracy, this metric heavily depends on the policy sample distribution. Notably absent is an analysis of Best-of-N (BoN) performance with the proposed reward model, which would be crucial for understanding practical impact.
* The practical implementation raises significant concerns. The requirement for 32 majority votes and the limitation to pairwise comparison settings could make it challenging to implement in real-world scenarios, particularly in PPO. The paper doesn't adequately address how to handle these computational requirements in practice.
* The core concepts of GenRM (no CoT) and CoT-RM have been previously explored in literature, such as PairRM [1] and Self-Rewarding LMs [2]. The paper would benefit from more thorough comparisons with existing methods to better establish its novel contributions.
Yes
No ethics review needed.
I am wondering if there are results showing the improvements of policy models trained on STaR-DPO.
The strength of this paper is its potential significance and novelty. The proposed direct formulation of the preference indicator looks interesting to me, which may allow more types of approaches and data sources to be combined to train stronger reward models, especially in real-world scenarios that require strong generalization ability.
This paper proposes a reward modeling formulation GenRM that explicitly indicates the pairwise preference with $\pi_{\phi}(I | x, y_1, y_2)$, which allows a natural combination of RLHF and RLAIF approaches. By incorporating 1) CoT of Self-taught Reasoning traces of rationalization models, and 2) DPO-style optimization objective, the trained model STaR-DPO achieves on-par performance with Bradley-Terry reward models on IID settings and generalizes much better on OOD settings, e.g. safety and reasoning tasks.
The clarity and quality of the paper can be significantly improved. Specifically, for clarity, the organization of the experimental section can be re-structured to highlight the answers for the proposed questions in lines 296-302. Also, the accuracy numbers in Figure 2 and Figure 4 are recommended to be included in the paper, at least in the Appendix.

Concerning quality, it would be strongly recommended to have policy models trained on STaR-DPO to see whether this improvement in reward modeling can truly translate into gains in better policy models, such as better instruction-following/math/code LLMs. The detailed comments are listed as follows.

### Detailed comments
  * [Clarity] Section 4: Pseudo-code of the proposed algorithm is recommended, at least for STaR-DPO.
  * [Clarity] Section 5: it is recommended to provide separated tables/figures to highlight each point in lines 296-302. Some additional rewards that explicitly answer the question can greatly improve clarity.
  * **[Clarity]** line 306-323: the analysis here can be organized and summarized in several important remarks, especially for those relevant to the questions in lines 296-302. Others can be moved to the appendix.
  * [Clarity] line 224: typo, there is a redundant comma ","
  * [Clarity] line 235": typo, "fine tuning" -> "fine-tuning"
  * [Clarity] line 471: "by a a" -> "by a"
  * [Clarity] line 234, 244, 258: As $\pi$ normally indicates policies in RL, it would be recommended to change this with another symbol.

  * [Quality, Clarity] Section 5, Figures 2 & 4: better include the specific number of accuracies in Tables.
  * [Quality, Clarity] Section 5: a table that explicitly states the gain of the utilized techniques would be strongly encouraged, including GenRM formulation, CoT with other models, CoT with Self-Taught Reasoning, DPO, and test-time computing.
  * **[Quality]** It is strongly encouraged to include results that show the improvements in policy models trained with STaR-DPO.
Yes
No ethics review needed.
1. Are STaR-SFT and STaR-DPO both trained on the same base model, rather than STaR-DPO being trained on STaR-SFT?
2. Where does the RLHF component fit into your framework? I see that you mention "incorporates human feedback in initial stages" in Section 7, but what exactly do you define as the initial stage, and how is human feedback utilized within this stage?
3. Could you provide a detailed summary of all the model settings, including the eight models depicted in Figure 2? Specifically, what is the base model, if it has been trained, what the training objective is, and what the dataset construction process entails? Thank you.
The paper presents an iterative framework called GenRM designed to train an LLM using self-generated reasoning traces. This framework is novel to some extent and has been validated on both in-distribution and out-of-distribution tasks through various ablation settings. It demonstrates scalability and good efficiency. The experimental results indicate that integrating chain-of-thought reasoning within preference modeling enhances the model's reasoning ability, contributing valuable insights for future studies in RLAIF and RLHF.
This paper tackles the problem that feedback from current RLAIF methods does not align with human preferences. It introduces a framework named GenRM, which integrates the strengths of RLHF and RLAIF. The authors utilize self-generated reasoning traces and iterative training loops, achieving results on in-distribution tasks that are comparable to Bradley-Terry models and outperforming them on out-of-distribution tasks. Additionally, GenRM surpasses the performance of using LLMs as judges for both types of tasks.
1. The evaluation of RM is inadequate, lacking experiments that assess agreement with human preferences. Additionally, only one out-of-distribution dataset, RewardBench, is utilized for evaluation.
2. The qualitative results presented in Figure 3 are unconvincing. Both STaR-DPO and LLM-as-a-judge use the same system prompt, as noted in Appendix A.1, which states that evaluations should consider factors such as helpfulness, relevance, accuracy, depth, creativity, and detail. However, your explanation for why STaR-DPO performs better than LLM-as-a-judge suggests that you prioritize instruction following over depth, helpfulness, and detail. While STaR-DPO includes an additional animal species, which is more detailed and helpful, it does not strictly follow the instructions. Furthermore, the system prompt is not always appropriate, as it can sometimes be contradictory.
3. The paper's title is somewhat unclear, making it difficult to discern your contribution. Additionally, some model settings are not sufficiently clear, such as the differences between GenRM and the STaR-SFT or STaR-DPO models.
Yes
No ethics review needed.
1. Mixing RLHF and RLAIF gets GenRM both their advantages, but also inherits both their disadvantages. Would you mind analyzing it a little more?

2. The foundation model in the experiment only uses Llama3.1 8B, which is not quite demonstrating the generality and robustness of the method. Experimenting on a more variety (e.g., mistral series) and size (e.g., 13B, 70B) of models would make this paper more solid.

3. Why does this method perform better on out-of-distribution (OOD) data? It's not very intuitive, not very easy to understand, and it would be better if there was more discussion and analysis of it.
This research topic is interesting.
This paper proposes a Generative Reward Modeling (GenRM) approach that combines the main ideas of RLHF and RLAIF, aiming at fine-tuning LLMs with self-generated reasoning traces to generate synthetic preference labels that match human preference judgments. Experiments on Llama 3.1 8B confirm the effectiveness of the proposed method.
1. Mixing RLHF and RLAIF gets GenRM both their advantages, but also inherits both their disadvantages, and the paper lacks a discussion of that. And the idea doesn't seem very novel.

2. The foundation model in the experiment only uses Llama3.1 8B, which is not quite demonstrating the generality and robustness of the method. Experimenting on a more variety (e.g., mistral series) and size (e.g., 13B, 70B) of models would make this paper more solid.

3. Why does this method perform better on out-of-distribution (OOD) data? It's not very intuitive, not very easy to understand, and it would be better if there was more discussion and analysis of it.
We develop an approach to fine-tune LLM judges to align with human judgments over preferences by bootstrapping their own reasoning.
@inproceedings{
anonymous2024generative,
title={Generative Reward Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=MwU2SGLKpS},
note={under review}
}
Reinforcement Learning from Human Feedback (RLHF) has greatly improved the performance of modern Large Language Models (LLMs). The RLHF process is resource-intensive and technically challenging, generally requiring a large collection of human preference labels over model-generated outputs. Reinforcement Learning from AI Feedback (RLAIF) addresses this data collection challenge by leveraging synthetic preferences generated by an LLM. However, recent work has shown that synthetic preferences labels may not align well with human preference judgments.

To address this, we propose a hybrid approach that unifies RLHF and RLAIF methodologies. We introduce GenRM, an iterative algorithm that trains an LLM on self-generated reasoning traces, leading to synthetic preference labels matching human preference judgments. Empirically, we show that zero-shot LLM-based judgments under-perform compared to Bradley-Terry reward models on in-distribution tasks (between 9-36%). In contrast, GenRM achieves in-distribution accuracy comparable to Bradley-Terry models, while significantly outperforming them on out-of-distribution tasks (between 10-45%). Moreover, GenRM surpasses the performance of using LLMs as judges on both in-distribution (by 9-31%) and out-of-distribution tasks (by 2- 6%). Our results show that combining the strengths of RLHF and RLAIF offers a promising approach for improving the quality of synthetic preference labels.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
RLHF reward model LLM Judge reasoning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d6fd8fcdfce08f6fddc3fdc0b9675e70e059bc0a.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Generative Reward Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission