Robust RLHF with Noisy Rewards
Alignment;LLM;RLHF
Yes
No ethics review needed.
1. Could the approach be combined with other regularization techniques to further enhance stability?
2. Are there plans to address other types of noise beyond reward model noise to make the framework more comprehensive?
1. Clear motivation: The paper emphasizes a common issue (noisy reward signals) in RLHF and offers a practical solution.
2. Theoretical backing: The authors justify the contrastive reward mechanism with robust mathematical grounding.
3. Empirical support: Extensive experiments with both human and automated evaluation show meaningful performance gains.
This paper, titled "Robust RLHF with Noisy Rewards," addresses a critical challenge in reinforcement learning from human feedback (RLHF): the vulnerability of reward models to noise. The authors propose a novel approach using contrastive rewards to enhance RLHF robustness by penalizing uncertainty, promoting stability, and improving learning focus. The paper includes theoretical justifications, as well as extensive empirical results that demonstrate the effectiveness of the proposed method, showing consistent improvements over established baselines. This paper’s contribution could have a meaningful impact on real-world RLHF applications where noisy feedback is prevalent.
1. Limited baseline comparisons: The paper benchmarks against some strong baselines but lacks comparisons to more recent methods that handle reward noise differently.
2. Applicability to real-world scenarios: The method assumes a binary reward noise model, which may oversimplify complex noise distributions in actual data.
Yes
No ethics review needed.
How different is your method from existing methods that subtract the average reward to the reward?
The paper rediscovers the use of a baseline in policy gradient methods.
The paper proposes to use offline generated rewards to calibrate the reward at training time in the RLHF pipeline.
This method is similar to RLOO or VinePPO in the one turn case. The only slight difference is that the calibrated reward is produced offline.
However, it seems that this offline calibration has already been published in this study:
https://arxiv.org/pdf/2410.01679 (see section 4.1.1 called Calibrated Regularized Policy Gradient)
I think the work has been produced concurrently and therefore there is no added scientific contribution.
Yes
No ethics review needed.
I like the paper, and I think it is quite complete; I am only a bit sceptical about the novelty of this work, hence a lower score. As pointed out by the authors, it would be interesting to see how well the method performs over multiple rounds, recalibrating the base rewards after each iteration.
The paper is well-written and easy to follow. The experimental methodology is rigorous, with many ablation studies, and the authors even employed human annotators. The method is intuitive and easy to implement.
The authors address the noise in the reward model by introducing a contrastive reward function that compares the reward of a generated response to the average reward of baseline responses generated by a previously trained model. They show this simple modification outperforms the vanilla PPO algorithm by 20%.
The idea of subtracting the value of a reference answer to reduce reward model variance between various prompts [has been there at least since January 2023.](https://wandb.ai/carperai/summarize_RLHF/reports/Implementing-RLHF-Learning-to-Summarize-with-trlX--VmlldzozMzAwODM2#gotcha-1:-normalization) I am not aware of any research papers on this topic, but it makes me question the impact of this paper if the practitioners already use this method.
Yes
No ethics review needed.
* Please see questions in weakness
* Have the authors tried aligning models with proposed algorithms and test the resulting models’ performance on benchmarks like AlpacaEval?
The paper paid attention to a direction of the existing alignment research which is less explored, however quite natural: the reward model can be (actually indeed) inaccurate. The authors thus proposed subtracting the baseline reward function from usual reward utilized in PPO algorithm for RLHF.
The paper proposed a contrastive reward method that intends to address possible noisy rewards from inaccurate reward models. The authors justify the effectiveness of their methods by empirical results on models alignment.
**Writing**: The overall writing has a significant space to improve. There are confusing descriptions and notations, making some parts of the paper hard to follow and understand. For example, Section 2.1, which should be the most important section to introduce the notion of Robust RLHF, is very short and not elaborated enough to clearly explain the motivations and formulation details of this paper. In addition:
* Table 1 is rather confusing. Is this error rate the dataset’s error rate or the error rate of the reward model?
* In Equation (4) and (5), there are many new pop-up elements: what is the second expectation for in (4) and (5)? I think in (4) the second expectation should be $\Psi$ right? What is $\mathcal{D}_{RL}$ in (5)? Also, are there any reasons why authors do not consider any type of regularization, like KL regularization in RLHF, but include this regularization back in the practical algorithm design?
* On line 123-125: The confusion function needs to be formally defined. The results for analysis are also limited to the discrete-value setting which seems to be rather restricted and undermine the generalizability of the theorem results.
* Robustness in 129-131: Here the definition of such robustness is unclear and confusing to me. Since the paper is developing around robustness, a readily defined definition shall be necessary. 

**Novelty**: I am also confused about the novelty of this paper, as the proposed mechanism for increasing robustness and its relation to resulting algorithms is still unclear after my reading. The final proposed improvement over PPO baseline looks to be subtracting a baseline function from reward, and this is a standard practice even in RLHF original paper, like normalization to mean 0 for reward models. It is also a widely adopted technique for variance reduction, as the author also mentioned on line 196-198. It will be beneficial to further explain the version and details of PPO the paper utilized for baseline comparison. Any further evidence of noisy reward and how algorithms behave with respect to the noisy level of the reward models shall be a better way to showcase the effectiveness.

**Possible missing references**: are there any connections between the approach in this paper and other papers like MaxMinRLHF[1] or other robust preference optimization papers? There lacks a discussion of existing literature on this line.

[1] MaxMinRLHF: https://arxiv.org/pdf/2402.08925
We introduce a new robust RLHF algorithms that explicitly acknowledge the potential noise in a reward model
@inproceedings{
anonymous2024robust,
title={Robust {RLHF} with Noisy Rewards},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Cfbr56K4gp},
note={under review}
}
Reinforcement learning from human feedback (RLHF) is the mainstream paradigm to align large language models (LLMs) with human preferences. Yet existing RLHF heavily relies on accurate and informative reward models, which are vulnerable and sensitive to noise from various sources, e.g. human labeling errors, making the pipeline fragile. In this work, we formulate the problem of performing robust RLHF with noisy reward models. Our goal is to design robust RLHF algorithms that explicitly acknowledge the potential noise in a reward model. Our first contribution is an analysis that revealed a certain transformation of the preference function improves its robustness to noise in the reward function. This observation leads to a new reward function design that involves two steps: (1) an offline sampling step to obtain responses to prompts that serve as baseline calculation and (2) a contrastive reward calculated using the baseline responses in Proximal Policy Optimization (PPO). We show that our suggested rewards enable the LLM to penalize reward uncertainty, improve robustness, encourage improvement over baselines, calibrate according to task difficulty, and reduce variance in PPO. We also empirically demonstrate contrastive reward can improve RLHF substantially, evaluated by both GPTs and humans, and it consistently outperforms strong baselines.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Alignment LLM RLHF
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/165932aa11b03e77d39b1b5c33fd3f910417a8d7.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Robust RLHF with Noisy Rewards
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission