RRM: Robust Reward Model Training Mitigates Reward Hacking
Reward model;RLHF;Alignment
Yes
No ethics review needed.
The method proposed in this paper to 'disentangle prompt-driven preferences from prompt-independent artifacts' seems to only eliminate biases unrelated to the input while preserving input-related preferences. This can effectively reduce bias under the 'helpful' alignment objective. However, if our alignment goal is 'safety', the response might be prompt-independent, typically just refusing to reply. Yet humans may still have preferences for different ways of refusal. In this case, how can we disentangle prompt-independent preferences from prompt-independent artifacts?
This paper addresses an important issue in preference datasets: how to disentangle prompt-driven preferences from prompt-independent artifacts.
Traditional reward model (RM) training for large language models (LLMs) struggles to separate prompt-driven preferences from irrelevant artifacts like response length and format. This work introduces a causal framework and data augmentation technique to remove these artifacts, resulting in a robust reward model (RRM) that improves performance across benchmarks, enhancing accuracy and alignment scores significantly.
While the problem studied in this paper is significant, the proposed solution is relatively straightforward, focusing only on expanding the dataset. To remove prompt-independent artifacts, the dataset size needs to be several times larger, increasing the training cost. Additionally, as shown in Figure 5, the effect of this costly augmentation is not very significant—after inserting 30% artifacts, the RRM only decreases from the original RM’s 25% to 20%.

Regarding Table 1, the reasoning performance declines, and the authors explain this as due to math and coding tasks being less affected by non-contextual artifacts. This raises the question of whether the proposed method might have a negative impact on attributes unaffected by non-contextual artifacts.
Yes
NA
No ethics review needed.
- How did the authors tune the hyperparameters?
- What are the “three possible prompts” described in line 226?
- The authors study a very relevant and timely problem.
- The paper shows significant improvement in both reward modeling accuracy and in downstream policy performance.
- The authors do a really interesting study where they artificially add artifacts to the preference dataset, and find their method is more robust to it.
This paper aims to learn reward models that are unbiased by “artifacts” that come from human preference data: namely length biases and formatting. To do so, they propose to augment the RLHF preference dataset with preference pairs where one sample is in response to the prompt, and the other example is not in response to the prompt. They evaluate their learned reward model on reward bench and use it to improve downstream policy performance.
- The main weakness of this paper is that the experiments are not that comprehensive. Although the results are strong, the experiments are only conducted on one dataset, with one model, and with one random seed. Therefore we cannot tell if the results are statistically significant or generalizable.
- The baseline is pretty weak with regards to reward bench. RLHFFlow with Llama 3 8B gets 87.1 on reward bench, while the author’s baseline gets 80.
- The writing describing the methodology not that clear. I think more plainly describing the data augmentation would be better.
- The data filtering step used in their method hurts their experiments interpretability, as it may be the case that the sole reason RRM gets good performance is due to the data filtering step. They should include a baseline where the original dataset is filtered, and then a RM is trained based on that.
Yes
No ethics review needed.
- The same augmentation and disentangle technique can be applied to learn a artifact only model. Some of the artifacts would improve the quality of the answer, if there is a consensus preference on such artifacts in the labelers. Is there any proposal to combine the contextual & non-contextual models? Would adding these two reward models work, or do we need to take the min of these two models? Any suggestions from the authors?
- The causal framework has been effective and widely used in other problems, but this is the first time I see it applied in LLM reward modeling, and it very well suits the problem under consideration.
- The implementation is very straightforward, just augment the data, no empirical tricks.
This paper presents a method to enhance the robustness of the reward model for LLM alignment. More specifically, rewards models are trained based on the question and the human labelers' preference over the answer pairs. A commonly observed un-robust behavior of the reward model is that there are spurious correlations between the label and the artifacts in the answers which is not relevant to the question. The existence of such spurious correlations causes the reward model to favor certain types of artifacts, e.g. the length of the answer.

In previous works, some of the artifacts are specifically dealt with, like introducing a penalty for the length of the answer. However, the solutions lack a principle and can not handle other artifacts like the word preference and style of the answers. In this work, a systematic approach is proposed based on a causal framework, which is effective in debiasing the reward model from various artifacts.
- It seems suboptimal to me to train the reward model that is only dependent on the S, because the A part could contribute nontrivially to the reward function, in the case where both answers are equal in terms of the contextual quality, the labeler could prefer the answer which is organized as bullet points for readability.
- The explanation on why "reasoning" is performing worse under RRM is not quite convincing. Even if they are less affected by non-contextual artifacts, the augmented data should not have caused the degrade of performance either (as the augmented data are simpler pairs). I would suggest two other hypothesis to investigate:
  - Look into how "-Neutral" works on "Reasoning" as requiring 50/50 preference over some random pairs could be too strong.
  - Try to add back the "A only" reward model, as mentioned above the artifact could contribute nontrivially to the reward, removing it entirely could be detrimental especially for math and code, where a high quality answer should be good in formatting which is noncontextual.
Yes
No ethics review needed.
1. Explain why it is needed to add neutrals and set both non-contextual responses as tie in Section 3.2. What is the necessity of this step?
2. In Table 2, why the performance of RRM is not consistent in two chat datasets such as Chat and Chat Hard? In Chat dataset, RRM outperforms RM while in Chat Hard dataset, RRM is even worse.
Reward hacking is an important problem in LLM alignment. This paper introduces causal inference framework in reward modeling to mitigate reward hacking, which offers an interesting perspective. This idea makes a lot of sense since the essence of reward hacking mitigation is to find out the irrelevant artifacts that have no causal relationship with the preference label, and causal inference algorithms just offer a useful tool. The idea is clearly presented and illustrated. The experiment results verify the effectiveness of proposed method.
This paper proposes a robust reward model (RRM) training method to mitigatge reward hacking by utilizing causal inference framework. A causal graph for human preference modeling is introduced to enable the model to distinguish contextual preference signals and context-free artifacts. The training data is augmented by re-organizing the (prompt, positive, negative) triplet guided by the causal inference framework. Experiments show that the proposed approach can filter out undesirable artifacts in reward model training and yields a more robust reward model. Specifically, the reward model is trained on Gemma-2-9b-it, and RRM improves RM by an absolute 3.54% accuracy gain on Reward-Bench. Experiments also demonstrate the policy induced by RRM outperforms RM and ODIN in MT-Bench and AlpacaEval-2 benchmark. Analysis show that length artifact and deliberately designed artifact can be eliminated.
The experiments are insufficient. First, the experiments in this paper only cover two possible artifacts, namely length and a deliberately designed phrase, which is not compatible with author's claim that this method should be capable of handling "all potential exploitation patterns". More patterns such as format and emojis should be investigated in the experiments. Second, the reward model is trained only using Gemma-2-9b-it. More experiments should be conducted on different model types such as llama, and larger model sizes such as 57B and 72B. It is in doubt that when the model size increases, the reward hacking problem can be naturally mitigated and the effectiveness brought by this method may diminish. Third, in terms of policy evaluation, this method is only verified in DPO training whereas various other methods should also be considered such as PPO, whose performance relies heavily on reward model quality. Last, the author is suggested to add more baselines in experiment, such as reward model ensemble which can also alleviate reward hacking. 

The data augmentation method is trivial. The new combination is derived by randomly selecting answers from other prompts. Since this negative sample is not related to the prompt with high probability, it is too easy for the model to distinguish the positive and negative answers. The author is suggested to incorporate some hard negatives to make the training more robust.
The first paper to point out the reason for reward hacking from a causal perspective.
@inproceedings{
anonymous2024rrm,
title={{RRM}:  Robust Reward Model Training Mitigates Reward Hacking},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=88AS5MQnmC},
note={under review}
}
Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such as response length and format. In this work, we expose a fundamental limitation of current RM training methods, where RMs fail to effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we introduce a causal framework that learns preferences independent of these artifacts and propose a novel data augmentation technique designed to eliminate them.  Extensive experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model (RRM). Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on Reward-Bench, increasing accuracy from 80.61% to 84.15%. Additionally, we train two DPO policies using both the RM and RRM, demonstrating that the RRM significantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Reward model RLHF Alignment
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9acb61eed6b70d4d62a8e4face83d38d478c8734.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
RRM:  Robust Reward Model Training Mitigates Reward Hacking
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission