Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
RLHF;DPO;large language model;alignment
Yes
No ethics review needed.
Using Llama-3-8B-Instruct it seems the current win-rate can be 20%+ (see Table 1 of https://arxiv.org/pdf/2405.00675). The authors reported ~7% win-rate of using Llama-3-8B. Could you explain the reason here?
Tackling the corruption, overoptimization, and verbosity issues of LLM preference alignment at the same time is not achieved before, and this work makes a reasonable contribution.

The paper is well written and easy to understand.
This paper proposed designs to mitigate Corrupted preference, reward Overoptimization, and bias towards Verbosity observed in RLHF and DPO. The authors use noise model to mitigate corruption, pessimistic and optimistic regularization to mitigate overoptimization, and length regularization to mitigate verbosity. They show a generalization error guarantee for the proposed methods, and experiments on Alpaca Eval 2.0 to support the proposed methods
The experimental results are not satisfactory.
Yes
No ethics review needed.
How fine should we set the grids and how sensitive are the results to the hyperparameters?
The paper is sound in it's theory part. The derivations of the proposed objective, along with the mathematical proofs, are sound. The objective is novel to the best of my knowledge.
This paper combines several techniques in addressing preference corruption, model optimization, and preference to verbosity into one objective, and shows the effectiveness of the objective both theoretically and empirically.
The objective itself, although reduced from minimax problem to just one minimization problem, has many hyperparameters in it. Authors mentioned that they used grid search over three out of the four hyperparameters in their experiment section. This is concerning due to the uncertainty and cost behind the hyperparameter search. Moreover, comparing all the methods against one much more powerful model (GPT4 in this case) on just one test set (AlpacaEval 2.0) isn't very convincing as the evidence to show the efficacy of the proposed objective. I would suggest the authors to add head-to-head win rates between pairs of the algorithms listed for comparison on more test sets.
Yes
No ethics review needed.
In Theorem 1, it should be $G _ {\mathcal{D}}$ rather than $\mathcal{G} _ {\mathcal{D}}$?
The literature review is detailed and the presentation is clear.
This paper studies how to deal with corrupted preferences, overoptimization and verbosity in RLHF and DPO. The authors propose RLHF-COV and DPO-COV algorithms that address these issues simultaneously in both offline and online settings. Their approach combines the existing techniques and they provide theoretical guarantees for their method. Experimental results demonstrate the efficacy of their approach compared to methods which only deal with part of the three issues.
The novelty of this paper is very limited and not sufficient enough for ICLR.
1. Algorithmically, the proposed algorithms are a simple combination of existing works. The anti-corruption component comes from (Bukharin et al., 2024), the anti-overoptimization component from (Liu et al., 2024c) and the anti-verbosity component comes from (Park et al., 2024). The online version adds (Bukharin et al., 2024) and (Park et al., 2024) on (Xie et al., 2024). I see no obvious novelty in the algorithm design.
2. Theoretically, the contribution is also relatively small. Most of the proof utilizes the techniques in (Bukharin et al., 2024), (Liu et al., 2024c) and (Xie et al., 2024). The authors claim that the novelty lies in the analysis order of the noise terms, which seem not significant to me.
Yes
No ethics review needed.
1. In Theorem 1, the choice of the parameter $\eta$ depends on the quantity of partial coverage coefficient $\mathcal{G}_{\mathcal{D}}$, which somehow seems impractical. How does the learner know the exact number of the partial coverage coefficient? Even though in practice the parameter is typically chosen via empirical search, but this theoretical choice still does not make sense to me.
2. Beyond the experiments in the paper, how to understand does the proposed algorithm indeed handle the challenge of corrupted data, reward over-optimization, and verbosity? In other words, is there additional experiment which supports that the superior performance of the new algorithm is exactly brought by the mitigation of these issues? For example for reward overoptimization, can the authors show that compared with the algorithm that does not consider this explicitly, the "reward" is indeed not over-optimized. Similar questions for corruption and verbosity.
3. What about the capabilities of the LLMs trained by DPO-COV in complex reasoning tasks, e.g., math and coding?
4. Typos: inconsistent notation for the partial coverage coefficient, see Section 3.3.
**Clarity and Quality:**
1. The paper is clearly written and the theoretical results are sound.

**Originality and Significance:** 
1. The paper is the first to consider the joint solution that mitigates all of data corruption, reward over-optimization, and verbosity while being theoretically supported.
2. New technical methods to handle corrupted data in the RLHF setup.
3. The experiments show the competitiveness of the proposed algorithm in aligning LLMs to some extent.
This paper studies how to fine-tune LLMs to align it with human preference data, aiming to address the change of corrupted preference, reward Overoptimization, and verbosity. Inspired by previous works on these challenges respectively, this work proposes new algorithms named RLHF-COV and DPO-COV that can simultaneously mitigate these three issues simultaneously. Theoretical results prove that under suitable assumptions,  the DPO-COV algorithm enjoys a converging length-regularized generalization error rates trained on corrupted data. 
Experimental results further demonstrate the effectiveness of the proposed algorithms.
From the reviewer's perspective, despite that the joint handling of these challenges is new in RLHF theory literature , the main idea of the algorithm design is not novel given existing works, i.e., the value-guided method to handle over-optimization/encourage exploration (see [1, 2, 3, 4, 5]) without the new components of handling corrupted data and verbosity, which makes the theoretical contributions of the paper weakened.  Also for this reason, the earlier work [1, 2] of this line in standard RL needs also to be mentioned in the paper for the completeness of literature review.

Another weakness is that the experiments are limited, which does not quite demonstrate the overall effectiveness of the proposed algorithm. E.g., see Question 3 below.

**References:** 

[1] Xie T, Cheng C A, Jiang N, et al. Bellman-consistent pessimism for offline reinforcement learning[J]. Advances in neural information processing systems, 2021, 34: 6683-6694.

[2] Liu Z, Lu M, Xiong W, et al. Maximize to explore: One objective function fusing estimation, planning, and exploration[J]. Advances in Neural Information Processing Systems, 2023, 36.

[3] Liu Z, Lu M, Zhang S, et al. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer[J]. arXiv preprint arXiv:2405.16436, 2024.

[4] Cen S, Mei J, Goshvadi K, et al. Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF[J]. arXiv preprint arXiv:2405.19320, 2024.

[5] Xie T, Foster D J, Krishnamurthy A, et al. Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF[J]. arXiv preprint arXiv:2405.21046, 2024.
We proposed the first RLHF and DPO algorithms that can simultaneously mitigate corruption, overoptimization and verbosity in large language model (LLM) alignment, with theoretical generalization error rates.
@inproceedings{
anonymous2024provably,
title={Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online {RLHF}/{DPO} Alignment},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=K2OWrXUVby},
note={under review}
}
Reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) are emerging and important techniques to align large language models (LLM) with human preference. However, the quality of RLHF and DPO training is seriously compromised by ***C**orrupted* preference, reward ***O**veroptimization*, and bias towards ***V**erbosity*. To our knowledge, most existing works tackle only one of these important issues, and the few other works require much computation to estimate multiple reward models and lack theoretical guarantee of generalization ability. In this work, we propose RLHF-**COV** and DPO-**COV** algorithms that can simultaneously mitigate these three issues, in both offline and online settings. This ability is theoretically demonstrated by obtaining length-regularized generalization error rates for our DPO-COV algorithms trained on corrupted data, which match the best-known rates for simpler cases with clean data and without length regularization. Moreover, our DPO-COV algorithm is simple to implement without reward estimation, and is proved to be equivalent to our RLHF-COV algorithm, which directly implies the equivalence between the vanilla RLHF and DPO algorithms.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
RLHF DPO large language model alignment
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/893016f4098d3d40cbf316d25bdaf5b36136e47d.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission