Length Desensitization in Direct Preference Optimization
large language model;reinforcement learning from human feedback;preference optimization
Yes
No ethics review needed.
Are lines 808-810 correct? Should both $\frac{\partial \mathcal{L}{DPO}(\chi_1; \chi_2)}{\partial \chi_1}$ and $\frac{\partial \mathcal{L}{DPO}(\chi_1; \chi_2)}{\partial \chi_2}$ increase when $\chi_2$ decreases? Should $\frac{\partial \mathcal{L}{DPO}(\chi_1; \chi_2)}{\partial \chi_1}$ decrease and $\frac{\partial \mathcal{L}{DPO}(\chi_1; \chi_2)}{\partial \chi_2}$ increase when $\chi_1$ decreases?
- The logic flow is clear.
- The authors identify the reason for length sensitivity in the DPO algorithm.
- Based on their analysis, the authors propose the LD-DPO algorithm, which performs well in terms of length control and alignment.
- Experiments with three models across two datasets demonstrate the generalizability of LD-DPO.
- LD-DPO is a simple yet effective method.
The authors demonstrate the existence of length sensitivity in the DPO algorithm and analyze this issue theoretically. They propose the LD-DPO algorithm to address this sensitivity. Experiments on three open-source language models show the effectiveness of LD-DPO.
- Although the authors claim to have theoretically proven the sensitivity of DPO to length, the description is still insufficiently rigorous. For example, from Equation 4 to Equation 5, the expectation sign is omitted without further explanation.
- The explanation from lines 211 to 215 is vague and overly intuitive, especially regarding the relationship between length and probability.
- In Equation 7, the authors take the absolute value of the ratio of two Jacobians, a less clear motivation that complicates the analysis.
- The term "probability bias" in line 416 is unclear.
- The effect of the hyperparameter \alpha remains unclear. In lines 497-500, the authors state, "Conversely, when \alpha is too small, excessive length decoupling leads to a loss of human-like preferences in the text, thereby reducing the optimization effectiveness." Figure 4 shows that different selections for \alpha lead to varied effects across different experiments. In AlpacaEval 2, choosing either 1 or 0 results in similar LC-win rates; however, in MT-Bench, choosing 0 (i.e., strong desensitization to length) leads to significantly lower performance compared to the original DPO. The authors do not provide further explanation for this.
    
- The design of the hyperparameter \alpha has a relatively strong impact on LD-DPO performance, as reflected by the results in Figure 4.
    
- A minor issue: the color differentiation of the lines within the same subfigure in Figure 3 makes it difficult for readers to distinguish them.
Yes
There are no apparent violations of the code; however, it is worth noting that this paper is available on Arxiv.
No ethics review needed.
See weekness.
1. Addresses a popular issue of DPO's sensitivity to length.
2. Good presentation and easy to read.
3. Good empirical performance.
This paper addresses the issue that DPO tends to over-optimize for verbosity and proposes a method to desensitize DPO to data length. Evaluations show the proposed LD-DPO algorithm consistently outperforms existing algorithms with less average tokens than DPO.
1. Although theoretical insights on why DPO favors longer response is provided, the proposed LD-DPO is a heuristic method. It directly cuts off the importance of the tokens exceeding the public length. It is disappointing to see the solution to the well-formulated length sensitivity problem is just a code-level heuristic method. Why not try to modify the DPO loss for a loss landscape[1] that is length-desensitized?
2. The description for eq.(10) is not rigorous. Why $p^\alpha$ is "human-like preferences" and  $p^{1-\alpha}$ is "verbosity preference"? Just by definition?

Overall, it is just another LLM paper following DPO. There's nothing particularly exciting, and there isn't much to comment on. While it does not present anything particularly novel or insightful, it is a well-structured paper with a thorough evaluation.

[1] Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
Yes
No ethics review needed.
1. The analysis of length bias in DPO is interesting. However, it seems this analysis only applies to DPO-based methods. Since RLHF-based methods also tend to increase generation length after training, how is it different from the length bias in DPO? Is it possible to apply your analysis to length bias in RLHF?

2. In LD-DPO, probabilities of excessively long portions (response after $l_p$) are decayed by $\alpha$ to close the gaps between the magnitudes of chosen and rejected responses' probabilities, which inevitbaly introduces information losses. And you also admitted that "additional text can convey more human-like preferences"; "$\alpha$ is actually the result of a compromise to achieve desensitization of DPO based on model capabilities and to prevent the loss of human-like preferences"

Therefore, the decrease of generation length is reasonable, but it is weird that LD-DPO consistently demonstrate better scores than DPO even with this information loss. Is there any reasonable explanation?

3. LD-DPO seems to be very sensitive to hyperparameter $\alpha$ (the values are different for all models in your experiments). Is there any way to improve it?

To be honest, I'm currently undecided between 5 and 6. Considering the issues mentioned above, I'll give this paper a 5 for now and reconsider it upon seeing authors' feedback.
1. Length bias widely exists in a wide range of LLM alignment methods and should be disentangled from real human preferences,

2. Motivation of LD-DPO is clearly expressed by the theoretical analysis.

3. Proposed method is evaluated on multiple benchmarks and base models.
This paper proposes LD-DPO, a DPO-based method to desensitize LLMs from length bias during preference optimization. The authors first gave analysis on why DPO methods are sensitive to the length bias in the response pairs. Based on the analysis, LD-DPO decays  the probability of the excessively long  part of responses to attenuate the sensitivity of DPO to longer responses. Results on several benchmarks demonstrate that compared to DPO, LD-DPO sucessfully reduces the length of generated responses after preference optimization.
1. Redundant symbol definitions. I do not think the definitions of $\mathcal{X}_1$, $\mathcal{X}_2$,$\mathcal{K}_1$,$\mathcal{K}_2$ are necessary. It just adds to the diffculty to understanding.

2. The colors in the fig. 3 are difficult to distinguish. And this figure is also a bit hard to comprehend.

3.Some spelling and grammartical mistakes, e.g. "Length **Desentsitization** of DPO, termed LD-DPO"
Propose that Direct Preference Optimization(DPO)  has length sensitivity and analyze it theoretically, and design the corresponding length desensitization algorithm: LD-DPO.
@inproceedings{
anonymous2024length,
title={Length Desensitization in Direct Preference Optimization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=CuwjD3cazX},
note={under review}
}
Direct Preference Optimization (DPO) is widely utilized in the Reinforcement Learning from Human Feedback (RLHF) phase to align Large Language Models (LLMs) with human preferences, thereby enhancing both their harmlessness and efficacy.  However, it has been observed that DPO tends to over-optimize for verbosity, which can detrimentally affect both performance and user experience. In this paper, we conduct an in-depth theoretical analysis of DPO's optimization objective and reveal a strong correlation between its implicit reward and data length. This correlation misguides the optimization direction, resulting in length sensitivity during the DPO training and leading to verbosity. To address this issue, we propose a length-desensitization improvement method for DPO, termed LD-DPO. The proposed method aims to desensitize DPO to data length by decoupling explicit length preference, which is relatively insignificant, from the other implicit preferences, thereby enabling more effective learning of the intrinsic preferences. We utilized two settings (Base and Instruct) of Llama2-13B, Llama3-8B, and Qwen2-7B for experimental validation on various benchmarks including MT-Bench and AlpacaEval 2. The experimental results indicate that LD-DPO consistently outperforms DPO and other baseline methods, achieving more concise responses with a 10-40\% reduction in length compared to DPO. We conducted in-depth experimental analyses to demonstrate that LD-DPO can indeed achieve length desensitization and align the model more closely with human-like preferences.
”Brevity is the Soul of Wit.''—William Shakespeare
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language model reinforcement learning from human feedback preference optimization
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2673bc3e4491b408c54c7b75655c0e916479f665.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e5323351579d45e96a25747422d860deb27930de.zip
Length Desensitization in Direct Preference Optimization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission