Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision
unreliable human supervision;language model post-training;scalable oversight
Yes
No ethics review needed.
1. I was a bit unclear on how the unreliable DPO dataset is constructed: so, you generate responses from a model checkpoint, and train a classifier to pick the original ground truth label instead of the model response, and then use this classifier to pick between two samples from the same model checkpoint? Or is the classifier the same across rounds?
- I think the experimental setup is good, looking at a variety of models and tasks, and the human labeling experiment is also interesting.
- The analysis of why DPO does not perform well with unreliable data is interesting, and provides reasonable evidence that DPO can fit too hard to the unreliable preferences.
- The performance of the proposed method seems robust, with it being tested both in multiple settings and with (time-constrained) human annotators. The performance gains seem robust across the tasks and models tested.
The authors examine how to improve model performance when SFT and preference data is unreliable (contains some proportion of mistakes). They find that doing SFT training still results in improvements with unreliable data, but DPO does not improve. They propose a new method, iterative label refinement, which involves iteratively training models on data subsets, and then using the models to relabel the SFT data (with an unreliable supervisor deciding if the new label should be accepted), and retraining. The proposed method outperforms DPO when applied to SFT models both in an artificial setup (using small models to train larger ones) and a time-constrained human-labeller setup.
- Missing Baseline: it would be good to compare against just doing SFT on the chosen sample or unreliable DPO pairs (my understanding is that this is not the same as training only on the model’s proposals because there is an additional trained classifier that may pick the right answer, albeit unreliably).
- I wonder if the comparison between DPO and SFT/ILR methods are entirely fair since the DPO method doesn’t use the original ground truth answer, just two model generations. Could the DPO results perform better if the original ground truth label was also provided as an option when picking chosen and rejected pairs (as it is for the ILR approach)? Would it be possible to run the same ILR algorithm outlined in section 5.1, but rather than replace labels, just use the accepted/rejected proposals to construct chosen/rejected pairs instead?
- Figure 6b is quite hard to parse, as the label box overlaps with a large chunk of the figure, and the differences between round 1 and 2 are somewhat small.
- Did the authors try longer DPO training with a high beta? They clearly show that using a larger beta with unreliable feedback can improve a bit, I wonder if training for longer with higher beta results in further improvements with DPO or if it then starts to over-optimize as well (potentially a different way that the issues with DPO could be alleviated).

Overall, I think the paper presents an interesting issue (that DPO doesn't seem to work well with unreliable label) and proposes an interesting alternative method that appears to work well in across a few models and tasks, although I think the comparisons between DPO and ILR could be made a bit tighter.
Yes
No ethics review needed.
1. How does ILR compare to having annotators review each sample in the SFT data to remove low-quality ones? This approach seems less expensive, as it avoids generating new responses with the SFT model and lets the annotators avoid pairwise comparisons.

2. How frequently are the model-generated responses considered better than the SFT data depending on the size of the model?

3. Typos: “Ground Truth” in Figure 5.
1. **Motivation.** Safe post-training of language models with noisy fine-tuning data is a highly practical problem with a lot of relevant research done to address the issue.

2. **Experiments and analyses.** The paper conducts a range of experiments, simulating unreliable supervision using small LMs and time-constrained human evaluators to evaluate ILR under multiple settings.

3. **Performance.** ILR demonstrates improvements over DPO under the unreliable supervision settings considered in the work.
This paper proposes iterative label refinement (ILR) as an alternative to reinforcement learning from human feedback (RLHF) for language model post-training under unreliable supervision. The authors argue that in settings where human supervision is unreliable, methods such as direct preference optimization (DPO) become less effective. Instead, ILR iteratively refines the supervised fine-tuning (SFT) dataset by replacing unreliable human demonstrations with better model-generated responses, thus improving the model’s training data. Experiments show that ILR outperforms DPO in tasks such as math, coding, and safe instruction-following, especially when human supervision is noisy or unreliable.
1. **Limited evaluation of alternatives.** Several approaches exist for handling noisy feedback data, including conservative DPO [1], which combines regular DPO with label smoothing, IPO [2], MMPO [3], and filtering noisy labels, among others. However, empirical evaluations of some of these methods in the unreliable supervision setting considered are missing.

2. **Unreliable supervision not well-defined.** The concept of unreliability is not quantitatively well-defined. For example, it is unclear whether a small LM trained as a classification model actually provides unreliable comparison feedback, and if so, to what extent the labels are considered inaccurate. Relatedly, it is unclear how unreliable the “time-constrained” human supervision really is.

3. **Questionable assumptions.** For the ILR framework to be effective, it seems that the SFT model needs to generate responses that are often better than those in the SFT dataset, and the annotator must be able to identify these better responses. While the latter assumption is discussed in the paper, the former depends on the SFT model being sufficiently capable and not overfitted to the SFT data, so that it can generate responses than the SFT data at least some of the time. This depends on factors such as the size of the SFT dataset, the proportion of unreliable data, and the size of the SFT model, raising questions about how generally applicable ILR is.

---
[1] Mitchell. "A note on DPO with noisy preferences & relationship to IPO." (2023). \
[2] Azar et al. "A general theoretical paradigm to understand learning from human preferences." (2024). \
[3] Kim et al. “Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback.” (2024).
Yes
No ethics review needed.
- How does ILR compare to other RLHF methods like PPO? Would similar overoptimization issues arise in other preference optimization methods, or are they more resistant to unreliable supervision?
- Could ILR be extended or adapted to address tasks that involve more subjective or open-ended feedback, such as creative writing or opinion-based tasks?
- Does ILR introduce any risks of bias through its iterative process, particularly as models continue to generate and refine their own data? How would you address or mitigate such risks?
- Novelty and Significance: ILR introduces a fresh approach to addressing the key challenge of unreliable human supervision in language model post-training. This is quite crucial as RLHF is becoming increasingly important for scaling LMs in practical applications.
- Comprehensive Evaluation: The experiments cover different tasks including math, coding, safe instruction-following and use both LM and real human feedback, ensuring the findings are well-supported. The robustness of ILR under different conditions and tasks is thoroughly tested and clearly explained.
- Clear Motivation: The paper presents a strong investigation and reasoning for why RLHF, especially DPO, struggles under weak supervision. The overoptimization problem is well-identified and investigated, which motivates the need for a method like ILR, which focuses on refining the training data rather than the model directly.
- Effective Use of Feedback: The authors highlight the importance of directing comparison feedback towards improving the dataset, not just the model. This is a significant insight and could influence future work in RLHF, encouraging researchers to rethink more about how to utilize human feedback in post-training.
This paper proposes a novel approach, Iterative Label Refinement (ILR), aimed at improving language model (LM) post-training when demonstration and human supervision is unreliable. The authors identify that RLHF (specifically DPO) struggles under unreliable feedback due to overoptimization, and they propose ILR as an alternative that improves the supervised fine-tuning (SFT) dataset itself. The approach replaces low-quality human demonstrations with (small) model-generated alternatives based on comparison feedback, which is then used to retrain the model. The paper provides evidence that ILR outperforms DPO in various tasks, including math, coding and safe instruction-following, under both LM-simulated and human supervision settings.

Comments:
- Scalability and Computational Cost: The paper could include a more detailed discussion on the computational trade-offs involved in using ILR versus DPO. Given that ILR requires multiple rounds of refinement, it would be useful to know how this affects training time and computational resources.
- Clarification on Theoretical Boundaries: Theoretical analysis could be expanded to explore the long-term behavior of ILR. For example, will the refinement process converge effectively without introducing new errors or biases into the dataset over time?
- Additional Visualizations: It would be helpful to see additional visualizations that show the progression of ILR versus DPO over multiple training rounds, especially in terms of how the refined dataset improves in quality and how the model’s performance evolves.
- Task Extension: Future work could explore extending ILR to more diverse task types, including more subjective ones or tasks where human demonstrations are more error-prone (e.g., complex visual tasks or tasks requiring nuanced judgment).
- Potential ensembling effect that may explain performance gain: 5.1 describes training one model on each half of the SFT data, then using these models to cross-label during ILR. There does not seem to be an ablation either. I think a fair comparison would be against a two-model ensemble, as opposed to a single model baseline. 
- Limited Comparison with other RLHF Methods: While the paper focuses on comparing ILR to DPO, it does not address how ILR might perform compared to other RLHF algorithms, like PPO. Including more comparisons could make the paper’s conclusions more robust and introduce a broader impact.
- Computational Cost and Scalability: The ILR approach, which requires iterative refinement and cross-evaluation of datasets, which means  both SFT model and DPO needs to be retrained for every iteration. This might introduce higher computational complexity compared to DPO or traditional RLHF methods. The paper could benefit from a more detailed discussion of the trade-offs between performance improvement and computational cost.
- Task Diversity: While the paper tests ILR on math, coding, and safe instruction-following, it would be beneficial to see it applied to a broader range of tasks, particularly more complex or open-ended ones where human feedback is highly subjective or unreliable (e.g., creative writing or subjective question-answering or toxicity).
- Theoretical Depth: The theoretical tracking of ILR is solid, but the paper could elaborate more on the theoretical limits of ILR. In particular, the long-term stability of ILR, and whether it risks introducing new forms of bias through the iterative refinement process, could be explored.
Yes
No ethics review needed.
- Is it specifically necessary to do 1:1 replacement of training data points with new synthetic ones? Does it make sense to decouple the number of data points removed due to low quality, and the number of new synthetic data points accepted? How much of the effectiveness is due to the filtering vs the addition of synthetic data points?
- Demonstrating the failure of DPO via overoptimization in the presence of unreliable feedback is important, given the prevalence of unreliable feedback.
- The introduced method is simple and performs clearly better.
- Validation on human setting increases confidence that the results on synthetic setting are reasonable
The paper shows that DPO is ineffective in the presence of unreliable feedback. The paper introduces a new method that replaces data samples in the SFT demonstrations dataset with synthetic demonstrations if the reward model prefers the synthetic demonstration. This process is iterated.
- Doesn't compare against PPO
We find SFT+DPO breaks down under unreliable human supervision and show that it is better to direct unreliable feedback towards improving the training *data* rather than continually training the *model*.
@inproceedings{
anonymous2024iterative,
title={Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=q5EZ7gKcnW},
note={under review}
}
Language model (LM) post-training relies on two stages of human supervision: task demonstrations for supervised finetuning (SFT), followed by preference comparisons for reinforcement learning from human feedback (RLHF) via algorithms like proximal preference optimization (PPO) or direct preference optimization (DPO). As LMs become more capable, the tasks they are given become harder to supervise. Will post-training remain effective under unreliable supervision? To test this, we simulate unreliable demonstrations and comparison feedback using small LMs and time-constrained humans. We find that in the presence of unreliable supervision, SFT still retains some effectiveness, but DPO fails to improve the model beyond SFT. To address this, we propose iterative label refinement (ILR) as a replacement for RLHF with unreliable supervision. ILR directly improves the SFT data by using comparison feedback to decide whether human demonstrations should be replaced by model-generated alternatives, then retrains the model via SFT on the updated data. SFT+ILR outperforms SFT+DPO on several tasks with LM-simulated unreliable supervision (math, coding, safe instruction-following), with results further verified by human experiments on instruction-following. Our findings suggest that as LMs take on complex tasks where human supervision is unreliable, RLHF may no longer be the best use of human comparison feedback; instead, it is better to direct feedback towards improving the training data rather than continually training the model.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
unreliable human supervision language model post-training scalable oversight
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/550fc01e99f66cb12c33a39752a9ed0a104128fd.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission