Policy Filtration in RLHF to Fine-Tune LLM for Code Generation
LLM;RLHF
Yes
No ethics review needed.
I strongly recommend the reviewer to significantly reduce the unnecessary discussions and elaborations in section 3/4 and focus their attention on 
1. analysis of the impact of the algorithm. In particular, insights into what data are selected and discarded as a result of the proposed algorithm, and their consequent impacts on the RLHF procedure, will be highly useful for RLHF practitioners.
2. generalization of the proposed algorithm to other problem domains (e.g. other tasks with objective ground truth such as mathematical reasoning, and tasks where such objective ground truth do not exist)
1. While data selection during reward modeling has previously been discussed, the paper addresses a novel issue of post-hoc data selection during PPO training after RM has been trained. The authors found simple yet effective heuristic of choosing samples that are unlikely to have incorrect reward signal.
2. The empirical results shown in the paper, given the simplicity of the data selection method, is impressive. I could see this kind of method be widely adopted in RL.
The paper introduces a new way of filtering model generated samples during the PPO training stage that the authors call Policy Filtration PPO (PF-PPO). The method changes the sampling distribution of the target LLM by post-hoc selecting the generated samples using the reward model, via a heuristic that filters for generated samples that are likely to have reward values that correlate highly (measured by R^2) with ground truth accuracy. In particular, the authors found that selecting the generated samples with high/low valued rewards improve the final target LLM’s code generation performance. The authors show that PF-PPO is able to achieve SOTA result on code generation in 7B class of open source LLM models.
1. The paper broadly suffers from writing clarity, and has the tendency to overcomplicate concepts that appear to have simpler alternative explanations.  For example, the formulation of policy filtration could be drastically simplified. If I understand the method correctly, the algorithm is in essence sampling model generated responses by some heuristics which maximizes the R^2 between reward value and ground truth accuracy. The discussion around parameterized sampling, straight through estimator, seem redundant and was confusing. It is not clear how the “actual score” for code generation task was obtained and the appendix section seems straight from the template.
2. Some key discussion seem to be missing that limit the general applicability of the paper. The paper discusses application in code generation domain, where there exist oracle reward modeling (via code execution). As such, using R^2 to find the samples with the least RM noise seem redundant (why not just use the oracle RM)? Where this question seems much more pertinent is in tasks that don't have objective ground truths? What if the "actual score" is noisy?
Yes
No ethics review needed.
Given that PF-PPO’s improvements are incremental, can the authors provide more justification for why this method is advantageous over simpler techniques like weighted sampling?

Could the authors share examples of generated code to illustrate how the proposed filtering affects code quality, readability, or maintainability beyond accuracy metrics?

How does PF-PPO’s computational complexity compare to existing methods when deployed in real-world environments, particularly for larger models and datasets?
The paper addresses the challenge of reward model noise in RLHF, a known issue in the field. By focusing on reward reliability and noise reduction, it touches on a potentially impactful area for RL-based code generation.
The paper "Policy Filtration in RLHF to Fine-Tune LLM for Code Generation" proposes a filtering strategy for Reinforcement Learning from Human Feedback (RLHF) to improve code generation in large language models (LLMs). The authors introduce Policy Filtration for Proximal Policy Optimization (PF-PPO), a method that aims to mitigate noise from inaccurate reward models by filtering unreliable samples during training. The method is benchmarked on the HumanEval, MBPP, and a newly created LeetCode Contest dataset, demonstrating some performance gains. The paper claims that filtering improves reward model reliability, leading to better overall policy outcomes.
Lack of Novelty in Approach: The proposed method, PF-PPO, builds upon existing Proximal Policy Optimization (PPO) by simply filtering samples. This approach does not introduce substantial innovation, as similar filtering and sampling techniques are well-explored in RL literature. The study could benefit from a deeper analysis of why existing filtering methods are inadequate for RLHF in code generation.

Limited Practical Impact: Although the paper demonstrates slight improvements on specific benchmarks, it does not provide a strong case for the broader utility of PF-PPO. The marginal performance gains (e.g., 0.7% to 10% across different benchmarks) may not justify the added complexity. Furthermore, focusing primarily on accuracy metrics without detailed qualitative analysis limits the assessment of how this approach improves code generation's real-world usability.

Over-reliance on Benchmark Metrics: The paper heavily emphasizes quantitative metrics without exploring the implications of these improvements on code quality or robustness. A qualitative analysis of code examples generated by PF-PPO would help clarify its value, especially given that code generation tasks require more nuanced evaluation beyond pass/fail metrics.

Omission of Real-World Considerations: The filtering strategy relies on preset thresholds and assumptions about reward distribution, which may not generalize to more diverse or complex RLHF tasks. Additionally, the computational overhead and feasibility of implementing this method in large-scale production are not discussed, raising concerns about practical applicability.
Yes
No ethics review needed.
Please see the questions above.
The idea of dropping samples with unreliable reward predictions during RLHF is novel.
The paper proposed to filter reliable RLHF samples to mitigate inaccurate reward model predictions
1. [Soundness] The proposed method was largely based on an observed property: "the reward model is more reliable for the responses assigned with high/low rewards". However, this argument is not well-defined, and the property works in what scope is not discussed. Specifically,
  - What do high/low rewards exactly refer to, esp. across different benchmarks and tasks? 
  - Is this property universal for different reward models across different tasks? The paper do not clearly state when this property is applicable, and there is no sufficient quantifiable analysis on this property. Therefore the universality of this property remains unclear, and I question about it. In particular, does this property persists across benchmarks and tasks? The proposed method appears general, but is only evaluated on code generation, what about other tasks?
  - I don't understand how Figure 1 supports the suggested property. I notice Figure 1 show the comparisons between reward estimations and labels, but involve very small number of prompts (164) in only one benchmark (HumanEval). The plots are noisy, and the paper do not clearly explain how such plot support the suggested property. 
  
2. [Contribution] The paper proposed three heuristic-based designs of policy filtration. 
- The design is ad-hoc, and the paper lacks clear explanations on why specific design work better or worse. 
- Ad-hoc designs limit the contribution of the proposed method. Specifically, if I focus on performance on other aspects, e.g., math reasoning, helpfulness, how to design the policy filtration weight vectors? In the current state, the contribution is limited. I think at least make more in-depth analysis on why specific designs work, and/or discover essential principles that make policy filtration work well, so that it informs how the specific policy filtration can be designed, across tasks.

3. [Presentation] The foundation of the proposed method is the reliability measure (or accuracy) for reward model prediction. The paper propose to use the the coefficient of determination (R-squared), but the paper do not include details for computing this measure, and do not clearly state the motivation and/or why this measure is valid to measure the reliability of reward estimate. For instance, why choose this measure? why not other measures, e.g., errors of the statistical estimation? Lack of clarity hinders understanding for the audience who are not familiar with R-squared. There are many other presentation issues, which makes the paper hard to understand.
We design a method to filter samples during RLHF to mitigate the inaccuracy of reward model.
@inproceedings{
anonymous2024policy,
title={Policy Filtration in {RLHF} to Fine-Tune {LLM} for Code Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=YilY5fGQny},
note={under review}
}
While direct policy optimization methods exist, pioneering LLMs are fine-tuned with reinforcement learning from human feedback (RLHF) to generate better responses under the supervision of a reward model learned from preference data. One major challenge of RLHF is the inaccuracy of the intermediate reward model, especially in code generation tasks that requires complex reasoning for the reward model to score a response. We find that the reliability of the reward model varies across responses assigned with different rewards. This motivates us to filter the samples whose rewards may be unreliable to improve the signal-to-noise ratio during policy learning, resulting in Policy Filtration for Proximal Policy Optimization (PF-PPO). To choose a proper policy filtering strategy, we use the coefficient of determination ($R^2$) between the rewards and actual scores on filtered samples as the metrics to help us find promising strategies since it measures how well the rewards filtered by PF-PPO indicate real performance. We provide extensive experiments to validate the effectiveness of PF-PPO in code generation tasks. We find that some variants of PF-PPO are highly effective and achieve the state-of-the-art performance of 7-billion-parameter models on HumanEval (+7.9\%) and MBPP (+0.7\%). Moreover, we create the LeetCode Contest benchmark and demonstrate the advantage of PF-PPO (+10.0\%) on this more challenging benchmark.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM RLHF
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d98b3cbae19b73d005fad8a32fa657df003fa9f0.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/22c1711f5e1f0980b5b409bba19e789eba02763e.zip
Policy Filtration in RLHF to Fine-Tune LLM for Code Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission