Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF
Reinforcement Learning;Reinforcement Learning from Human Feedback
Yes
No ethics review needed.
- Could this approach also be applied to LLMs only trained with the language modelling objective?
- How would this approach perform in more challenging planning scenarios such as task-oriented dialogue?
- Presents an interesting contribution to the area of RL in LLM's, making a stride in solving the problem of longer term planning.
- Illustrates the effectiveness of the technique not only on large models but also on smaller LLMs.
- Presents and proves performance guarantees of the approach.
Long-term and multi-turn planning remains a recognized limitation in large language models (LLMs), limiting their effectiveness in complex, goal-oriented scenarios such as extended dialogue interactions. While reinforcement learning from human feedback (RLHF) with extended context lengths has been employed to address this, these methods encounter high computational costs and are vulnerable to covariate shift. This work presents REFUEL, a novel reinforcement learning framework designed for long-term planning. REFUEL optimises the model for the relative future, thus mitigating covariate shift.
- REFUEL was only applied to the Llama 3 family of LLMs and further only the instruction tuned versions of these models. It would be more complete if the approach was also tested on models with different pretraining, e.g. the supervised language modelling pretrained models, models such as Gemma or Phi etc.
-  The datasets used for evaluation involve reasoning and harmful and helpful conversations with LLMs, but true goal oriented datasets such as those in task-oriented dialogue would present more of a long term planning challenge to test this approach.
Yes
No ethics review needed.
See weaknesses discussion.
Some strengths include the following:

* The construction of the optimization algorithm I find novel and elegant.

* Very strongly grounded in theory of RL with the most being entirely explained from the common formulations of Q-learning paired with optimiziation techniques. Rather clearly written overall, as well taking into account less familiar readers with the details of the theory through the presence of a "Intuitive explanation" section.

* The use of two benchmarks (UltraInteract, AnthropicHH), as well as open-source models (LLama, ArnoRM) make results reproducible and ground the contributions.
The manuscript introduces a training technique, grounded in RL theory/practice, for improving multi-turn dialogue generation using large-language models. The key problem addressed is that in standard multi-turn dialogue generation, training is done usually by relying on single-turn techniques, i.e. optimizing the last step of a dialogue conditioning on all previous steps. The manuscript calls out as this being a problem, since the policy generating previous data is not the one being optimized and refers to it as *covariate shift*. The proposed solution is grounded in Q-learning, avoiding the need for training separate actor / critic networks which are cumbersome to maintain and unstable to train.

The manuscript points out that with Transformers state can easily be restored (by just feeding back in a partial sequence) and alternative (counterfactual) steps can be generated (by just sampling). Then through a series of tricks the policy training can be tied to the change in relative rewards.

The evaluation setup leverages UltraInteract, making use of a LLama-8B for experiments and LLama-70B for simulating a user. Each turn quality (i.e. win-rate over other responses) is determined using GPT-4. The results clearly indicate improvements on turns 3, 4, 5 but are below baselines on turns 1, 2.
Although I could follow the theory behind the paper and I'm familiar with most of it, I'm not the best reviewer for the correctness of the derivation of the formulations and final objective trained. Therefore my confidence is rather low on this aspect.

Some weaknesses I'd like to ask authors about:
* Why not extend beyond H=5? This seems rather limiting and although improvements are noticeable in the GPT-4 evaluations, presumably the benefits should be much stronger at say H>10.

* Although it is commonly accepted practice for GPT-4 to evaluate win-rates (or quality as judge), are there other measurements that can help quantify usefulness of REFUEL? For example, the qualitative examples in Appendix D. indicate repetitive text by other baselines. Yet looking at green highlights of long responses for math problems, seems to indicate some reasoning improvements. 

* Can the authors comment on utility beyond multi-turn dialogue generation, e.g. perhaps for multi-step reasoning or tool-use?
Yes
No ethics review needed.
* Why are dialogues filtered to a maximum of 5 turns? It would seem that the performance improvement would be larger on longer dialogues.
* Why is performance degraded over most alternative setups for turns #0 and #1?
# Originality

To the best of my knowledge, the refuel technique and its application to multi-turn dialogue tasks is novel, as most existing work leverage an actor-critic framework.

# Quality & Clarity

The paper is well-written and easy to follow. It states clearly the limitations of current approaches and presents its contributions in a straightforward manner. The chosen hyperparameters and detailed proofs are clearly described in the appendix.

# Significance

The task of designing high-quality dialogue agents has become ever more visible in the past couple of years. Controlling generations over multiple dialogue turns is critical to the success of such agents, as it could enable more powerful and human-like capabilities with dialogues that carry an extended amount of context and information in previous turns and that are strategically generated in order to achieve a macroscopic goal. The paper is making a significant stride towards that.
This work introduces REFUEL, a method addressing the covariate shift issue appearing when commonly applying single-turn RLHF methods to multi-turn dialogues. It frames the problem as a multi-step MDP and merges the two-step of actor-critic procedures into a unified procedure. It shows improvements on benchmarks such as UltraInteract and AnthropicHH when having sample conversations evaluated head-to-head by GPT4.
The evaluations proposed could be more extensive. In particular, it may be interesting to show evaluations on more common single-turn dialogue tasks (e.g. QA tasks) in order to show whether the quality is degraded for these tasks. It is also surprising to see a strong degradation in quality for turns #0 and #1 compared to the raw Llama 8B model, and the paper does not provide an explanation for this.
Yes
No ethics review needed.
1. Does REFUEL's approach work for other multi-turn tasks beyond dialogue, such as multi-step reasoning or sequential decision-making tasks?
2. Can you provide ablation studies showing the individual impact of key components in REFUEL? Specifically, how much does the relative future reward regression contribute versus the on-policy data generation?
3. Can you demonstrate REFUEL's effectiveness on significantly longer conversations (e.g., 10-20 turns)?
1. The paper addresses a significant limitation in current RLHF approaches by properly handling multi-turn dialogue scenarios instead of treating them as single-turn tasks.
2. The theoretical analysis is thorough and proves that REFUEL requires weaker conditions than previous methods like Natural Policy Gradient.
3. The empirical results are impressive, showing that a smaller model trained with REFUEL can outperform much larger models on long dialogues.
4. The method is computationally efficient by avoiding the need for an additional critic network.
This paper introduces REFUEL (REgressing the RElative FUturE for reinforcement Learning), a novel approach for multi-turn Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). The key contributions include:
1. A new algorithm that addresses covariate shift in multi-turn dialogue by using self-generated data and treating the task as a sequence of regression problems
2. Theoretical guarantees showing REFUEL can match any policy covered by the training set
3. Strong empirical performance, where an 8B parameter model fine-tuned with REFUEL outperforms a 70B parameter model on long multi-turn dialogues
4. A novel approach that eliminates the need for an explicit critic network by using a reparameterization trick and relative future rewards
1. The experimental evaluation is limited to dialogues of 5 turns or less, which may not fully demonstrate the method's capability for very long conversations.
2. The simulator used for evaluation, while sophisticated, may not fully capture real human interaction patterns.
3. The paper lacks ablation studies on different components of the algorithm and their contributions to the final performance.
4. The comparison with PPO-based methods is omitted due to computational constraints, which leaves some questions about relative performance against all potential baselines.
@inproceedings{
anonymous2024regressing,
title={Regressing the Relative Future: Efficient Policy Optimization for Multi-turn {RLHF}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=cVyELMpMRS},
note={under review}
}
Large Language Models (LLMs) have achieved remarkable success at tasks like summarization that involve a *single turn* of interaction. However, they can still struggle with *multi-turn* tasks like dialogue that require long-term planning. Previous works on multi-turn dialogue extend single-turn reinforcement learning from human feedback (RLHF) methods to the multi-turn setting by treating all prior dialogue turns as a long context. Such approaches suffer from *covariate shift*: the conversations in the training set have previous turns generated by some reference policy, which means that low training error may not necessarily correspond to good performance when the learner is actually in the conversation loop. In response, we introduce REgressing the RELative FUture (REFUEL), an efficient policy optimization approach designed to address multi-turn RLHF in LLMs. REFUEL employs a single model to estimate Q-values and trains on self-generated data, addressing the covariate shift issue. REFUEL frames the multi-turn RLHF problem as a sequence of regression tasks on iteratively collected datasets, enabling ease of implementation. Theoretically, we prove that REFUEL can match the performance of any policy covered by the training set. Empirically, we evaluate our algorithm by using Llama-3.1-70B-it to simulate a user in conversation with our model. REFUEL consistently outperforms state-of-the-art methods such as DPO and REBEL across various settings. Furthermore, despite having only 8 billion parameters, Llama-3-8B-it fine-tuned with REFUEL, outperforms Llama-3.1-70B-it on long multi-turn dialogues.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Reinforcement Learning Reinforcement Learning from Human Feedback
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9626028e3e86cde693c174d3bc1db3a57d7f5971.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/384ace18c5a89c9e0249bda1c68807e55c1c4c08.zip
Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission