Sample Efficient Alignment for LLMs
rlhf;online dap;llm alignment;sample efficiency
Yes
No ethics review needed.
- Do you have results that can showcase this method on a variety of tasks and many different datasets? How generalizable is this method (in terms of how well it works)?
+ The paper builds upon the direction of aligning LLMs that considers the epistemic uncertainty of the reward function and utilizing the variance to guide the model to collect preference data in directions that maximally obtains information.
+ The algorithm is sound and theoretically solid. They build upon well performing methods for uncertainty estimates using the ensemble reward model method.
+ The results shown are promising (although see weakness for caveats).
The paper considers the problem of aligning LLMs to humans via online interaction and in a sample efficient manner that minimizes the overall human annotations consumed. It proposes a practical algorithm that can be implemented and using the tldr dataset shows that it can achieve extremely high win-rates when compared against the baseline alignment algorithm.
- The main weakenss of this paper is the lack of siginificant experimentation. The paper only considers the summarization task with the tldr dataset. It does not consider more diversity in tasks/datasets and does not utilize some of the most common tasks in the LLM literature (e.g., reasoning, code, general knowledge). Even within summarization, it does not consider > 1 datasets. Improvements in one dataset is typically very limiting.
Yes
No ethics review needed.
Please see the last part.
1.	The paper studies the LLM alignment problem from an online view. It is reasonable to formulate the problem as a contextual dueling bandit problem. 
2.	The paper is inspired by the classic Thompson sampling algorithm to actively explore the preference function and proposes a corresponding version suitable for real-world application.
3.	The paper builds a learning system for studying methods in online LLM alignment. This provides efficient tools for the community on this research line.
This paper studies the online alignment for LLMs. It formulates the LLM alignment as a contextual dueling bandit problem. It proposes a Thompson sampling-like algorithm for both the regret minimization and best arm identification setting. For the challenge encountered when implementing this algorithm including the model constraints, the intractable operation to sample a posterior and argmax action, the paper proposes corresponding solutions to enable real-world application. Finally, the authors build a learning system for studying methods in online LLM alignment and compare the proposed method with available baselines in different scale levels of experiments. The proposed method shows superior sample efficiency compared with baselines.
1.	In Line 383, the authors claim they omit a baseline SELM since it shares a very similar algorithmic design as XPO. This is generally not an acceptable reason for excluding a baseline comparison.
Yes
No ethics review needed.
see above.
This paper is nicely written and well-executed. The details are well-presented.
This paper considers explorations by approximate TS in LLM alignment.
This paper is a bit incremental and the idea is well expected. The contextual bandit formulation is well-known and the implementation of TS using ensemble is pretty straightforward.  

There are multiple key questions this paper fails to answer:

1. Why ensemble on the top of DPO makes sense and how good the uncertainty estimation is? This is not a standard supervise learning problem. This paper lacks a way to directly measure it. I do not think ensemble + X can quantify uncertainty in every problem.  

2. The reward model is too small (0.4B) and the base model Pythia family is outdated such that the empirical result may become useless in a practical setting. It is very common that the benefit of active exploration could disappear when the base model and reward model become much stronger. 

3. The baseline number is not comparable with other literatures since the author uses its own oracle reward model. There are tons of implementation details that could be hidden for other baselines. I think it would be good to have an apple-to-apple comparison with the same setting in the original DPO paper.
Yes
No ethics review needed.
Concretely, in what ways does SEA differ from APL? How do you explain the large increase in sample efficiency?
- The contextual dueling perspective is interesting, insightful, and to the best of my knowledge, new.
- The paper does a good job of covering the most relevant related literature, and how it relates to the proposed approach.
- The description is very limited with no details in the supplementary material, but it is possible that the testbed they develop to evaluate different approaches is a valuable contribution in its own right
This paper frames the problem of aligning LLMs with pairwise preferences as a contextual dueling bandit problem. The authors propose a method that adapts a well-know & effective bandit algorithm, Thompson sampling (TS), to LLMs. This is not trivial, because TS requires maintaining and sampling from a belief over reward functions. To address this, the authors provide tractable approximations. They evaluate their proposed approach empirically on LLMs of various sizes, and provide comparisons to related work.
- The writing could be improved significantly; parts of the paper are unclear, imprecise, and many details are omitted and should at least be present in the supplementary materials. A non-exhaustive list of examples:
    - line 144: "based on a binary stochastic feedback $z$" -> z is never referred to anywhere outside of this line as far as I see
    - line 196: "offline RL" -> I think the authors might be conflating offline RL with offline preference collection. RLHF typically uses online RL. PPO is an online algorithm.
    - I struggle to make sense of Figure 3, for example I fail to understand how (d) is meant to differ from (b).
- The posterior approximation scheme (Eq. 9) is not well motivated. I can see it is computationally attractive, but why do we believe this is a reasonable way to approximate the posterior? A cursory glance at the papers mentioned around Eq. 9 did not help. Since this is one of the central components of the proposed approach, I would expect a more rigorous justification here.
- The experiments are comprehensive, but I struggle to take them at face value. Unfortunately, few details are provided, neither in the main text, nor in the supplementary materials.
    - There are design decisions involved in setting up baselines, and these are not discussed.
    - I can't reconcile any of the curves in Figure 6 with the curves in Figure 5.
- Some of the metrics are problematic, as I see it. The "online win rate" (line 387) would make a model that deliberately samples poor responses look great.
@inproceedings{
anonymous2024sample,
title={Sample Efficient Alignment for {LLM}s},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Pf8i7cv2CH},
note={under review}
}
We study methods for sample-efficiently aligning large language models with human preferences given budgeted online feedback. 
We first formulate the LLM alignment problem in the frame of contextual dueling bandits.
This bandit formulation, subsuming the recently emerging online RLHF / online DPO paradigms, naturally quests for sample-efficient algorithms.
Leveraging insights from bandits, we investigate two algorithms for active exploration based on Thompson sampling and shed light on their use cases. Our agent, termed as Sea ($\textbf{S}$ample $\textbf{E}$fficient $\textbf{A}$lignment), is empirically validated with extensive experiments, across 3 scales (1B, 2.8B, 6.9B) and 3 preference learning algorithms (DPO, IPO, SimPO). The results show that Sea aligns the LLM with oracle's preferences highly sample-efficiently, surpassing recent SoTA methods. We will open-source our codebase to accelerate the research in this field.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
rlhf online dap llm alignment sample efficiency
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/54dddb6fb2ddcee10f95276174da65e4dd758e1e.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Sample Efficient Alignment for LLMs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission