Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
Online Alignment;Large Language Model
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
@misc{
zhang2024selfexploring,
title={Self-Exploring Language Models: Active Preference Elicitation for Online Alignment},
author={Shenao Zhang and Donghan Yu and Hiteshi Sharma and Han Zhong and Zhihan Liu and Ziyi Yang and Shuohang Wang and Hany Hassan Awadalla and Zhaoran Wang},
year={2024},
url={https://openreview.net/forum?id=4zQ5eIPtMp}
}
Preference optimization, particularly through Reinforcement Learning from Human Feedback (RLHF), has achieved significant success in aligning Large Language Models (LLMs) to adhere to human intentions. Unlike offline alignment with a fixed dataset, online feedback collection from humans or AI on model generations typically leads to more capable reward models and better-aligned LLMs through an iterative process. However, achieving a globally accurate reward model requires systematic exploration to generate diverse responses that span the vast space of natural language. Random sampling from standard reward-maximizing LLMs alone is insufficient to fulfill this requirement. To address this issue, we propose a bilevel objective optimistically biased towards potentially high-reward responses to actively explore out-of-distribution regions. By solving the inner-level problem with the reparameterized reward function, the resulting algorithm, named Self-Exploring Language Models (SELM), eliminates the need for a separate RM and iteratively updates the LLM with a straightforward objective. Compared to Direct Preference Optimization (DPO), the SELM objective reduces indiscriminate favor of unseen extrapolations and enhances exploration efficiency. Our experimental results demonstrate that when fine-tuned on Zephyr-7B-SFT and Llama-3-8B-Instruct models, SELM significantly boosts the performance on instruction-following benchmarks such as MT-Bench and AlpacaEval 2.0, as well as various standard academic benchmarks in different settings.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Shenao_Zhang1 ~Donghan_Yu2 ~Hiteshi_Sharma1 ~Han_Zhong1 ~Zhihan_Liu1 ~Ziyi_Yang1 ~Shuohang_Wang1 ~Hany_Hassan_Awadalla1 ~Zhaoran_Wang1
Shenao Zhang Donghan Yu Hiteshi Sharma Han Zhong Zhihan Liu Ziyi Yang Shuohang Wang Hany Hassan Awadalla Zhaoran Wang
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Online Alignment Large Language Model
I certify that there is no acknowledgement section in this submission for double blind review.
zhang|selfexploring_language_models_active_preference_elicitation_for_online_alignment
/pdf/068bd2cbc229b13a233275af3792af892675429a.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/d7aaa3fdf6770dc66cb3895afd2fc7369615f10c.zip
Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission