Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model
weak-to-strong;model alignment
Yes
No ethics review needed.
see weakness.
1. Innovative Transfer of Alignment Capability: WSPO innovatively transfers alignment capabilities from a weaker model to a stronger one, which is a novel approach in the field of LM alignment.
2. Performance Improvement: The paper shows significant improvements in win rates on challenging benchmarks, indicating the effectiveness of WSPO in aligning stronger models.
3. Empirical Evidence: The paper provides extensive experimental results, demonstrating WSPO's effectiveness across various tasks and datasets, including summarization, dialogue, and reasoning tasks.
4. Generalization Beyond Weak Labels: WSPO not only imitates weak model behavior but also generalizes beyond it, as shown by the strong model's improved performance on benchmarks.
The paper presents Weak-to-Strong Preference Optimization (WSPO), a pioneering approach for aligning language models (LMs) with human preferences by leveraging the alignment capabilities of weaker models to enhance stronger ones. Through learning distributional differences before and after the alignment of the weaker model, WSPO effectively transfers and amplifies alignment behaviors, demonstrating improved performance on key benchmarks such as Arena-Hard, AlpacaEval 2, and MT-bench.
1. Dependence on Quality of Weak Model Alignment: The success of WSPO is heavily dependent on the quality of alignment in the weak model, which might not always be optimal.
2. Hyperparameter Sensitivity: The performance of WSPO seems to be sensitive to the choice of hyperparameters like γ, which might require careful tuning for different models and tasks.
3. Efficiency Considerations: During WSPO training, the algorithm requires the loading of parameters from four models, two of which are weak models with limited parameters. The paper omits a comparison with previously established efficient methods like SimPO, which could potentially deliver better performance and higher training efficiency.
Yes
No ethics review needed.
1. Have you tested WSPO across different model architectures or families beyond Qwen? If so, could you provide insights into its effectiveness across various architectures?

2. Could you explain why the alignment capability amplifies when transferred from a weak model to a strong model? Are there theoretical or empirical insights that could clarify this phenomenon?

3. Given WSPO's reliance on certain hyperparameters (e.g., γ), do you have any practical guidelines for setting these across different model sizes or tasks? Could automatic tuning methods be integrated to simplify deployment?

4. Could you provide further details on WSPO’s computational efficiency, particularly in comparison to RLHF in real-world applications? How significant are the resource savings, and are there trade-offs in alignment quality for complex, large-scale tasks?

5. How critical is the quality of the initial alignment in the weak model to WSPO's success? Would a weak model with suboptimal alignment still provide useful signals for the strong model?
1. **Innovation in Model Alignment**: WSPO introduces a unique method of transferring alignment signals from weak to strong models, showcasing effective model alignment without the need for a dedicated reward model.
2. **Experimental Validation**: The paper provides thorough experimental evidence across several tasks, including summarization, single-turn dialogue, and complex benchmarks like MT-Bench and AlpacaEval 2. The results demonstrate WSPO's strong performance relative to baselines like DPO.
3. **Efficiency**: WSPO presents a more streamlined alignment process, avoiding the costly RLHF approach. This approach may offer practical advantages for resource-constrained settings, as the alignment signal is derived directly from weaker models.
4. **Comprehensive Analysis**: The authors conduct various analyses, including the impact of model size and the role of key hyperparameters, providing insights into the approach's scalability and stability.
The paper presents a novel approach called Weak-to-Strong Preference Optimization (WSPO), designed to enhance model alignment by leveraging alignment signals from weaker models to improve stronger ones. Inspired by weak-to-strong generalization, WSPO transfers the alignment abilities of a weaker model to a stronger one by optimizing the differences in distributions before and after weak model alignment. WSPO demonstrated competitive results across various benchmarks, showing a significant performance improvement over existing approaches, such as Direct Preference Optimization (DPO).
1. **Limited Cross-Model Analysis**: While the paper demonstrates the impact of weak-to-strong transfer within a specific model family, it lacks analysis of WSPO’s effectiveness across diverse architectures.
2. **Lack of Theoretical Explanation**: Although the approach works empirically, the authors do not provide a detailed theoretical rationale for why transferring alignment from weak to strong models amplifies alignment effectiveness.
3. **Reliance on Specific Datasets**: The experimental results heavily rely on specific datasets (UltraChat, Anthropic HH), potentially limiting the generalizability of WSPO to other alignment contexts. Broader dataset testing could validate the approach further.
4. **Parameter Sensitivity**: The method’s effectiveness appears sensitive to hyperparameter tuning (e.g., γ in WSPO), which might complicate deployment across different model types and alignment scenarios.
Yes
No ethics review needed.
1. The method relies on aligning the weak model first and then transferring this alignment to a stronger model. Are there scenarios where aligning the strong model directly would be more beneficial, and how does WSPO compare in terms of resource efficiency in such cases?
2. Is WSPO applicable to non-language model tasks, such as reinforcement learning agents or vision-language tasks? It would be interesting to understand the generalizability of the approach beyond language models.
Novelty: WSPO is a unique approach that effectively transfers alignment signals from weak to strong models, providing an alternative to reward model training. This is a valuable addition to existing alignment techniques such as RLHF and DPO.

Quality: The experimental results demonstrate that WSPO performs competitively with strong models trained using PPO, and in some cases even surpasses it in alignment capabilities.

Clarity: The paper provides a well-organized discussion of prior work, situating WSPO within the broader context of model alignment. The use of diagrams to illustrate alignment processes helps in understanding the proposed method.

Significance: WSPO has practical significance for reducing the computational cost of aligning strong models, as it eliminates the need for repeated reward model training.
The paper introduces a novel method called Weak-to-Strong Preference Optimization (WSPO), which aims to align large language models (LLMs) with human preferences by leveraging the alignment signals of weaker models. The proposed method builds on the concept of weak-to-strong generalization, where a strong model can amplify the alignment capability learned from a weaker model. WSPO works by learning the distributional differences before and after the alignment of a weaker model, which enables a stronger model to effectively inherit and improve upon the alignment behavior of the weaker one. The authors demonstrate that WSPO improves the win rate and alignment quality of the Qwen2-7B model in several benchmarks, including Arena-Hard, AlpacaEval 2, and MT-bench.
Complexity of Theoretical Derivations: The theoretical sections, particularly those related to deriving the WSPO objective and gradient, are quite dense and might be challenging for readers without a strong background in reinforcement learning. Consider providing more intuitive explanations or additional examples.
Yes
No ethics review needed.
1. The MMLU score (shown in Table 2) seems very low for QWen2 model, can you explain why it’s in the range of 40s? 

2. Do you have experiment results for other model families?

3. You showed that alignment from an even weaker model like Qwen2-0.5B can be helpful. I am curious about what happens if the weaker model is not weak anymore (e.g. use 7B model’s SFT and DPO checkpoint to compute the ratio as if they are pi_{r}^{weak} and \pi_{ref}^{weak}).
WSPO is broadly applicable to Reinforcement Learning with Human Feedback (RLHF) tasks, and the benchmark results validate its effectiveness. Remarkably, even though a weaker model like Qwen2 performs poorly on its own (as shown in Table 1), the distribution changes induced by reference and DPO-tuned models provide meaningful guidance. This enables the stronger model to achieve significant performance improvements, demonstrating the method's potential to enhance model alignment despite the initial limitations of weaker models.
This paper introduces Weak-to-Strong Preference Optimization (WSPO), a novel method for aligning large language models (LLMs) with human preferences. Inspired by the concept of weak-to-strong generalization, WSPO leverages distribution shifts observed during the alignment of weaker models to guide the fine-tuning of stronger models.
1. All experiments are conducted using the Qwen2 model family, and it would strengthen the study to include results from other open-source LLMs, (e.g., Llama3). Additionally, incorporating a Proximal Policy Optimization (PPO) baseline in Table 1 would provide a more comprehensive comparison.

2. While the proposed method and experiments are innovative and comprehensive from my perspective, the manuscript requires significant improvements in clarity and organization. Specifically, in Section 3.2, the process for obtaining $\pi_{r}^{weak}$ and $\pi_{ref}^{weak}$ should be explicitly detailed, as currently, **readers have to refer to the appendix** to understand their derivation through Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), and for the summarization task, via PPO.

3. Furthermore, the methods section needs clearer explanations of each notation upon the first introduction. The experimental section also lacks essential details and attempts to cover three sets of experiments simultaneously. To enhance readability and focus, it is recommended that Section 4.3 be emphasized with more comprehensive details, while at least one of the experiments in Sections 4.1 and 4.2 could be relocated to the appendix to streamline the presentation.
We propose a method called WSPO, which effectively transfers the alignment ability from a weak model to a strong model, resulting in an amplified alignment phenomenon.
@inproceedings{
anonymous2024weaktostrong,
title={Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=f7KxfUrRSb},
note={under review}
}
Aligning language models (LMs) with human preferences has become a key area of research, enabling these models to meet diverse user needs better. Inspired by weak-to-strong generalization, where a strong LM fine-tuned on labels generated by a weaker model can consistently outperform its weak supervisor, we extend this idea to model alignment. In this work, we observe that the alignment behavior in weaker models can be effectively transferred to stronger models and even exhibit an amplification effect. Based on this insight, we propose a method called Weak-to-Strong Preference Optimization (WSPO), which achieves strong model alignment by learning the distribution differences before and after the alignment of the weak model. Experiments demonstrate that WSPO delivers outstanding performance, improving the win rate of Qwen2-7B-Instruct on Arena-Hard from 39.70 to 49.60, achieving a remarkable 47.04 length-controlled win rate on AlpacaEval 2, and scoring 7.33 on MT-bench. Our results suggest that using the weak model to elicit a strong model with a high alignment ability is feasible.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
weak-to-strong model alignment
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7837f69956a84b4aaeed8d312c79336c4d2bd550.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/00530b3b0ec81636aef0f4a561a1391fe2cdd275.zip
Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission