Self-Alignment Optimization for Language Models
Self-Alignment;Annotation-free;Large Language Model
---

Thank you for your thoughtful comments. We would like to address some of your concerns and misunderstandings as follows:

> **Regarding the ability of LLMs to self-improve without external supervision:**

We respectfully disagree with the assertion that LLMs cannot self-improve without external supervision. In fact, recent work, such as (1) boosting reward models with preference-conditional multi-aspect synthetic data generation and (2) Self-Rewarding Language Models, demonstrates that LLMs can indeed leverage self-generated signals for self-improvement. Our approach builds on this foundation, showing that self-improvement can be achieved through self-supervised signals rather than relying solely on external supervision.

---

> **Regarding the simplicity of the idea and the surprising improvements on AlpacaEval 2.0 and Arena-Hard:**

While the core concept may appear simple, we believe the effectiveness of the SAO framework lies in its novel combination of dataset-free and annotation-free alignment techniques, which have not been fully explored before. The improvements seen on AlpacaEval 2.0 and Arena-Hard are significant, and we have provided a detailed analysis in Section 5.4 to explain the factors contributing to these results. This includes the influence of dataset size, different preference optimization algorithms, persona effectiveness, and self-judgment ability.

---
> **Regarding the experimental results and improvements on AlpacaEval 2.0:**

We have included a detailed explanation in lines 427-431 to clarify this point. Our primary goal was not to generate task-specific data pairs (such as for math or code), but rather to maintain downstream NLP performance while improving alignment ability. While models like Gemma-2-9B-it-SimPO, which are trained on specific datasets, sacrifice some downstream performance to achieve better alignment, our method offers a more balanced approach. SAO demonstrates proactive value by improving alignment without sacrificing downstream task performance.

---

> **Regarding the use of few-shot settings in Table 2:**

We follow the official few-shot evaluation settings used in the OpenLLM leaderboard, which are the standard in the field and have been adopted by works like SimPO (NeurIPS 2024). We understand that you are suggesting zero-shot settings; however, we believe that comparing with the few-shot setting is more representative of current practices, and we wanted to align with the prevailing methodologies used in the evaluation of models.

---

> **Regarding the need for larger LLMs (at least 70B) to support the claims of this paper:**

While we acknowledge that larger models can offer potential benefits, it is important to note that in academic research, especially in alignment, working with models of less than 10B parameters is more common due to the constraints of training large-scale models. Despite this, the performance we demonstrate with smaller models has been shown to outperform some of the most advanced models, such as GPT-4. This highlights the practical value of our approach, which can achieve competitive alignment performance without the need for enormous model sizes.

---

We hope that these clarifications address your misunderstandings and concerns and provide a clearer understanding of our contributions. 

---
---

Thank you for your feedback. We would like to clarify some points as follows:


> **Regarding “the variation of existing approaches” of the paper and several related works**

We appreciate your comments regarding the novelty of our work. However, we respectfully disagree with the suggestion that our proposed method, SAO, is merely a variation of existing approaches such as Self-Rewarding LMs. The overall workflow of SAO is significantly different, particularly with components like persona-based diverse prompt generation and pairwise self-judgment. Specifically, Self-Rewarding LMs rely on external datasets as seed datasets and require a few-shot setting to generate prompts. In contrast, SAO is fully dataset-free and annotation-free, which means it does not require external data for initialization or labeling. As shown in the Self-Rewarding LMs paper, their method performs poorly with only synthetic data, and combining it with external datasets yields better results. This highlights the distinction between our work and theirs.

We also thank you for pointing out relevant prior works such as SALMON, OAIF, and SAMI. We will include a discussion of these works in the Related Work section to better highlight how SAO differentiates itself. However, to better demonstrate the effectiveness of SAO, we focused on comparing it with modern baselines (e.g., Gemma-2-9B-it-SimPO and Gemma-2-9B-it-SPPO) in our main tables.

---

> **Regarding the use of GPT-4o-mini as a judge model for Arena-Hard and MT-Bench:**

We appreciate your concern about using GPT-4o-mini as a judge model for Arena-Hard and MT-Bench. The primary goal of using GPT-4o-mini was to evaluate the models in a cost-effective manner. GPT-4o-mini has demonstrated reasonable judgment capability in many existing works. To address your concern, we also evaluated Arena-Hard using GPT-4-Turbo as a judge, and the results show consistent improvement: Gemma-2-9B-it: 40.8 vs. Gemma-2-9B-it-SAO: 54.3. We will include these results in the appendix for further clarification.

---

> **Regarding the reported improvements on the Open LLM Leaderboard:**

We understand your concern that the reported improvements on the Open LLM Leaderboard appear minimal. However, we want to clarify that the primary objective of SAO is not to significantly improve downstream NLP performance but to enhance alignment performance without decreasing the performance of the model. 

When comparing SAO to Gemma-9B-it-SimPO, SAO demonstrates comparable alignment performance while maintaining or slightly improving downstream task performance, as shown in Table 2. In contrast, Gemma-9B-it-SimPO, which relies on an external labeled dataset, sacrifices downstream performance for alignment improvements. This trade-off is a critical consideration for real-world applications, and SAO addresses this issue by maintaining performance across tasks.

---

> **Regarding the primary novel contribution of the work:**

To clarify, the primary novel contribution of SAO is that it introduces a dataset-free and annotation-free framework for self-improvement in alignment, which incorporates persona-based prompt generation and pairwise ranking annotation-free approaches—both of which, to the best of our knowledge, have not been proposed previously in the field. The SAO framework can be applied to various preference optimization methods, such as SimPO, and achieves promising performance comparable to models trained on external datasets, while maintaining downstream task performance. This framework enhances the model’s intrinsic capabilities and ensures consistent performance across a wide range of tasks without significant trade-offs.

---

We hope these clarifications address your concerns and provide a clearer understanding of our work.
---

Thank you for your constructive feedback. We would like to clarify some points as follows:


> **Regarding the novelty of SAO's components:**

We appreciate your feedback. The motivation of our paper is to propose a novel self-alignment framework that is dataset-free and annotation-free. While we respectfully disagree with the assessment of novelty in terms of individual components, we also provide a detailed analysis of each in Section 5.4. Specifically:
- **Persona-based prompt generation**: As shown in Section 5.4.3, even without persona role-playing (i.e., using more repetitive prompts), the model still shows significant improvement based on the SAO framework. However, incorporating persona role-playing allows for more diverse prompts, which further enhances alignment performance.
- **Self-judgment**: While we adopt criteria from Shen et al. (2024) (shown in Fig.1), we introduce a novel pairwise self-judgment method in Section 5.4.4, where our results show that our model outperforms baselines like Random-Judge and ArmoRM-Judge.
- **Preference optimization**: As demonstrated in Section 5.4.2, SAO is flexible in improving vanilla models using various optimization algorithms, including DPO, ORPO, and SimPO, with SimPO providing the best results, which we adopt as the default in our framework.

---

> **Regarding the performance improvement compared to direct counterparts trained on external datasets (e.g., Gemma-9B-it-SimPO):**

We understand your concern regarding the quality difference between self-generated datasets and externally labeled datasets. While human-level labels and curated datasets do have better alignment potential, as is commonly acknowledged, our paper introduces a self-improvement framework that operates without external signals. Therefore, it is more intuitive and reasonable to compare SAO against vanilla models (e.g., Gemma-2-9B-it), as the model improves itself in this case. When comparing our SAO method with Gemma-9B-it-SimPO, SAO demonstrates **comparable alignment performance**, while **maintaining or slightly improving downstream task performance**, as shown in Table 2. In contrast, Gemma-9B-it-SimPO, which relies on an external labeled dataset, sacrifices downstream performance for alignment improvements—this is a crucial consideration for practical applications. Additionally, as SAO is still in an emerging research area, we have included the limitations of our framework in Section 7, where we acknowledge that although our approach has shown promising results with simple prompt templates, exploring more complex templates may lead to further improvements.

---

> **Regarding the practical impact of being dataset-free:**

We acknowledge the importance of improving RLHF pipelines; however, the primary focus of SAO is **data-center efficiency** during the pre-RLHF stage, where it eliminates the need for expensive human labeling or GPT-4-level labor and filtering processes. We believe this approach makes a significant contribution to the field of self-improvement through synthetic data. It offers a new perspective on how self-generated data can enhance LLM alignment with minimal external input. The importance of self-synthetic data is also highlighted in the works we compare to, such as Self-Rewarding and SPPO.

---

> **Regarding efficiency and cost concerns:**

Thank you for raising this important point. We would like to clarify that SAO does not introduce additional training costs compared to existing RLHF methods (e.g., SimPO). Our focus is primarily on **training dataset construction** rather than increasing training costs. The techniques you mentioned like prompting, representation editing, and reward-guided test-time decoding are aimed at improving the decoding stage and are not directly comparable to SAO. These methods generally focus on training-free approaches, which still lag behind in performance compared to SAO and other mainstream RLHF methods. Furthermore, these decoding-optimation models are not as easily deployable as Hugging Face models.

---

We hope these clarifications address your concerns and provide a clearer understanding of our work! Please feel free to reach out with any further questions during the rebuttal stage.
Yes
No ethics review needed.
1. Suggestion on the writing: reduce the length of section 3, which is commonsense knowledge in the field now
2. Questions about the efficiency: the proposed techniques still require the significant cost of training/fine-tuning, how this paper should be positioned in the current trend of much cheaper alignment techniques, like using prompting, representation editing, or reward-guided test-time decoding to improve the alignment performance
1. The SAO framework operates without the need for external, human-annotated data, eliminating costly annotation requirements while maintaining alignment effectiveness.
2. Empirical results show that models fine-tuned with SAO achieve substantial improvements over their vanilla base models, particularly in alignment-specific metrics like Length-Controlled Win Rate (LC) and Win Rate (WR) across multiple benchmarks.
3. The paper provides comprehensive empirical results, evaluating SAO across a range of benchmarks and scenarios, including AlpacaEval 2.0, MT-Bench, and Arena-Hard, which thoroughly demonstrate the robustness and scalability of the proposed method.
This paper proposes Self-Alignment Optimization (SAO), a framework for enhancing language model alignment without external datasets or human-labeled feedback. Traditional alignment methods, like Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF), are resource-intensive due to the need for specialized reward models or substantial external data. SAO addresses these limitations by enabling models to align autonomously: a model generates prompts based on various personas, produces responses, and then ranks these responses through self-assessment. The framework relies on this self-generated dataset and uses preference optimization (SimPO) to refine model alignment, reducing the need for human input or costly external tools.

Empirical results show that SAO achieves improvements in model performance across several benchmarks. For instance, on AlpacaEval 2.0, SAO-tuned models exhibit significant gains in Length-Controlled Win Rate (LC) and Win Rate (WR) over baseline models, with similar enhancements observed on the MT-Bench and Arena-Hard benchmarks. Additionally, SAO-tuned models maintain or slightly improve downstream NLP performance metrics as measured by the Open LLM Leaderboard, suggesting that SAO can enhance alignment without compromising general task capabilities. Further analysis in the paper explores factors that influence SAO's effectiveness, including synthetic dataset size, persona role diversity, and optimization methods, underscoring the potential of SAO as an efficient and scalable alternative to traditional alignment approaches.
1. Most of SAO’s components—such as persona-based prompt generation, self-generation/self-judgment, and preference optimization using SimPO—are adapted from prior work, potentially limiting the novelty of the technical approach. To strengthen the contributions, the paper needs to explore innovative adaptations or improvements to these existing methods that demonstrate clear enhancements in model performance or alignment capabilities.
2. While SAO achieves improvements over vanilla models, its enhancements compared to more meaningful direct counterparts (e.g., Gemma-2-9B-it-SAO vs. Gemma-2-9B-it-SimPO) are relatively minor. Given that SimPO relies on external datasets, this limited improvement suggests that self-generated preference data may currently be less effective. The study needs a deeper examination of self-generated data’s limitations and a discussion of methods to enhance its quality, which may reveal additional advantages over using external data.
3. Although the paper promotes SAO’s dataset-free and annotation-free aspects, the practical impact of this approach is not fully substantiated. Alignment research aims for impactful, real-world outcomes, and it would be helpful if the authors clarified specific advantages or applications where being dataset-free provides measurable benefits (e.g., scenarios with limited access to annotation resources or environments needing continual self-adaptation). For example, people probably don't need a full dataset/annotation-free pipeline, but just a single significant improvement in the RLHF pipeline, like the SimPO technique. Further analysis of the unique applications and limitations of a dataset-free strategy could better ground this approach in practical relevance.
Yes
No ethics review needed.
* Given that dataset-free or annotation-free alignment approaches are already established in the field, could the authors clearly articulate in one sentence what they consider to be the primary novel contribution of this work?
* The approach demonstrates meaningful performance gains without requiring external labeled datasets, which could make fine-tuning more accessible and cost-effective for smaller organizations.
* The paper provides nice ablation studies and analyses, particularly in Figure 3, exploring various aspects like dataset size impact, optimization algorithms, persona role-play influence, and self-judgment capabilities.
* The experimental evaluation is systematic, covering both subjective and objective benchmarks, with clear performance metrics and comparisons against baseline models.
* The paper proposes Self-Alignment Optimization (SAO), a framework for fine-tuning language models without external labeled datasets by leveraging self-generated prompts and responses through persona role-play.
* The study evaluates SAO on two main models: Gemma-2-9B-it and Llama-3-Instruct-8B, testing them on multiple benchmarks including AlpacaEval 2.0, MT-Bench, and Arena-Hard.
* Key findings suggest performance improvements on subjective benchmarks (e.g., Gemma-2-9B-it-SAO achieves 69.2% LC and 66.0% WR on AlpacaEval 2.0) while maintaining performance on objective downstream tasks.
* The paper's technical novelty is limited, as similar self-improvement approaches have been explored in previous works like Self-Rewarding LMs. The core concept appears to be a variation of existing methods rather than a new approach.
  * The claimed benefits of being "dataset-free and annotation-free" are not novel in the field
  * The paper lacks comparison and discussion with relevant prior work on self-alignment, for example SALMON, OAIF, and SAMI.
* There are several methodological concerns:
  * The use of GPT-4o-mini as a judge model for evaluating Arena-Hard and MT-Bench seems questionable given its relative weakness compared to state-of-the-art models
  * The reported improvements on the Open LLM Leaderboard are minimal to non-existent
Yes
No ethics review needed.
Is there a specific reason for using different prompts for MT-Bench and AlpacaEval?
- The proposed Self-Alignment Optimization (SAO) framework removes the need for expensive human annotations or AI-labeled preference data. By doing so, SAO makes model alignment significantly more affordable and accessible, lowering the barrier for effective model refinement.
- SAO leverages persona role-play to generate a wide variety of prompts, which enriches the diversity of inputs and responses the model can handle. This method shows effective results in generating robust, varied prompts, as demonstrated by the paper's experiments.
- SAO’s effectiveness is shown across different language models and multiple benchmarks (e.g., AlpacaEval 2.0, MT-Bench, Arena-Hard). The framework consistently improves performance across diverse tasks and evaluation settings, indicating its general applicability and robust alignment capabilities.
This paper presents Self-Alignment Optimization (SAO), a framework that bypasses the need for expensive human-annotated datasets or AI-labeled preference data in aligning LLMs. Instead of relying on annotated data or costly labeling with larger models, SAO aims to replace with a self-supervised alignment process. In SAO, a chat-based model generates its own prompts and feedback through persona-based role-play, evaluates its responses, and refines itself based on these self-assigned preferences. Experiments show that models trained with SAO outperform baseline models on various benchmarks (e.g., AlpacaEval 2.0, MT-Bench, Arena-Hard) in alignment tasks.
- **Dependence on Initial Model Performance for Self-Alignment:** The SAO framework appears to rely on a certain level of initial model performance for effective self-alignment, as the model must be able to understand and follow instructions to generate meaningful responses. However, the paper only demonstrates SAO’s efficacy with relatively well-performing LLMs. It remains unclear how SAO would perform with smaller LLMs, such as those with 1B or 3B parameters, which might struggle with instruction following and generating coherent responses. Further investigation is needed to evaluate SAO’s effectiveness on these smaller-scale models.
- **Challenges with Output Consistency in Self-Supervision:** In some cases, models may fail to produce outputs in the desired format, which could lead to instability or even failure in the training process. This issue poses a risk to the reliability of the self-alignment process. How do authors address instances where the model’s output deviates from the required format during training—whether through corrective measures, filtering techniques, or additional constraints to ensure format consistency.
- To assess the effectiveness of persona-style prompt generation, how does the model's performance compare when using existing prompts instead of persona-based prompts, while keeping all other components of the method unchanged?
Yes
No ethics review needed.
The experimental results must be well-examined: a) why the improvements on AlpacaEval 2.0 are so high, but there is no improvement in average score on downstream NLP tasks; b) Why experiments in Table 2 are in few-shot settings? Those models are all instruct models, please test in zero-shot setting; c) Larger LLMs, at least 70B, are required to support the claims of this paper.
1.	The improvements on AlpacaEval 2.0 and Arena-Hard Benchmarks over base models are surprising.
This paper proposes a way of self-improving without any data annotation, specifically, they randomly sample some personas from Persona-Hub (Chan et al. 2024), and ask base LLMs to produce prompts, then, generate two responses for each prompt, followed by prompting LLMs again for pair-wise ranking. With those data, they can run SimPO to update base LLMs. Experiments show significant improvements on AlpacaEval 2.0 and Arena-Hard Benchmarks, however, there is no difference in many downstream NLP tasks (e.g. GSM8K, MMLU etc.)
1.	Many researchers are working on self-improving with external supervision and guidance to achieve some improvements over base policy models. And I believe all those studies have already confirmed that LLMs itself cannot be self-improved without using external supervision. 
2.	The idea of this paper is very simple, the improvements on AlpacaEval 2.0 and Arena-Hard are very surprising, further investigation and analysis must be done on those results. 
3.	The novelty of this paper is very limited.
Yes
NA
No ethics review needed.
* Why is there  such a big difference between SAO's effectiveness on Llama-3 and on Gemma-2? Is it possible that difference arises from that Gemma-2 is a much better reward model when being prompted to pick responses? 

An experiment comparing the LLM-as-a-judge accuracy between Llama-3 and Gemma-2 model may be helpful to answer the question.
* Novel method of self-generating prompt using persona library as seed. 

* Extension of RLAIF work that uses itself as the judge model to perform self-improvement. This is highly efficient and resource-friendly, without needing to collect external feedback or using external reward models. 

* It designed a comprehensive alignment experiment, and ablated on size of alignment dataset, and importance of having diversity in prompt space, and optimization algorithm. 

* it shows impressive result, especially for Gemma-2 model.
Paper proposes Self-Alignmenet Optimization(SAO) claiming it is a data-free, annotation-free framework to perform preference alignment. 
The paper shows that without using an external prompt space, and an external reward model, LLM can learn from an on-policy sampled dataset annotated by using itself as a judge. The paper demonstrates improvements on 3 trusted alignment Benchmarks, AlpacaEval, MT-Bench, and ArenaHard over the starting model, and for gemma model in particular, it shows comparable performance as models aligned using SoTA trained reward models on AlpacaEval. Meanwhile, experiments on academic Benchmarks show no noticeable regression overall from the starting model.
* "data-free, annotation-free framework " is in accurate. For example, it uses Persona Library as seed, and it uses self-judge to annotate data. Better claim to be free of external annotator and generate prompt space from seed. 

* Lack of novelty in the method. It is overall an extension of existing RLAIF literature, using self-judge plus on-policy sampling dataset. The main difference is that it proposes to self-generate the prompt space using Persona library as seed. 

* If one major point of novelty in the paper is Persona based prompt space generation, then, experiment will be needed to compare it against existing open-source prompt space, such as UltraFeedback or HelpSteer. I encourage authors to include such baselines to your main experiment tables. 

* If the Self-judge portion is the major novelty of the paper, then, experiment should be performed to compare self-judge vs external trained reward or other LLM-as-a-judge reward. An experiment table demonstrating the LLM-as-a-judge's reward accuracy would be helpful to understand the contribution here. 

* Evaluation could be improved to be more consistent and clear by comparing AlpacaEval, ArenaHard, and MT-Bench of baseline models + your method all in Table-1, similar to the presentation in SimPO paper. I would like to see how much improvement proposed method brings on all three Benchmarks compared to other methods. Placing heavy emphasis on AlpacaEval2.0 risk over-fitting to one Benchmark.
@inproceedings{
anonymous2024selfalignment,
title={Self-Alignment Optimization for Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=QWMgTMRUnB},
note={under review}
}
Traditional reinforcement learning from human feedback (RLHF) relies heavily on costly and time-consuming human-annotated datasets. Even Reinforcement Learning from AI Feedback (RLAIF), which trains a reward model using AI-generated preference data before refining the language model through reinforcement learning, remains expensive. These methods often necessitate either specialized reward model designs or larger models (e.g., GPT-4) for external labeling. In this paper, we introduce a dataset-free and annotation-free framework called Self-Alignment Optimization (SAO), which addresses the aforementioned issue by aligning the model with its own prompts and feedback as preferences. SAO begins with a chat-based model that engages in persona role-play to generate diverse prompts and responses, which are then self-evaluated and used for preference optimization.
Extensive experiments with two strong LLMs on several benchmarks demonstrate the effectiveness of SAO. Specifically, on AlpacaEval 2.0, Gemma-2-9B-it-SAO achieves a Length-Controlled Win Rate (LC)  of 69.2\% and win rate (WR) of 66.0\%, surpassing the baseline model by 18.1\% and 27.9\%. Llama-3-Instruct-8B-SAO reaches 33.3\% LC and 39.0\%  WR, with performance improvements of 10.4\% and 16.4\%, respectively. On the MT-Bench benchmark, Gemma-2-9B-it-SAO and Llama-3-8B-Instruct-SAO score 7.41 and 6.76, compared to their pre-SAO scores of 7.09 and 6.70. The Arena-Hard benchmark shows even greater gains from SAO, with Gemma-2-9B-it's WR increasing from 52.6\% to 70.1\% and Llama-3-Instruct-8B's WR rising from 40.3\% to 56.4\%. In addition, our further experiments demonstrate that models fine-tuned with SAO exhibit similar or even superior performance on downstream NLP tasks compared to baseline models, rather than those trained with external labeled datasets, which enhance alignment ability but may compromise some general capabilities. We anticipate that this work will provide new insights for future research on self-improvement in LLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Self-Alignment Annotation-free Large Language Model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/704f5584a2a0c3d9161bb50c1ec6b2085350fd4e.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/038e53dbd28ae8fc30b843fc6a1242fff66976fb.zip
Self-Alignment Optimization for Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission