Weak-to-Strong Extrapolation Expedites Alignment
Chujie Zheng Ziqi Wang Heng Ji Minlie Huang Nanyun Peng


The Directed Landscape is a Black Noise
    [
    April 29, 2024
=======================================




Although the capabilities of large language models (LLMs) ideally scale up with increasing data and compute, they are inevitably constrained by limited resources in reality.
Suppose we have a moderately trained LLM (e.g., trained to align with human preference) in hand, can we further exploit its potential and cheaply acquire a stronger model?
In this paper, we propose a simple method called  to boost LLMs' alignment with human preference.
 assumes that a medium-aligned model can be interpolated between a less-aligned (weaker) model, e.g., the initial SFT model, and a better-aligned (stronger) one, thereby directly obtaining this stronger model by extrapolating from the weights of the former two relatively weaker models. 
On the AlpacaEval 2.0 benchmark, we show that  pushes models trained with less preference data (e.g., 10% or 20%) to reach and even surpass the fully-trained one, without any additional training.
Furthermore,  also significantly improves off-the-shelf DPO/RLHF models and exhibits decent scalability across model sizes from 7B to 70B.
Our work demonstrates the efficacy of model extrapolation in exploiting LLMs' capabilities, suggesting a promising direction that deserves future exploration.






§ INTRODUCTION



Modern large language models (LLM) typically undergo additional fine-tuning to align with human expectations <cit.>, including both supervised fine-tuning (SFT) on demonstration outputs <cit.> and alignment training with human preference <cit.>.
Similar to the pre-training phase <cit.>, the alignment of LLMs can also be continuously improved by increasing data and training steps <cit.>.
However, in reality, alignment training is inevitably constrained by available resources and thus cannot grow indefinitely.
Suppose we have a moderately-trained LLM in hand, is it possible to further exploit its potential and cheaply acquire a stronger model?

We draw inspiration from the literature of model interpolation, also known as model/weight averaging.
It aims to integrate different models fine-tuned from the same base model into a unified one by interpolating between their weights <cit.>, relying on the mode connectivity of neural networks <cit.>.
Previous work showed that with the basic uniform interpolation (i.e., using the same interpolation ratio for all the model modules), the obtained new model usually achieves trade-off performance between the original ones <cit.>.
We similarly observe this phenomenon when we interpolate between an SFT model and a model further trained by direct preference optimization (DPO) <cit.> or reinforcement learning from human feedback (RLHF) <cit.>, as shown in Figure <ref>.



We are intrigued by another question:
If we treat the DPO/RLHF model as an intermediate result of interpolation between the SFT model and some stronger model, can we obtain this stronger model by reversely extrapolating from the former two models' weights?
If so, we can actually start with two relatively weaker models from the training process and straightforwardly obtain a stronger one.
As indicated by the gray arrow in Figure <ref>, this could also improve off-the-shelf already-aligned models, such as the many open-sourced LLMs on HuggingFace.

Based on the above motivation, we propose a simple method called  (model extrapolation) to boost LLMs' alignment with human preference (<ref>).
 assumes a medium-aligned model  can be interpolated from a less-aligned (weaker) model _w (e.g., the SFT model) and a better-aligned (stronger) one _s.
Then, we can directly obtain this stronger model _s by extrapolating from the weights of the two relatively weaker models  and _w, without any additional training on top of them.

Despite its simplicity, we demonstrate that  is quite effective in improving the alignment of various LLMs, as summarized in Figure <ref>.
Specifically, for standard DPO training, we show that  pushes the models trained with less data (e.g., 10% or 20%) to reach and even surpass the fully-trained one, as evaluated on the AlpacaEval 2.0 benchmark <cit.> (<ref>).
Furthermore,  also remarkably improves off-the-shelf DPO/RLHF models, by up to 6.8% on AlpacaEval 2.0 (<ref>), and manifests satisfactory scalability across model sizes from 7B to 70B.
Our work demonstrates model extrapolation as a promising method for boosting LLMs' alignment with human preference and better exploiting the capabilities of LLMs, which we believe deserves more future exploration.



§ METHODOLOGY





 §.§ Overview



Inspired by the observation in Figure <ref>, we make the following assumption: 


    A model  can be interpolated between a weaker model _w and a stronger model _s, which satisfy the relationship in terms of their alignment with human preference: _w <  < _s.



Specifically, we suppose the medium-aligned model  (parameterized by ) to be one that has been moderately trained for human preference alignment.
We also suppose the less-aligned weaker model _w (parameterized by _w) simply to be the SFT model used for initializing .
The above assumption suggests that there exists a better-aligned stronger model _s (parameterized by _s) and an interpolation coefficient γ∈ [0, 1] such that:

    = ( 1-γ ) _w + γ_s.

Here we consider the simplest form of uniform linear interpolation.
With the substitution of α = 1/γ - 1 ∈ [0, +∞), we can obtain the assumed stronger model _s by extrapolating from the weights of the relatively weaker _w and  (i.e., weak-to-strong extrapolation).
Our proposed  method is formulated as follows:

    _s = (1+α)  - α_w =  + α ( - _w) =  + α,

where the coefficient α serves as the hyperparameter that controls the length of extrapolation.
In practice, α can be cheaply tuned as a decoding hyperparameter (like the sampling temperature) on a development set with no model training involved.



 §.§ Insights on 



R5.7cm



    < g r a p h i c s >


 can be viewed as a “global gradient update” that moves the model weight along the direction of  in which the model's alignment with human preference is improved (measured by a reward score).






We first use Figure <ref> for an intuitive illustration of .
Specifically,  can be viewed as a “global gradient update”, based on the global weight change =  - _w from the initial _w to the final .
The weight change  indicates a direction in the parameter space, in which the model's alignment with human preference is improved (measured by a reward score).
Hence,  essentially aims to amplify the learned reward signal through the extrapolation α.

Based on the above illustration, we identify two prerequisites for .
First, the model  should have not yet been trained to its optimality.
This prerequisite is generally valid, as evidenced by the most powerful LLMs such as GPT-4 and Claude that are undergoing constant optimization for better alignment.
We will show in <ref> that even the open-source models that have been extensively trained for human preference alignment still have significant room for further improvement.

Second, also more importantly, the weight change  from _w to  should be of “high quality”, meaning it should as accurately as possible indicate an extrapolation direction in which the alignment can get improved. 
In mainstream preference alignment algorithms such as DPO or RLHF, this prerequisite can also be generally established, as  is initialized from _w (the SFT model) and is essentially trained to maximize the reward signal of human preference, either from preference data or reward models.
Nonetheless, the “quality” of  can vary depending on the training configuration of  and the capability of _w, as we will discuss in <ref> and <ref>.

Combining the two prerequisites, when the model  initialized from its SFT checkpoint _w has undergone moderate alignment training, it can potentially get better aligned by .
We will experimentally verify this in <ref> and <ref>.
However, other model combinations for _w and , such as a Base and an SFT models or two separately-trained RLHF models, usually cannot guarantee the second prerequisite. 
We will discuss this more in <ref> in conjunction with empirical results.



 §.§ Highlights


We underline the following appealing properties of :

    
  * Simplicity:
     is extremely simple and quick to implement.
    It merely involves performing extrapolation based on the weights of two checkpoints _s and , which can be accomplished within just a few lines of code.
    
  * Efficiency:
     needs no additional model training on top of _s and .
    The only hyperparameter α is also cheap to tune as no training will be involved.
    Moreover, we believe more efficient means of hyperparameter search can be developed in future work, as evidenced by the advances in adaptive model interpolation <cit.>.
    
  * Scalability:
      is in principle applicable to various LLMs, including those of large sizes or that have been extensively trained for human preference alignment.
    We will show in <ref> that  can improve off-the-shelf already-aligned models of varying sizes and capabilities.




§ EXPERIMENTS



We first demonstrate the effectiveness of  in a controlled setting, i.e., training the model  with less preference data, so we can know in advance that  still has room for further improvement (corresponding to the first prerequisite in <ref>).
We show that  endows the models trained using less data (e.g., 10% or 20%) with equivalent or superior performance to the fully-trained one.



 §.§ Experimental Setup





  
Models
To train models for human preference alignment, we refer to the alignment handbook[<https://github.com/huggingface/alignment-handbook>] <cit.>, a widely-used code base released by HuggingFace for alignment training of LLMs.
We follow the their setup of training the Mistral-based <cit.>  and  models <cit.>.
Specifically, we use the same preference dataset but varying data sizes to train the models.
We employ the same mainstream DPO <cit.> algorithm for alignment training, where the SFT model  is used as the reference model in DPO and also used for initializing the policy models.
We adopt the same hyperparameter configuration as  (see Appendix <ref>) and train all the models on 4 A100 80GB GPUs.
We use  as the fully-trained baseline (i.e., trained with 100% data).



  
Data
We use the same preprocessed UltraFeedback[<https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized>] <cit.> dataset for DPO training.
UltraFeedback is a large-scale preference dataset, containing diverse instructions and response pairs with GPT-4-annotated preference labels.
It has been popularly used by the open-source community for training aligned LLMs <cit.>.
The preprocessed version provided by HuggingFace contains 61K and 1K preference data in the training and development set, respectively.
Each data consists of an instruction and a pair of responses, with one labeled as preferred.



  
Evaluation
We evaluate the models on AlpacaEval 2.0 <cit.>, a leading and popular benchmark that assesses LLMs' alignment with human preference.
It contains 805 instructions representative of real user cases.
For each instruction, the response of the evaluated model is compared head-to-head with that of the GPT-4 baseline.
An evaluator based on GPT-4 (its version is  during our work) produces the probability of preferring the evaluated model, which provides an affordable and replicable alternative to human preference annotation.
Then, the win rate over the GPT-4 baseline is computed as the expected preference probability on all the 805 instructions.

Recently, AlpacaEval 2.0 has introduced the new length-controlled (LC) win rate metric <cit.>, which aims to alleviate the length bias of the GPT-4 evaluator (i.e., the prior preference toward longer responses) <cit.>.
According to <cit.>, the LC win rate metric currently has the highest correlation (a Spearman correlation of 0.98) with real-world human evaluation <cit.>, which consolidates the reliability of AlpacaEval 2.0 evaluation.

For , we decide the optimal α based on the performance on the UltraFeedback development set, as evaluated by an open-source reward model[<https://huggingface.co/weqweasdas/RM-Mistral-7B>].
It ranks among the top on RewardBench[<https://huggingface.co/spaces/allenai/reward-bench>] <cit.>, a leaderboard that assesses the performance of reward models.
More importantly, this reward model is not involved in either preference annotation or RLHF training of all the models we experiment with in this work, thus reducing the risk of reward hacking.
In our experiments, we also report the average scores produced by the reward model on the 805 AlpacaEval 2.0 instructions as a reference.



 §.§ Results





R5.2cm



    < g r a p h i c s >


The “quality” of  and the effectiveness of  can vary depending on the training configurations of .
Here, _2 indicates a superior extrapolation direction to _1.






In Table <ref>, we show the performance of the models trained with less (10% and 20%) preference data as well as the results of further applying  on top of them.
As expected, training with less preference data results in lower-tier performance, as indicated by their LC win rates on AlpacaEval 2.0.
For instance, compared to the 17.3% of using 100% data, using 10% and 20% only achieves the performance of 10.4% and 12.9%, respectively.
However, after applying , using only 10% data can achieve competitive performance to the fully-trained model (16.3% vs. 17.3%), while using 20% of the data already achieves beyond that (21.3%), giving a remarkable advantage of 21.3% - 17.3% = 4%.

We also observe that the model trained with 20% data obtains a greater improvement from  than that trained with 10% data (8.4% vs. 5.9%).
It implies that the former gives a superior extrapolation direction  to the latter, as illustrated in Figure <ref>.
However, the “quality” of  is not simply correlated with the amount of data, as shown in Table <ref> where using 20% data slightly outperforms using full data when both applying  (21.3% vs. 20.2%).
This is because the increasing size can also amplify the biases in the preference data, which becomes more likely to be learned by the model  as shortcuts.
We next analyze the impact of 's training configuration on  in detail.



 §.§ Analysis





  
Comparison between Data Sizes
Figure <ref> presents the reward scores and output lengths on the UltraFeedback development set versus the extrapolation coefficient α values in .
We have two main observations.
Firstly, for the models  trained with different data sizes, the optimal α of  varies and generally decreases as the data size increases, as indicated by the vertical dashed lines in the left part of Figure <ref>.
This is because the larger data size usually leads to the more thorough convergence of training, even if only to a local optimum, which naturally narrows the viable range of the extrapolation coefficient α.

Secondly, the global optimal reward score (6.08) achieved by  is obtained with a medium size (20%) of training data, rather than the smaller (5% or 10%) or larger (40%) ones.
For the former (5% and 10% data), although ExPO significantly improves the performance (from the reward score 3.13 to 4.79, and 3.59 to 5.82, respectively), the limited data still cannot provide an accurate , thus capping the potential performance after model extrapolation.
For the latter (40% data), we conjecture that the model may have learned the spurious features in preference data as shortcuts, especially the length bias[
The average lengths of the preferred and unpreferred responses in the UltraFeedback training set are 319 and 277 tokens, respectively.
] <cit.> where the preferred responses are usually longer.
As shown in the right part of Figure <ref>, for the model trained with 40% data, using a very small α results in a dramatic increase in the output length.
In this case,  becomes more likely to contain the spurious features, and in particular, the length bias can be amplified by model extrapolation.
But this does not lead to sustained improvement of performance, as shown in the right part of Figure <ref>, where the optimal rewards typically correspond to moderate output lengths ranging between 500 and 600.





  
Comparison with Hyperparameter Tuning 
As  can be viewed as a “global gradient update” (<ref>), we also compare with simply tuning the training hyperparameters.
Specifically, we use the same 20% training data but increase the learning rate or training epochs.
From the left part of Figure <ref>, we observe that increasing the two hyperparameters indeed somewhat improves the original reward score.
However, it is still inferior to the optimal reward score achieved by  under the default configuration, and also noticeably impairs the gains from model extrapolation (the peak points are lower than that of the default configuration).
This is probably because the model is overfitted to the training data and similarly learns the spurious features (like the length bias), thus failing to provide an accurate .
The overfitting issue can also be evidenced by the right part of Figure <ref>.
The models trained with larger learning rates or for more epochs become prone to generating longer outputs with a small α, but do not obtain noticeable reward improvement (the left part of Figure <ref>), implying that  is very likely to contain the spurious length feature rather than the true human preference.



Based on the above empirical analysis, we emphasize the critical role of  in .
Particularly, we show that the “quality” of  requires the appropriate choice of the training configuration for , including both the preference data and the training hyperparameters.
In the subsequent <ref>, we will further discuss the impact of _w's capability on the effectiveness of .



§ MODEL EXTRAPOLATION BOOSTS OFF-THE-SHELF MODELS



We next demonstrate the impressive efficacy of  in improving off-the-shelf already-aligned LLMs from HuggingFace, based on their SFT and DPO/RLHF checkpoints.
We particularly underscore the scalability of  across different model sizes and capabilities.



 §.§ Experimental Setup


When selecting open-source LLMs for experiments, we found that many well-known aligned LLMs, such as LLaMA-2/3 <cit.>, Gemma <cit.>, and Qwen <cit.>, do not release the corresponding SFT checkpoints.
Such an opacity hinders the feasibility of experimenting with these more representative models.
To facilitate reproducible research, we select the following open-source DPO/RLHF models that (1) have also publicly accessible SFT checkpoints, (2) have disclosed the training data, and (3) are popularly downloaded on HuggingFace or have been evaluated on the AlpacaEval 2.0 leaderboard:

    
  *  <cit.>, a LLaMA-2-based model suite.
    Since the three-sized models undergo the same SFT and DPO training processes (including both the data and configuration), they can serve as a reasonable testbed for the scalability of  across different model sizes.
    
  *  and  <cit.>, three Mistral-based models.
    They are trained with different hyperparameter configurations and on slightly different preference data.
    
  *  <cit.>, two Mistral-based models.
    They are trained by the RLHF algorithm with different reward models.

Similar to <ref>, we select the optimal α from [0.1, 0.2, 0.3, 0.4, 0.5] based on the performance on the UltraFeedback development set, as evaluated by the aforementioned reward model.



 §.§ Results





The results in Table <ref> demonstrate that  enhances the performance of the already-aligned LLMs, by impressive increases of up to 6.8% LC win rate and 10.5% basic win rate on AlpacaEval 2.0.
The improvement is made across LLMs of various capabilities, from the weakest  and  to the strongest  and .
It suggests that most open-source LLMs have not been aligned with human preference optimally, and  enables the further exploitation of these models' capabilities.

Specifically for the model suite Tulu-2, where the 7B/13B/70B models are trained using the same preference data and configuration, the enhancement by  nicely scales up with the increasing model size.
We conjecture that this is because the larger/stronger _w enables the better learning of the reward signal in the preference data or reward models, leading to both a stronger  and a more accurate , which together result in the greater improvement for  after model extrapolation.
Therefore, with the same preference data and training configuration, we optimistically expect the improvement by  can also scale up as the capability of _w increases.



 §.§ Discussion



Finally, we discuss the impact of model choices for _w and  on the effectiveness of .
In previous analyses and experiments, we choose _w as an SFT model, and  as the model further trained for human preference alignment on top of _w.
Can other types of model combination _w and , such as a Base and an SFT model, or two separately-trained RLHF models, be able to produce meaningful extrapolated models?
We experiment with the following types of combinations:

    
  * Base + SFT:  <cit.> as _w and  as .
    
  * SFT 1 + SFT 2 (trained from different base models):  as _w and  as .
    
  * SFT 1 + SFT 2 (same base):  <cit.> as _w and  as .
    
  * RLHF 1 + RLHF 2 (same base):  <cit.> as _w and  as .
    Note that it is not disclosed whether the two models are initialized from the same SFT model.





From the results shown in Figure <ref>, we find that extrapolating from two SFT models that are trained from different base models can easily lead to the model collapse, probably because they do not meet the requirement of mode connectivity <cit.>, namely, the same or close initialization.
For the combination of Base and SFT, extrapolation degrades the performance.
One cause is that the training from Base to SFT does not naturally reflect human preference, which is exactly why we need additional preference alignment training.
Another cause is that compared to the Base model, the SFT one acquires the instruction-following ability and is also adapted to specified input/output formats <cit.>.
 can amplify both learned features (<ref>), but the latter does not aid in alignment and may instead similarly lead to model collapse.
For the two separately-trained SFT or RLHF models, we find that they also cannot benefit from model extrapolation.
We speculate that this is because  is not initialized from _w, so the path in the parameter space from _w to  is not in the direction along which the reward signal can be amplified.
As illustrated in Figure <ref>, even though  (_2) has not yet achieved optimality on its own optimization path, it still cannot be improved in another direction of .
Overall, our method  is currently applicable to the combination of an SFT model _w and a model  further trained on top of the former, which is a very realistic combination choice, as modern LLMs that are trained to align with human preference are almost all initialized from their SFT checkpoints.



§ RELATED WORK





  
LLM Alignment
Modern LLMs are typically first pre-trained on massive textual corpora (resulting in a Base model) <cit.> and then trained to align with human expectations <cit.>.
The alignment process generally contains two stages.
In the first stage, an LLM is supervisedly fine-tuned (SFT) on demonstration outputs and learns to follow human instructions <cit.>.
In the second stage, the LLM is trained to learn human preference and assign higher probabilities to human-preferred outputs over the disfavored ones.
This is usually implemented in the fashion of reinforcement learning (RL) <cit.> or contrastive learning <cit.>, as exemplified by the reinforcement learning from human feedback (RLHF) <cit.> and direct preference optimization (DPO) <cit.> algorithms, respectively.
Similar to the scaling law in the pre-training phase <cit.>, recent work also revealed that the capabilities of aligned models can also be constantly improved by scaling up the amount of alignment data <cit.> and increasing the training steps or iterations for human preference alignment <cit.>.
However, the data and computation resources available in reality are always finite, which may prevent the full exploitation of models' capabilities.
Our work proposes the  method to boost LLMs' alignment with human preference in a simple, efficient, and scalable manner.



  
Model Merging and Interpolation 
Model merging is a recently focal technique for building powerful LLMs based on existing ones <cit.>.
It aims to integrate multiple models fine-tuned from the same base model into a unified one that retains the respective strengths <cit.>.
The simplest form of model merging is model interpolation, also known as model/weight averaging <cit.>, which builds upon the mode connectivity of neural networks <cit.>.
In practice, the uniform interpolation usually results in trade-off performance between the two original models, as observed in previous literature <cit.> and our experiments in Figure <ref>.
One approach to addressing this issue is to adaptively adjust the interpolation coefficient for different model modules (e.g., different model layers) <cit.>.
Our proposed  method (<ref>) has a similar idea of blending model weights to improve the model capability, but works under a distinct premise and goal.
Rather than integrating multiple strong models into a generalist, our method aims to use two relatively weaker models to produce a stronger model that can even surpass the limits of the fully-trained one (<ref> and <ref>).



§ CONCLUSION



We present , a simple method to boost LLMs' alignment with human preference.
By extrapolating from the weights of an SFT model _w and a further trained one ,  enables directly obtaining a better-aligned model without any additional training.
We demonstrate the efficacy of  across various LLMs, from those trained with limited preference data to the off-the-shelf ones from HuggingFace, where  manifests decent scalability across varying model sizes and capabilities.
Given its simplicity, efficiency, and scalability, we recommend  as a promising approach for better exploiting LLMs' capabilities, which deserves more future exploration.




  
Limitations & Future Work
Our work is limited by the public accessibility to the checkpoints of the SFT and DPO/RLHF models.
Thus unfortunately, we are unable to experiment with the more representative LLMs like LLaMA-2/3 <cit.>, Gemma <cit.>, and Qwen <cit.>.
We hope for more open-source efforts in increasing LLMs' transparency and accessibility.
Outside the scope of this study, there are several problems that may attract future research.
First, since  is based on the simplest uniform linear extrapolation (Equation <ref>, using the same α for all the model modules), future work may devise methods to adaptively search optimal α for different model modules.
Second, while we currently rely on an external reward model for searching α, future work may get rid of such reliance by resorting to the capability of the models  and _w themselves.
Third, although our work provides intuitive illustrations for  and empirically demonstrates its effectiveness, future work may establish theoretical explanations and analyses for its underlying mechanisms.
Finally, it would also be interesting to apply  to multi-modal LLMs like LLaVA <cit.> and other model architectures like Mamba <cit.>.




§ ACKNOWLEDGEMENTS


We thank the open-source community, including the HuggingFace, AllenAI, and Nexusflow teams, for promoting the transparency of LLMs by releasing model checkpoints and disclosing training details.
This work would not be possible without these efforts from the open-source community.
We thank Wei Xiong for releasing the reward models and for the valuable discussion.



plain










§ OPEN-SOURCE MODELS USED IN THIS WORK







§ MORE IMPLEMENTATION DETAILS



For model training in <ref>, we adopt the global batch size 128 and gradient accumulation steps 4.
We train the models on 4 A100 80GB GPUs, with ZeRO-3 offload <cit.> and gradient checkpointing for reducing GPU memory usage.
We set the learning rate to 5e-7, with the cosine scheduling and warmup ratio of 0.1, and use the AdamW <cit.> optimizer to train the models for one epoch.
For DPO, we follow  and set the coefficient β to 0.01.

For response generation, we employ the  library for high-throughput inference.
We use top-k (k=40) and nucleus sampling <cit.> (p=0.9) with a temperature of 0.7.
To avoid repetition in generated texts, we set both the factors of presence penalty and frequency penalty to 0.1.

For hyperparameter search, we manually try different values of α.
We use the obtained model to generate responses on the UltraFeedback development set, score the responses with the reward model, and choose the optimal α corresponding to the highest average score.
We list in Table <ref> the search range and the optimal α in our experiments.







