HyperDPO: Hypernetwork-based Multi-Objective Fine-Tuning Framework
Direct Preference Optimization;Multi-Objective Optimization;Hypernetwork;Alignment
Yes
No ethics review needed.
In the experiment, how would you describe the significance of the improvements?
Fine tuning with multiple objectives is an important problem with many real world applications such as alignment of LLMs with multiple goals in helpfulness, trust and safety etc.
This work extends Direct Preference Optimization (DPO) to the multi-objective settings. 
This is achieved by replacing the BTL model to PL model that can handle list-wise rankings. 

The work also profiles the Pareto Front with hypernetworks. The Pareto Front is a set of solutions involving trade-offs such that no objective can be improved further without hurting other objectives.
The technical contribution of this work is quite incremental. It is a combination of known techniques.
- The generalization due to PL was originally proposed in Liu 2024.
- Using a hypernetwork to profile the Pareto Front was proposed in Navon 2020.
Yes
No ethics review needed.
Please refer to the weaknesses.
1. This work extends the DPO problem into the multiple-objective setting, which is important for the alignment of LLMs. 
2. The proposed method connects the learning-to-rank problem and the LLM alignment. This might be instructive for developing more effective alignment methods.
This paper focuses on Direct Preference Optimization (DPO) with multiple objectives. On top of the Plackett-Luce (PL) model, it proposes a generalized DPO problem that considers the Pareto front of multiple objectives. Afterward, by applying the hyper-network techniques, the authors propose a corresponding optimization algorithm through one-shot training, and offer a post-training control over the temperature hypermeter.
1. The main concern is that the contributions of this work might be overclaimed. After formulating the multi-objective DPO problem, this work uses Ruchte & Grabocka's algorithm to profile the Pareto front. 
2. Another contribution of this work is Prop. 3.1, which shows a linear transformation property between the optimal solutions under different $\beta$. However, according to the proof, this property is based on $\mathcal{L}\_{ListNet}$ (Eq. (12)) instead of $\mathcal{L}\_{Hypernet}$ 
 (Eq. (13)). Therefore, such a property ignores the Pareto fronts.
3. As one of the main contributions, the linear transformation property should have been validated in the experiments. Besides, how to apply this property in real scenarios should be further discussed.
Yes
No ethics review needed.
Here are my questions that requires further clarification from authors. 
1. Why the results of MO-DPO are not reported in the Table 2?
2. How many times/random seeds that the experiments are conducted? In the results, HV and training time are not reported with STD, while the average main score is reported with STD, why here is a difference?
3. Can HyperDPO be extended to non-linear utility functions in the context of MOO?
This work propose the HyperDPO method to address the MOFT problem. The technical contribution of this work is to incorporate a hypernetwork (through Hyper Prompt Tuning (HPT)) to DPO, such that the weights of objectives can be considered when fine-tuning LLM. While both Hypernetwork and DPO have been recently studied, the integration of both methods, espcially, using weights as inputs to the hypernetwork, seems to be a noval integration. In terms of writing, this paper is clearly written. In addition, it illustrate the theoretical foundation with clear detailed mathematical proofs. As DPO is a efficient way to fine-tune LLM, while source code is not available at the moment, HyperDPO might be an efficient way to fine-tune LLM under multi-objective setting.
This paper aims to tackle the multi-objective fine-tuning (MOFT) problem, where LLMs are trained towards satisfying multiple objectives simultaneously. As a main contribution, a method named HyperDPO is proposed. HyperDPO extends the Direct Preference Optimization (DPO) method with a hypernetwork that take weights of objectives and the input data. In this work, authors assumes the total utility (under MOO setting) is calculated via linear scalarization.  To demonstrate the effectiveness of HyperDPO, authors select three baseline methods, DPO-LS, DPO Soup, and MO-DPO. The authors select the hypervolume (HV) as the evaluation metric to compare the Pareto front. Experiments are conducted on two tasks, namely learning to rank and LLM alignment. Authors reported the results of fine-tuning GPT-2 and Alpaca-7B. The results demonstrate that HyperDPO could obtain higher HV with less training time, compared with baseline methods.
This paper is overall well-written, however, the results could be further enhanced to better support the claim.  The first issue is that the results only include the hypervolume metric as the evaluation of pareto fronts (PF). Other important metrics including sparsity is not reported. In authors' experiments, they fine-tune two models, GPT-2 and Alpaca-7B. While results demonstrate HyperDPO perform better for both experiments, results in Figure 4 show that the performance of evaluated methods seems to be related to the base model. For example, while the PF of DPO-LS on GPT-2 is dominated by the one of HyperDPO, DPO-LS on Alpaca-7B performs quite close with HyperDPO. Similar pattern also reflects on HV and IQR. Therefore, it is not certain whether hyperDPO is generally better than other methods on different LLMs. More comparisons and analysis on when HyperDPO is a better method are expected. 

In terms of citations, the inline citations could be improved. Generally, "(author, year)" in a sentence could not be used as a word.
Yes
No ethics review needed.
No ethics review needed.
The concerns I list in "weaknesses" make it hard to hard to judge the overall contribution and soundness of the work. Consequently, I gave low scores for these and the overall rating so far. I plan to revisit the scores after the author response.

* "hypernetwork"

I find *hypernetwork* not the correct term for the method you describe, given its use in the literature so far (see above). If you agree, please edit the manuscript accordingly. Otherwise, in case you have a different definition of hypernetwork in mind, please provide a reference for this. 

* Contribution

Could you please clarify which aspects/parts of HyperDPO exactly you consider new scientific contributions? Or is it more of a systems-type contribution, where the novelty lies in the combination of existing components?

* Experiments

Related to "Contribution": what is the exact scientific claim for which the experiments provide evidence? 

* Proposition 3.1

I have problems understanding the proof of Proposition 3.1 (page 16). Specifically, how does it follow from a comparison of $s_{\theta,c\beta}(y,w|x)$ with $s_{\theta,\beta}(y,w|x)$ that the numerators of both expressions have to be the same? Or do I misunderstand something else here?
- timely and relevant topic

The submission addresses the largely unexplored problem of how LLMs can be fine-tuned in a multi-objective way, i.e. a network should be adapted with respect to not just one but multiple, potentially contradictory, targets and the trade-off between should be adjustable at prediction time. This matches well to real-world tasks, where LLMs should perform well in different metrics, e.g. utility, safety and fairness, and one cannot afford to train individual models for any desired trade-off. 

- theoretical derivations as well as practical experiments

The submission presents mathematical derivations of their main components as well as experimental evaluations and ablation studies (but see below for shortcomings in presentation). 

- experiments show improvement over baselines

The experimental evaluation shows that the proposed method tends to find Pareto-better solutions than baselines, and that it also takes less time for training (but again, see below).
The submission describes a system for multi-objective model finetuning, where a base model is trained on several new objectives from listwise ranking data. The main component is an input-conditioned network that takes additional inputs weights for the different objectives, and it trained with a random mixture over these. At prediction time, the weights can be set to achieve different trade-offs between objectives. Experiments for learning-to-rank and LLM-alignment show improved performance compared to baseline approaches. The supplemental material provides further (ablation) studies as well as an additional formulation in which not only the objective weights but also the temperature of their softmax can be adjusted at prediction time.
While overall I believe the submission has merits, I found its presentation to leave several aspects unclear or even misleading. 

1) the manuscript uses the term *hypernetwork* throughout the text, but the system it describes is not actually hypernetwork-based. A hypernetwork is a network that outputs the parameters of another network [Ha et al, "Hypernetworks", ICLR 2017], see also e.g. the recent survey [Chauhan et al, "A brief review of hypernetworks in deep learning", AI Review 2024]. In contrast, the system described in the submissions uses an input-conditioned network as in [Ruchte & Grabocka. "Scalable Pareto front approximation for deep multi-objective learning", ICDM 2021]. These two approaches are simply different, in their theoretical properties (see e.g. [Galanti & Wolf, "On the Modularity of Hypernetworks", NeurIPS 2020]) as well as in practice.

2) From the writing I was not able to extract the core technical contribution. Section 1.1 list the HyperDPO system as the main contribution, but that consists of several parts which existed (at least in similar form) previously. Does the main contribution lie in the differences? Or in the combination of techniques? Or in their application to the finetuning task? 

To make my concern clearer: I believe for the reader it is most helpful if the "Preliminaries" section bundles prior work, and the "Methodology" section introduces the new scientific contribution on top of it. In the manuscript, the Methodology section indeed it introduces the multi-objective finetuning task, which I assume is a new formulation that can be seen as a generalization of standard LLM finetuning (thought as a special case of general multiobjective learing). But the Section discusses the classical Plackett-Luce model, and rederives the ListNet loss from [Cao et al, 2007]. Next, input-conditioned multi-objective learning [Ruchte & Grabocka, 2021] is introduced. Inserting the loss into the formulation and ultimately arrives at HyperDPO. It would be good to mark more clearly, which of the intermediate steps the authors consider scientific contributions and which are just provided for the convenience of the reader. The post-training control (via Proposition 3.1) appears new, but its relevance is not made clear here, and I have questions about the proof. 

Other contribution mentioned in Section 1.1 are the Hyper Prompt Tuning, which however is first described in the experimental Section 4.2 at much lower level of detail than I would expect from a core contribution. The final contribution of a "temperature hypernetwork" is even only discussed in the supplemental material. 

3) The experiments are only partially convincing. 

The learning-to-rank experiments of Section 4.1 show promising results, but the settings is not the one used for motivating the method. The setup of the LLM alignment task is described rather briefly, e.g. the description of dataset, objectives and optimization process are too short to allow for reproducibility. The visualized Pareto curves and hypervolume values look promising, but absolute differences are small. In the LLM experiments, no error bars are provided, and generally statistical significance is not established or discussed. The training times would need more explanation or analysis as well. The times for the baselines as well as HyperDPO will depend on several factors, such as how many w-vector to sample and the total number of epochs. A justification is needed why the provided analysis is fair.

On a more fundamental level I am unsure what claim the experimental evaluation support (also related to 2). Section 4 compares the proposed input-conditioned network to simpler baselines, but [Ruchte & Grabocka, 2021] already did that (on different tasks, of course). Can the objective (13) be expected to be different from other MO tasks, so new evidence is required? Then experiments that highlight how (13) relates to alternative objectives would make the evidence stronger. If the multi-objective way of training is indeed the core contribution, I would have expected a comparison to other multi-objective methods from the literature, in particular actual multi-objective hypernetworks, such as [Navon et al, "Learning the Pareto Front with Hypernetworks", ICLR 2021] or [Hoang et al, "Improving Pareto Front Learning via Multi-Sample Hypernetworks", AAAI 2023]. 

(Please note that I do not ask for new experiments of this type to be reported in the author response)

Additional comments:
- the reference section has some issues: e.g. acronyms and conferences are often incorrectly capitalized, arXiv versions are cited where published version exist, some publications years seem incorrect (e.g. NeurIPS 2024 proceedings don't exist yet).
- floating point notation, such as "1.473e-01", should be change to mathematical notation, "0.1473e"
We propose HyperDPO, an efficient and versatile hypernetwork-based multi-objective fine-tuning framework, proving effective in large-scale ML tasks like Learning-to-Rank and LLM alignment.
@inproceedings{
anonymous2024hyperdpo,
title={Hyper{DPO}: Hypernetwork-based Multi-Objective Fine-Tuning Framework},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=qBKA2844I4},
note={under review}
}
In LLM alignment and many other ML applications, one often faces the *Multi-Objective Fine-Tuning (MOFT)* problem, *i.e.* fine-tuning an existing model with datasets labeled w.r.t. different objectives simultaneously. To address the challenge, we propose the *HyperDPO* framework, a hypernetwork-based approach that extends the Direct Preference Optimization (DPO) technique, originally developed for efficient LLM alignment with preference data, to accommodate the MOFT settings. By substituting the Bradley-Terry-Luce model in DPO with the Plackett-Luce model, our framework is capable of handling a wide range of MOFT tasks that involve listwise ranking datasets. Compared with previous approaches, HyperDPO enjoys an efficient one-shot training process for profiling the Pareto front of auxiliary objectives, and offers flexible post-training control over trade-offs. Additionally, we propose a novel *Hyper Prompt Tuning* design, that conveys continuous weight across objectives to transformer-based models without altering their architecture. We demonstrate the effectiveness and efficiency of the HyperDPO framework through its applications to various tasks, including Learning-to-Rank (LTR) and LLM alignment, highlighting its viability for large-scale ML deployments.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Direct Preference Optimization Multi-Objective Optimization Hypernetwork Alignment
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/23723a7149d98d13c41d4adb77ca7f1db8a65a8b.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/30e4500d914e335362f12002ecfd09f4518720fc.pdf
HyperDPO: Hypernetwork-based Multi-Objective Fine-Tuning Framework
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission