TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
Large Language Models;Importance Sampling;Preference Learning
Yes
No ethics review needed.
1. TIS-DPO(D) in Table 1 shows a significant improvement in performance far exceeds expectations, which seems a probable risk of overfitting. Can you conduct cross-validation to prove that there is no overfitting?
2. The experimental results of other methods on the Anthropic-HH dataset in Table 1 were not found in the cited papers, such as TDPO. Were all the comparative experiments were reproduced?
3. As quite a few hyperparameters have been introduced on top of DPO and TDPO, are there any sensitivity test on the newly added hyperparameters $k$, $\mu$, $L$ and $U$ in Eq. 19?

P.S.
- Zeng2024a and Zeng2024b are different versions of the same paper.
- Conflict usage of $\mu$ in Theorem 2, eq. 14 and eq. 19.
- TIS-DPO introduces a token-level importance sampling method that addresses the limitations of treating all tokens equally in Direct Preference Optimization (DPO), enhancing optimization efficiency.
- The proposed method is evaluated across multiple datasets, showing its robustness and versatility in different contexts and tasks.
- By using actual dataset with importance sampling instead of taking that dataset as an ideal dataset, the method is more practical for real-world applications where optimal data distribution is typically unavailable.
The paper presents TIS-DPO (Token-Level Importance Sampling for Direct Preference Optimization), a novel approach designed to enhance the optimization of Large Language Models by assignig different weights to tokens based on their predicted rewards. It aims to achieve unbiased optimization by approximating an ideal dataset where each token has equal expected rewards.

Rely on the previous work TDPO (Token-level Direct Preference Optimization), this method estimates token importance weights using differences in prediction probabilities between pairs of contrastive LLMs. Three approaches are explored for constructing these contrastive models:
- Guiding the original LLM with contrastive prompts.
- Training separate LLMs on winning and losing responses.
- Performing forward and reverse DPO training.

Extensive experiments are conducted to evaluate TIS-DPO on tasks related to harmlessness, helpfulness, and summarization. Results indicate that TIS-DPO significantly outperforms several baseline methods, demonstrating improvements in both safety and response quality.
- The proposed method involves steps for constructing contrastive LLMs and tokeniwse importance weights estimating, which increases computational costs and training time compared to simpler baseline methods.
- The Hoeffding inequality is applied in the proof of Theory 1, which requires that the summation variables (i.e. token rewards) be i.i.d., but tokens in a sentence dose not conform to the condition of being i.i.d. along with their observed rewards.
- TIS-DPO(S/D) distributes the sentence "value" to the token level and in addition estimates token importance via reward evaluator driven by the same sentence "value". The potential cost is that the method may be more sensitive to the noise in data annotation. For example, if there is a certain proportion of noise in the training set, let's assume that some of the triples $(x, y_w, y_l)$ are mislabeled in reverse order as $(x, y_l, y_w)$ which are common in the real world. Then TIS-DPO(S/D) may be more affected than DPO. Can you add 20% of noise (i.e. swapping the chosen and reject of a triple) into the training data and compare the performance degradation of TIS-DPO(S/D) versus DPO?
Yes
NA
No ethics review needed.
Could you explain why or provide some intuition as to why the the improvements is much better for safety alignment than helpful alignment?

Also, it would be great to have some empirical experiments to see whether TIS-DPO is leading to faster convergence (which may be  case based on the limited experiment setup shown in paper) or in fact better convergence, leading to meaningfully better aligned models. Could you expand on this, and how you may design experiments to illustrate it?
* Very novel idea of using contrastive LLM likelihood as token-level weights for importance sampling. 

* Very important area of research to improve the popular off-line alignment DPO algorithm. 

* Great insight about limitation of sequence level DPO, thus introducing the need for nuanced token-level optimization 

* Surprisingly good improvements against DPO baseline using token-level importance sampling based TIS-DPO. 

* Very thorough proof and derivation process for the TIS-DPO objective
The paper makes a key claim that "the most stable form of DPO loss occurs when tokens in winning and losing responses have identical expected rewards." And the best way to estimate the unavailable ideal DPO dataset is to use importance sampling. The paper did a thorough proof combining prior works of Zeng et al. to show a token-level DPO objective, and then enhance it with importance sampling to assign importance weight on each token of actual data distribution to approximate the data distribution of ideal equal importance DPO dataset. 

It then proposes 3 ways to estimate the token-wise weighting by contrasting LLM likelihood of 1. a prompt-based method. 2 SFT two models to desirable and undesirable. 3. DPO two models to desirable and undesirable. Experiment results show 1 doesn't work, and 3 works surprisingly well, significantly exceeding sequence level DPO baseline on HH, PKUSafe, and Summarization dataset.
* “Although Rafailov et al. (2024a) demonstrate that DPO possesses a certain degree of token-level interpretability,” It would be helpful to point to the exact section or figure of this analysis. 

* The strength of the method could benefit from more thorough experiment on stronger alignment datasets. HH and Summarization are commonly used but relatively poor quality data, compared with some newer alignment datasets like Ultrafeedback and HelpSteer2. 

* It is clear that TIS-DPO has significant strength in aligning models to safety and slight improvement to helpfulness. I am curious to see the dynamic of using TIS-DPO to align stronger models like Gemma2-9B or Llama-3-8B. 

Overall, great paper, excellent intuition, and solid experiment results.
Yes
No ethics review needed.
Please refer to the Weaknesses.
1. The paper presents a clear and well-structured method, TIS-DPO, which addresses a limitation of traditional DPO by considering the importance of individual tokens.

2. The experimental results show improvements over various baselines, indicating the practical value of the proposed method.
This paper introduces a new approach, Token-level Importance Sampling DPO (TIS-DPO), to improve the efficiency and effectiveness of Direct Preference Optimization (DPO) for Large Language Models (LLMs). The authors propose that optimal data for DPO should have equal expected rewards for each token in winning and losing responses. They use importance sampling with the original dataset to achieve unbiased optimization and estimate token importance weights using the difference in prediction probabilities from contrastive LLMs. Experiments demonstrate that TIS-DPO outperforms baseline methods in harmlessness, helpfulness alignment, and summarization tasks.
1. Lack of In-depth Analysis: The paper lacks a detailed analysis of why the three proposed methods for constructing contrastive LLMs can effectively measure token importance during training. Additionally, there is no detailed explanation for the significant differences in performance among these methods.

2. Fixed Token Importance: The token importance weights are generated before training and remain fixed throughout the process. However, as the model evolves during training, the importance of tokens may also change, which could affect the effectiveness of the method.

3. Scalability and Generalizability: The paper does not thoroughly address the scalability and generalizability of the proposed method. It is unclear whether every model would require its own set of token importance weights or if the one set of weights can be generalized to different models and tasks.

4. Limited Experimental Scope: While the method shows effectiveness in improving harmlessness and helpfulness, the experiments do not cover other critical areas such as code and mathematics, where token-level importance might be even more crucial. Further experiments in these domains would strengthen the validation of the method.
Yes
No ethics review needed.
- Why is the DPO result missing in Table 1?
- How many LLMs are needed in your training process? Is it 3?
- Will your proposed method be sensitive to sequence length in better or worse completions, where longer sequences may have higher rewards?
- Regarding the win rate, I understand that GPT-4 is the evaluator model - which model is being used as the competitor?
- In Table 1, is there any reason why the win rate increases significantly while the MT-bench score is not improved comparably?
- The paper proposes a new perspective of preference learning based on DPO by considering token-level importance.
- Nice appendix.
- The experimental evaluation is comprehensive, spanning 2 datasets, different metrics including MT-bench and win rates, and multiple baselines (DPO, PPO, IPO, TDPO, KTO) .
The paper proposes TIS-DPO (Token-level Importance Sampling for Direct Preference Optimization), which is built upon standard DPO by assuming that different tokens within a response have varying levels of importance. Specifically, the importance of tokens in the completion is estimated based on 2 separate reward models. The authors demonstrate that on two datasets the proposed method  outperforms baseline methods.
- Questionable Motivation. Even though we assume you can accurately estimate the token importance, the token-level DPO approach seems misaligned with how preference data is collected, where annotators make holistic judgments about complete responses, not token-by-token evaluations. 
- The accuracy and reliability of token-level importance estimation are not thoroughly validated 
- The approach assumes tokens can be evaluated independently, which ignores crucial context dependencies in language.
- Introducing reward models for token importance estimation seems to contradict DPO's core advantage of avoiding reward modeling. Should we compare against PPO, where the rewards are estimated?
 - The additional cost of training and maintaining contrastive LLMs for weight estimation is not analyzed. No discussion of the computational trade-offs compared to standard DPO. The scalability implications for larger models are not addressed
-  Introduction of new hyperparameter μ adds complexity. Limited analysis of hyperparameter sensitivity. No clear guidance for hyperparameter selection in different scenarios
- Missing analysis of how sequence length affects performance. No investigation of the relationship between token position and estimated importance.
- Table 2 presentation should have been corrected.
We propose TIS-DPO, a token-level importance sampling method for DPO that improves LLM alignment by assigning different weights to tokens based on their estimated importance.
@inproceedings{
anonymous2024tisdpo,
title={{TIS}-{DPO}: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=oF6e2WwxX0},
note={under review}
}
Direct Preference Optimization (DPO) has been widely adopted for preference alignment of Large Language Models (LLMs) due to its simplicity and effectiveness. 
However, DPO is derived as a bandit problem in which the whole response is treated as a single arm, ignoring the importance differences between tokens, which may affect optimization efficiency and make it difficult to achieve optimal results.
In this work, we propose that the optimal data for DPO has equal expected rewards for each token in winning and losing responses, as there is no difference in token importance. 
However, since the optimal dataset is unavailable in practice, we propose using the original dataset for importance sampling to achieve unbiased optimization. 
Accordingly, we propose a token-level importance sampling DPO objective named TIS-DPO that assigns importance weights to each token based on its reward.
Inspired by previous works, we estimate the token importance weights using the difference in prediction probabilities from a pair of contrastive LLMs. We explore three methods to construct these contrastive LLMs: (1) guiding the original LLM with contrastive prompts, (2) training two separate LLMs using winning and losing responses, and (3) performing forward and reverse DPO training with winning and losing responses.
Experiments show that TIS-DPO significantly outperforms various baseline methods on harmlessness and helpfulness alignment and summarization tasks. We also visualize the estimated weights, demonstrating their ability to identify key token positions.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models Importance Sampling Preference Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2204101b85406cefe0a1cfc497eb924f34a318a6.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission