Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
large language model;alignment;human preference;variational inference
Yes
No ethics review needed.
1. Let's first assume the results in Table 1 is reliable. Why are some results of using LPC is much worse than not using LPC? Have you analyzed about this? Any ablation results you can share?
1. The mathematical derivation looks fancy.
2. The paper is well-written.
3. The idea of using latent variables for various alignment objectives is intuitive and inspiring.
This paper deals with an important problem in the alignment research: modeling of complex human preferences and proposes a method using latent discrete variables, named LPC, to solve this problem. Authors first conduct extensive objective derivations and further conduct experiments on three alignment methods: DPO, SimPO, and IPO.
1. Although the mathematical derivation is elegant, but the experimental design and results do not demonstrate that this method is truly effective.
2. This work is like other works in past research in NLP using latent variables, presenting valid theoretical derivations but limited effectiveness, along with questionable robustness and training stability.
3. Although I really like the idea of using latent variables for various alignment objectives modeling, the architecture is too complex and seems unstable. I really worried about the practicability of this method. The authors try to prove the generalization of this method by using it on three alignment methods: DPO, SimPO, and IPO, but it is hard to ignore that the mathematical derivation is necessary for every alignment algorithms that have different alingment objective functions. This significantly undermines the practical applicability of this method. If others want to apply this method to other alignment algorithms, they would first need to perform complex mathematical derivations. Moreover, there is no way to guarantee the correctness of these mathematical derivations for other alignment algorithms.
4. The design of experiements are not good. Choosing to evaluate on the three tasks (commonsense reasoning, mathematical reasoning, truthfulness) are not convincing. I find the experiements fail to verify the effectiveness of LPC on modeling the "COMPLEX" human intentions, especially in the contradicting scenario, e.g., the scenario where we need to guarantee both helpfulness and harmlessness.
5.  The main experimental results in Table 1 are reward scores. However, the paper does not even mention what kind of reward model it is using, let alone any details that can help determine the trustworthiness of the evaluation results. If the reward model chosen in the paper is not reliable at all, the results presented in Table 1 are meaningless.
Yes
No ethics review needed.
See the weaknesses above.
1. This paper is well written and easy to follow.
2. This paper have extensive experiments to verify the effectiveness of their proposed method.
3. The idea of using latent code to model complex human preferences is interesting.
This paper addresses the challenge of aligning large language models (LLMs) with complex human preferences that may be conflicting across tasks and populations. The authors propose Latent Preference Coding (LPC), a framework that models implicit preference factors using discrete latent codes without relying on predefined reward functions. LPC integrates with various offline alignment algorithms and infers underlying preferences directly from data. Experiments demonstrate that LPC improves performance across multiple benchmarks and enhances robustness against noisy data, offering a promising approach for more versatile LLM alignment.
1. The time computational cost of this method during training process is unclear compared to existing methods like DPO, SimPO.
2. While the idea of learning a prior distribution for latent codes is straightforward and its effectiveness has been demonstrated empirically, it is essential for the authors to further explain how this prior network can capture complex human preferences like from some theoretical proof or previous work.
3. The authors state that the advantages of latent preference coding are mitigating the notorious posterior collapse issue and achieving better interpretability. It would be beneficial to conduct an ablation study using continuous representations to learn human latent preferences.
Yes
No ethics review needed.
1. In Equation (10), is \( h_{x,y} \) at each timestep summed with \( z \)?
2. The purpose of the flipped-label experiment is unclear. Could you clarify how LPC mitigates the significant drop in win rate in such cases? Additionally, how would the results vary if the flip rate were 10% or 90%?
1. The idea behind LPC is interesting. The framework eliminates the need for manually defined reward functions and predefined combination weights, a limitation present in many other methods.
2. Experiments were conducted using three different base models and three alignment algorithms, showcasing LPC’s broad applicability.
Latent Preference Coding presents an innovative approach that leverages discrete latent variables to model the underlying factors behind human preferences. By implicitly capturing these factors without relying on predefined objective weights or manually defined reward functions, LPC provides a unified framework compatible with various alignment algorithms.
1. The training settings for the discrete latent space are not clearly specified.
2. The TSNE visualization lacks distinct clusters, making it difficult to discern what the latent code has actually learned.
3. Table 1 does not include results for MMLU, which is a benchmark on the OpenLLM leaderboard.
4. The scores reported in Table 3 are generally low. For example, fine-tuning on Llama3-8B-Instruct using SimPO often yields scores of 20+ as reported in SimPO paper. Could there be differences in settings affecting these results?
5. Additional benchmarks, such as MT-Bench or Arena-Hard, would enhance the comprehensiveness of the evaluations.
Yes
No ethics review needed.
1. How do human preferences and discrete potential coding correspond in the LPC framework? How do we distinguish different preference elements?

2. What is the specific task division between a priori and a posteriori network, and what is the purpose of designing them? How do they operate when there are no preferences? How are the cues designed?

3. What is the performance of LPC in dealing with extreme cases (e.g., extremely noisy data or cases where preference labels are entirely missing)?
1. The overall framework of the paper is relatively novel, capturing the multifaceted nature of human preferences with the help of discrete latent variables.
2. The constructed LPC framework can seamlessly integrate multiple existing alignment methods with high applicability and scalability.
This paper proposes LPC for solving the problem of aligning large language models with human preferences. LPC captures the multifaceted nature of human preferences through the introduced discrete latent variables. In addition, LPC can be integrated with a variety of offline preference algorithms, including DPO, IPO, SimPO, and so on. A unified approach is provided to represent complex preference structures, which in turn enhances the performance of LLMs.
1. The discussion and analysis of existing work is relatively brief.

2. Generalization ability is unclear: While LPC performs well on noisy data, the validation of its ability to generalize across tasks is more underdeveloped and does not demonstrate its adaptability to a broader range of tasks.

3. Insufficient discussion of LPC complexity: LPC methods' computational cost and implementation complexity in practical applications are not discussed in detail, especially the computational resource requirements for large-scale models.
@inproceedings{
anonymous2024latent,
title={Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=bgk4O69SoL},
note={under review}
}
Large language models (LLMs) have achieved remarkable success, yet aligning their generations with human preferences remains a critical challenge. Existing approaches to preference modeling often rely on an explicit or implicit reward function, overlooking the intricate and multifaceted nature of human preferences that may encompass conflicting factors across diverse tasks and populations. To address this limitation, we introduce Latent Preference Coding (LPC), a novel framework that models the implicit factors as well as their combinations behind holistic preferences using discrete latent codes. LPC seamlessly integrates with various offline alignment algorithms, automatically inferring the underlying factors and their importance from data without relying on pre-defined reward functions and hand-crafted combination weights. Extensive experiments on multiple benchmarks demonstrate that LPC consistently improves upon three alignment algorithms (DPO, SimPO, and IPO) using three base models (Mistral-7B, Llama3-8B, and Llama3-Instruct-8B). Furthermore, deeper analysis reveals that the learned latent codes effectively capture the differences in the distribution of human preferences and significantly enhance the robustness of alignment algorithms against noise in data. By providing a unified representation for the multifarious preference factors, LPC paves the way towards developing more robust and versatile alignment techniques for responsible deployment of powerful LLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language model alignment human preference variational inference
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f5a7e81670fd62f3b74b087dea1ffdee72f67d86.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission