Declarative characterizations of direct preference alignment algorithms
neuro-symbolic modeling;logic;preference learning;RLHF
Yes
No ethics review needed.
- Could you please provide a toy example, and an analysis of this example for each of the introduced contribution of the paper?
- What could be an empirical setting, where your proposed framework could be investigated?
- Could you please elaborate on "While this can be remedied by modifying the SL to involve counting multiple formulas as in Rescher (1967), we instead define a relational structure called a preference structure that allows us to capture the semantics of losses in a modular fashion using a single propositional formula coupled with auxiliary constraints. Such a structure, which is based on a novel construction in propositional logic, will later make it easy to cleanly characterize different DPA losses and devise new variants through manipulation to their constraints." --- It is not clear to me, why the new method you propose is motivated by this. Aren't you compiling your loss to an SL as well? what are the main differences?
- The authors systematically investigate an interesting and relevant problem of semantically understanding and constructing preference alignment loss functions.
- The paper proposes a simple  algorithm to compile DPA loss to a logical expression
- They introduce a logic for modeling preferences, that allows to create new loss functions for a given preference structure.
The authors investigate losses for preference alignment. They analyze existing DPO functions with an aim to symbolically extract their semantics, and also investigate compiling DPO losses from symbolically given knowledge on preference structures.
- This could be due to my relative lack of expertise in the field of the paper. But the lack of any running example, and experiments, makes the presentation quite divorced from the original motivation of preference alignment in AI models.
- Hence, the larger potential utility of the framework is not clear to me.
Yes
No ethics review needed.
- **Q1:** I didn't get the number 4 in the double exponential form, i.e. $4^{2^n}$. Could you please be more specific or provide more context?
- **Q2:** Algorithm 1 uses "Simplify" to minimize (propositional formulas). As far as I am aware, the problem of minimizing propositional (logic) formulas by preserving equivalence is NP-hard. Thus, am I missing something here? Please clarify.
- **S1:** Precise related work.
- **S2:** Strong mathematical formulation, which I greatly enjoyed.
- **S3:** New perspective on neural-symbolic interplay.
- **S4:** Great presentation and structure of the work.
The paper introduces a novel framework for analyzing the space of DPA-like losses by systematically deriving a symbolic, logical expression characterizing its semantics. Different DPA losses can be derived thanks to such mapping, thus providing a comprehensive overview of the landscape. The authors do an excellent job of formalizing their framework, which gives researchers a new, fresh perspective on how to analyze the plethora of what they call successful DPA losses in the literature. Intuitively similar losses found in the literature can now have a place where analysis is done through formal methods if the paper is accepted. Be aware that my evaluation could have been overly optimistic, given my expertise; nonetheless, the paper should be accepted for me based on the novel contribution and rigorous mathematical treatment. As such, I would give a 7 (disabled by the system) instead of an 8, while a 6 seemed too pessimistic.
- **W1:** Given my unfamiliarity with the literature on DPA and similar, there seem to be exponential blowups. For example, to compute Eq 2, the weighted model counting must enumerate all $2^n$ propositional models ($\mathbf{w}$).
- **W2:** I would have given more context around the notation $y_w \succ y_l$ for those unfamiliar with the literature like me. I have interpreted $\succ$ as "the winner is preferred to the looser", but again, this may be the wrong interpretation. Either way, please clarify.
- **W3:** Some references are wrong. For instance, the reference to Table 7 does not exist in line 416; maybe it should be Table 5 (from the Appendix). Similarly, the caption of Table 4 references Algorithm 5.1, but 5.1 is a Section.

Minors:
- line 176: an a variant --> and a variant
- line 385 (and similar): Before the proposition's statement, the notation does not use parentheses around the superscripts, while the statement uses them (also in the Appendix). Please fix it for better readability.
- line 390: prefrence --> preference
- line 476: CEUNL --> CEUnl
- line 482: is much to learned by --> is much to *be* learned by?
- line 522: exactly these the losses --> exactly the losses?
- line 728: Table 5 is referred to in the proof, but Table 5 is the one with the translation rules
   - As a meta-observation, double-check all the other references (`\ref{}`). Some are okay, but some are not (and I tried to point to some of those).
Yes
No ethics review needed.
1. Can you provide additional real-world examples of applying formalized DPA losses as discrete reasoning problems to clarify the framework’s practical relevance?
2. While you suggest this framework aids in finding improved loss functions, is there empirical evidence or additional experiments comparing these new losses to existing DPA losses?
3. How does the complexity of your decompilation procedure scale with larger models or more complex DPA losses? This would clarify its feasibility for large-scale applications.
The paper’s strengths include a novel technique that clarifies the semantics of DPA losses through logical formulas over Boolean propositions, addressing a key gap in understanding what these losses capture and how they interrelate. The innovative decompilation procedure enables deriving symbolic formulas for complex loss functions, offering a structured view into the DPA loss landscape. This approach empowers practitioners to systematically explore new loss functions, advancing both theoretical insights and practical tools for human-AI alignment.
This paper addresses the challenges in understanding and developing direct preference alignment (DPA) loss functions, commonly used to align large language models with human preferences. Current DPA methods, like DPO, show promise but lack a conceptual framework for analyzing and differentiating variants. To address this, the authors propose a formalism to characterize DPA losses as discrete reasoning problems, enabling a systematic derivation of symbolic expressions that define their semantics. This approach reveals the extensive structure within the DPA loss landscape, showing a doubly exponential number of definable variations based on unique predictions. The framework highlights formal relationships, such as logical entailment and monotonicity, providing insights into efficiently exploring new DPA losses by modifying and testing established loss functions. This formal view aims to guide further development in human-AI alignment research.
The paper could be strengthened with more real-world examples demonstrating the practical relevance of formalizing DPA losses as discrete reasoning problems. While the formalization offers a structured approach, it’s primarily theoretical, and its effectiveness remains unproven. The claim that new losses derived from this framework are superior is speculative, as no substantial evidence is provided to show that these new losses outperform existing ones. Further empirical validation is needed to confirm the benefits and applicability of these new loss functions in real-world settings. Additionally, the hypothesis around the "constrainedness" of a loss function as a predictor of its success is only preliminary, requiring more in-depth experimentation.
Yes
No ethics review needed.
-How important is it for the formalization to follow the assumption about DPA losses suggested by Tang et al. (arxiv 2024)? As far as I am aware, that article was not yet peer-reviewed and it is hence not clear to me how well-established this particular formalization of DPA losses is (and the submission does not attempt to justify this on its own).
The submission is well-written and the research direction seems sound.
The submission devises a new framework that can translate DPA loss functions into a logical characterization. As a byproduct of that framework, the authors establish a double-exponential upper-bound on the number of different DPA loss functions that can be represented in their model. The submission is fairly well-written, but it is clearly targeted at experts in the specific subfield as, e.g., the introduction assumes knowledge of DPA and DPO and is not very accessible to the general ICML audience. 

I have two main concerns:

1) The contribution is comparatively weaker than what one would expect from a typical ICML paper. Of course, contributions can come in different forms, but aside from new ideas fully-fledged ICML papers typically support these ideas with a technical contribution, such as experimental evaluations or non-trivial mathematical proofs. The submission, however, lacks the former as well as the latter (all statements are established as direct observations or via essentially trivial proofs which do not seem to require novel insights or ideas). 

2) While the submission identifies a large number of mathematically well-defined DPA loss functions, it is less clear what is the envisioned contribution to the ICML community. After all - unlike when one establishes, e.g., theoretical upper/lower bounds or settles the computational complexity of fundamental problems - here there was no doubt that many different DPA loss functions exist. The authors claim that their results can be used to "map out" and find loss functions with better properties, but that is left entirely for future work (see Section 6.2).

Given these concerns, I feel that the submission would benefit from diving deeper into the topic and presenting a more well-rounded and thorough contribution. Note that space constraints are not a major factor here yet: the current submission spends a lot of space discussing tangential remarks which could easily be partly or wholly moved to the Appendix (see, e.g., the end of Subsection 6.1).
See the Summary.
Formal characterizations of direct preference alignment algorithms.
@inproceedings{
anonymous2024declarative,
title={Declarative characterizations of direct preference alignment algorithms},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=7UKHNQIErp},
note={under review}
}
Recent direct preference alignment algorithms (DPA), such as DPO, have shown great promise in aligning large language models to human preferences. While this has motivated the development on many new variants of the original DPO loss, understanding the the differences between these recent proposals, as well as developing new DPA loss functions, remains a formidable challenge given the lack of a technical and conceptual framework for reasoning about the underlying semantics of these algorithms. In this paper, we attempt to remedy this by formalizing DPA losses in terms of discrete reasoning problems. Specifically, we ask: Given an existing DPA loss, can we systematically derive a symbolic expression that characterizes its semantics? We show how this formal view of preference learning sheds new light on both the size and structure of DPA loss landscape, making it possible to not only rigorously characterize the relationships between recent proposals but to derive new loss functions from first principles. We also couple our formal findings with empirical results on a hitherto untested class of single model preference loss functions. Our experiments reveal interesting connections between symbolic constraint complexity and the empirical success and training dynamics of the corresponding losses, insights we believe can give useful guidance to AI practitioners working on AI alignment.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
neuro-symbolic modeling logic preference learning RLHF
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6dbba69453bfcf264cc7a056f52cf8856d083f7a.pdf
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Declarative characterizations of direct preference alignment algorithms
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission