Direct Judgement Preference Optimization
LLM-as-judge;generative judge;auto-evaluation
Yes
No ethics review needed.
See the Weaknesses.


The authors could enhance the paper by clarifying the relationship between Direct Judgement Preference Optimization and Direct Preference Optimization, highlighting key similarities and differences. Additionally, they could more explicitly state the main takeaways of the paper and explain how their use of three types of data for DPO differs from and improves upon the original DPO approach.

What are the takeaway messages of this paper? It seems like the authors use three types of data for DPO rather one type in the original DPO?
**Diverse Evaluation Tasks**: The paper proposes a comprehensive evaluation framework, including pairwise comparisons, single ratings, and binary classification. This makes the model adaptable to different evaluation needs​.

**Model Performance**: The paper demonstrates that their judge models outperform existing baselines, such as GPT-4o and specialized judge models, across multiple benchmarks, including pairwise comparison, single rating, and classification tasks​.

**Bias Mitigation**: The model robustly counters common biases, such as position and length bias, offering improved generalization across evaluation protocols.
The paper "Direct Judgement Preference Optimization" improves large language models (LLMs) used as generative judges by introducing Direct Preference Optimization (DPO), which enhances evaluation by learning from both positive and negative examples. The authors train models on pairwise comparison, single ratings, and classification tasks using a mix of human and synthetic data, including Chain-of-Thought (CoT) critiques and response deduction. Their models outperform baselines like GPT-4o across 13 benchmarks, mitigate biases, and adapt to various evaluation protocols, offering more robust and flexible performance while also providing valuable feedback for improving other models.
**Content organization **: The organization of this paper could be further improved, for example, structurize some long paragraphs.


**Design philosophy**: The paper could be strengthened by providing a more detailed explanation of the design philosophy, including the rationale for including each component, their individual functions and relationships, and how they relate to the background presented in Section 2. An ablation study or discussion of the impact of removing individual components would also be valuable.
Yes
No ethics review needed.
1. In Line 315, what do you mean by evaluating models trained for single ratings on classification? How do you use a model trained for single ratings to evaluate classification tasks?
2. In Line 314, “All baselines are evaluated only on the tasks they are trained for”.  Can you report which tasks these baselines are trained for? How can you get the results for the tasks that these baselines are not trained for in Table 1 & 2? Do you get the baseline models’ results from their papers or run the baseline models by yourself? Do you test your models on those benchmarks without further fine-tuning your models on the benchmark data at all?
3. In Table 5, what are the 6 types of biases? And How do you measure these biases? I understand the position bias can be measured by the consistency of choosing responses even if the order is swapped. But what about the other biases?
4. Line 464 claims that “we plot the average performance across all three evaluation tasks when removing each training task”, but Fig 4 may miss the SFT+DPO w/o CoT critique.
**Originality v.s. Significance**: This paper proposes to process existing annotated data into the preference data format, and train an LLM as a judge using DPO on the curated data. It’s a straightforward application of DPO to the task of training an LLM for evaluation. The novelty lies in the way of constructing preference data from existing labeled datasets for three evaluation tasks — but see the novelty concerns in the weakness section below. The technical novelty is weak, compared to the significance of releasing a good evaluation model to the community. However, I don’t find the statement of releasing the model checkpoints and the curated data from the paper. It’d be better to clarify this explicitly. 

**Quality**: Extensive experiments and analyses have been conducted on seven pairwise comparison benchmarks, four single rating benchmarks, and two classification benchmarks.
This paper curates the preference data to train an LLM as an evaluation judge using DPO, and demonstrates the effectiveness of the trained models in different sizes (e.g., 8B, 12B, 70B) on three types of evaluation tasks (e.g., pairwise comparison, single rating and classification). The contribution lies in the method of creating preference data for three evaluation tasks using a collection of labeled data from diverse data sources (annotated by humans or other models).
**Novelty of data curation**: As the main contribution of this paper is data curation, it’s better to clarify the difference between this work and existing studies on data curation for LLM evaluation. The key differences are still unclear even after reading the description in Lines 136-150. From Table 1, existing studies have explored the three evaluation tasks (Vu et al., 2024), and have used both SFT and DPO (Want et al, 2024c). This paper seems incremental to combine what has been done in existing studies at a larger scale of training data. 

**Presentation**: It’s hard to interpret the results and the evaluation with other baseline models (see details in the question section below). The tables and figures are far away from their corresponding text descriptions, making it hard to read them at the same time.
Yes
No ethics review needed.
1. Is it possible to quantify hardness of negatives and does that correlate with evaluation accuracy?
2. Choice of 70%:15%:15% for DCoT, DStd and DDed sounds somewhat arbitrary. Can you provide further reasoning behind this?
1. The problem of automated evaluation is important and well studied in recent years. Thus, the work is well motivated.
2. Use of DPO using positive and negative pairs to improve LLM eval is novel. Response deduction is also a nice idea.
3. The framework is comprehensive -- works on 3 eval setttings -- Single Rating, Pairwise Comparison, and Classification.
4. Comprehensive eval on seven pairwise comparison benchmarks, four single rating evaluation benchmarks, and two classification benchmarks. The results are convincingly better compared to our SOTA models and even gpt4o.
5. Bias evaluation results, ablations, flexible prompting strategies, downstream model improvements -- are all nice additions in the paper.
The paper explores enhancing large language models (LLMs) as generative judges for evaluating model responses and generating critiques. It introduces direct preference optimization (DPO) using diverse human and synthetic judgments to improve evaluation across pairwise, single ratings, and binary classification tasks. The trained models, especially the 8B parameter model, outperform strong baselines and specialized judge models, showing robust performance and adaptability across various benchmarks.
1. It will be interesting to see how easy vs hard negatives can help. Given the current way of generating negatives, it is unclear if they were hard enough and if a mix of easier and hard negatives could help improve further.
2. Choice of 70%:15%:15% for DCoT, DStd and DDed sounds somewhat arbitrary. Can you provide further reasoning behind this?
3. Broadly a comparison to evaluation systems with multiple models is not expected as such, but still it will be nice to compare and check how does your method works compared to methods like reconcile.

Typos:
1. citep vs citet should be used properly. E.g. line 41 should have "GPT-4 (OpenAI, 2023)" and not "GPT-4 OpenAI (2023)".
2. Notations M_t vs M_{teacher} or M'_{teacher} do not match between Section 3 and Fig. 2. Same for student model.
Yes
No ethics review needed.
1. This paper, Themis, and CriticGPT heavily rely on human-annotated judgment labels to automatically construct preference datasets. Therefore, the scalability of these methods is limited. 

2.  When performing evaluation in real-world scenarios, requiring the input of evaluation criteria limits the scalability of the method because it may require additional supervision of humans and advanced models.

I hope to receive some detailed discussion and analysis from the authors on these aspects.
* This paper thoroughly analyzes the contribution of three different training examples to enhancing model evaluation and reflection capabilities: CoT, standard judgment, and response deduction. 
* The authors conducted extensive testing on 13 benchmarks covering three types of evaluation protocols: pairwise, pointwise, and classification, with experimental results demonstrating superiority over existing baselines. 
* The proposed method maintains good reliability in terms of bias in LLM-as-a-judge.
The paper explores the use of preference learning algorithms (DPO) to enhance the evaluation/critique capabilities of LLMs. By relying on human-annotated judgment scores to automatically assess the quality of model-generated critiques, they are classified into chosen or rejected categories, facilitating the preference learning. Additionally, the authors designed three training methods: Chain of Thought (CoT), standard judgment, and response deduction to optimize the model, achieving impressive experimental results on 13 benchmarks, covering pairwise, point-wise, and binary classification evaluation protocols.
### 1. Some Prior Works Need to Be Discussed

Using preference learning for optimization sounds impressive, but some prior work have already utilized preference learning to enhance the evaluation/critique capabilities of LLMs, such as CriticGPT  [1] and Themis [2]. 

1. CriticGPT employs a standard RLHF workflow to improve the evaluation/critique capabilities of LLM in the code generation task, relying heavily on cost human annotations. 

2. Similarly, Themis also utilizes DPO to enhance the evaluation capabilities of language models. Themis also automatically construct the preference data by checking if the model's judgment scores are consistent to human-annotated judgment scores. In terms of motivation and implementation details, there is not much difference between this paper and Themis. Of course, the authors introduce three different preference examples to enhance the capabilities of LLMs. 

However, from my perspective, these two works need to be carefully compared and analyzed in this paper. Although the authors can compare the Themis with proposed method (CriticGPT is a closed-sourced model), I do not recommend doing so because I believe there may be numerous variables that could affect the experimental results, such as the size of the dataset and the optimization techniques (Themis re-designs the DPO loss).

Beyond above comments, this paper, Themis, and CriticGPT heavily rely on human-annotated judgment labels to automatically construct preference datasets. Therefore, the scalability of these methods is limited. I hope to receive some detailed discussion and analysis from the authors on this aspect.



### 2. Flexibility of Evaluation Critiera

Similar to models in the Prometheus family, the current model's input includes an additional evaluation protocol except for the user query and the evaluated response. Therefore, during real-world evaluation setting, additional craft of evaluation criteria is required, as demonstrated in lines 328-329. Although Section 5.4 demonstrates that the impact of the crafted evaluation protocol on evaluation is limited, both this paper and Prometheus-like models require the input of an evaluation protocol (no matter the human-annotated or automatically generated by GPT-4). Therefore, when performing evaluation in real-world scenarios, this severely limits the scalability of the method because it may require additional supervision of humans and advanced models (From my perspective).

### 3. Coarse-grained Evaluation Aspects

As demonstrated in Section 4.2 (line 231 tp line 234), the evaluation aspects for building training dataare general quality, like facuality, helpfulness, safety, etc, which is limited and coarse-grained, compared to previous works like Auto-J and Prometheus.
For example, Auto-J introduces 58 tasks and corresponding fine-grained evaluation criteria. Prometheus generate the score-rubrics-like customized evaluation criteria for specific user queries.

### 4. Without the evaluation of  textual critques quality (Chain-of-Thought)
This paper only evaluate the quality of judgment scores, while neglect the evaluation of textual critique quality (the quality of chain-of-thought). Previous works have demonstrated that even if the judgment score is consistent with human-annotated, their chain-of-thought may contains several flaws [3]. This will introduce numerous bias and drawbacks for application of the fine-tuned critique model in some important reseach fields, like the self-improvement of LLMs. 

---


[1] LLM Critics Help Catch LLM Bugs: https://arxiv.org/abs/2407.00215

[2] Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability: https://arxiv.org/abs/2406.18365

[3] The Critique of Critique: https://arxiv.org/abs/2401.04518
Using DPO, we train a family of high-performing generative LLM judge models capable of pairwise, single rating, and classification tasks
@inproceedings{
anonymous2024direct,
title={Direct Judgement Preference Optimization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ToWKyjwDqO},
note={under review}
}
Auto-evaluation is crucial for assessing response quality and offering feedback for model development. Recent studies have explored training large language models (LLMs) as generative judges to both evaluate model responses and generate natural language critiques. However, existing models have been trained almost exclusively with supervised fine-tuning (SFT), often only on a small number of datasets, resulting in poor generalization across different evaluation settings and tasks. In this paper, we investigate how learning from both positive and negative data with direct preference optimization (DPO) enhances the evaluation capabilities of LLM judges across three evaluation tasks: pairwise, single ratings, and binary classification. We achieve this by creating three forms of DPO data from a diverse collection of human and synthetic judgements on contemporary model outputs, with the goal of training our model to generate meaningful critiques, make accurate judgements, and understand what constitutes good and bad responses for a given user input. To demonstrate the effectiveness of our method, we train judge models of three sizes: 8B parameters, 12B, and 70B, and conduct a comprehensive study over 13 benchmarks (7 pairwise, 4 single rating, and 2 classification), measuring agreement with human and GPT-4 annotations. Our models exhibit the best aggregate performance, with even our 8B model outperforming strong baselines like GPT-4o and specialized judge models, such as OffsetBias-8B, Auto-J-13B, Prometheus-2-8x7B, and Skywork-Critic-70B, in pairwise benchmarks. Further analysis shows that our judge model robustly counters biases such as position and length bias, flexibly adapts to practitioner-specified evaluation protocols, and provides helpful language feedback for improving downstream generator models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM-as-judge generative judge auto-evaluation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/25f2e5f46ff400b3d70e7316f781bedc22a29e23.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Direct Judgement Preference Optimization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission