Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
LLM;large language model;alignment;post-training;benchmarking;evaluation
Yes
No ethics review needed.
Figure 2 claims that “in the SFT stage of post-training, the size of the dataset, rather than the method used to curate the data, is the strongest predictor of alignment.” Does this assume that all the data is deduplicated? Also, prior works seem to indicate that higher quality data (and data from different domains) can be more important than size. Can the authors comment on this?
1. The paper is well motivated and definitely investigates an important question. It is valuable to know the failure modes of LLM-as-judges.
2. Paper is well written and logically organized.
This work investigates when LLM-judge preferences translate to progress on other metrics for alignment, and in cases of failure, why not. In particular, this work finds that LLM-judges have implicit biases, and prioritize stylistic preferences over eg factuality and safety. This work also introduces SOS-bench, a new alignment benchmark with ground truth and with the goal of gauging progress on helpful, honest, and harmless principles. Experiments show that data scaling in SFT and improving prompt diversity are important predictors of improved alignment.
1. Since SOS-bench is a collection of existing benchmarks, I’m not sure whether this is a novel contribution (but rather a thoughtful choice for what benchmarks to use).
2. In table 2: Why not advocate for using correctness for the score (instead of using the raw score)?
Yes
No ethics review needed.
Please respond to the concerns in the "Weaknesses" part.

Q1. What specific role do you envision for static benchmarks with ground truth answers versus LLM-judge benchmarks in evaluating alignment? Are you suggesting completely replacing LLM-judge benchmarks or proposing a hybrid approach? How do you see these different evaluation methods complementing each other?

Q2. Could you provide more details on the process of converting LLM-judge preference results to the absolute percentage scores shown in Table 2? Specifically:

- 2.1 What exact steps were used to convert preference judgments to percentage scores?
- 2.2 Were any normalization or scaling steps applied?
- 2.3 How did you account for or mitigate potential biases in this conversion process?
- 2.4 Did you perform any sensitivity analysis on different scoring methods?"
1. This work tested the bias of LLM-judge between and within criteria and provided convincing results.
2. This work analyzed the performance of the latest LLMs on concrete evaluation metrics and summarized the relationship between performance and post-training settings.
This work exposes the issue of significant implicit bias in benchmarks that employ LLM judges to evaluate the alignment with human preferences. They evaluate LLMs with concrete metrics and suggest that the post-training and evaluation for the alignment of LLMs should be further designed to focus on more specific desirable factors.
1. The SOS bench proposed in this paper is basically a combination of existing static evaluation datasets regarding instruction following, world knowledge, safety, etc. What specific limitations of existing static datasets does SOS-Bench address? Are there any novel data processing, aggregation, or evaluation techniques used in creating SOS-Bench?
Yes
No ethics review needed.
NA
The paper proposes studying the drawbacks of the LLM-as-a-judge framework, a critical topic.
- The most interesting findings, in my view, are in Table 3;
- The approach to studying biases reported in Table 3 seems sound.
This paper studies the drawbacks of the LLM-as-a-judge framework and proposes SOS-Bench, a ground-truth meta-benchmark, to circumvent those drawbacks.
The paper's drawbacks are greater than its contributions. In the following, I list what I think the be the biggest weaknesses of the paper and offer some suggestions on some of these points:

1) The paper does not provide convincing evidence of biases of the LLM-as-a-judge framework except from what is presented in Table 3. I believe the results in Table 2 are misinterpreted as they do not provide enough evidence of such biases (e.g., style over content preferences). I have two main reasons to support my claim:
     (i) The rank correlation of content and overall score in Table 2 seems to be very high. The fact that the correctness scores are "compressed" does not mean anything by itself. For example, that fact could happen if the benchmark has a good number of very easy or hard questions;
     (ii) "Completeness" seems to be more related to content/correctness than to style.

     Suggestion 1: for Table 2, it would be informative to present the rank (and Pearson) correlations between overall scores and specific scores.

     Suggestion 2: you could try regressing the overall score against the fine-grained scores and statistically test for the coefficients of this regression model. One limitation is that your covariates can be highly correlated to each other, inflating the estimate variances; you might need to collect more data. A second idea is running factor analysis in the full set of scores and then analyzing the low-dimensional factors.

     Suggestion 3: it would be more interesting to focus more on controlled experiments as those presented in Table 3.

2) I do not see a big value in introducing another meta-benchmark like SOS-Bench. We already have HELM and Open LLM Leaderboard, which are already very comprehensive. Moreover, size should not matter that much in the quality of benchmarks as emphasized by the authors; the accuracy of scores does not change much after a certain sample size (think about concentration inequalities, the law of large numbers, etc). Bigger sizes only make evaluation harder.

     Suggestion: instead of just combining many benchmarks by running LLMs in all of them, the authors could try to propose a method to derive a small and comprehensive "summary" (meta) benchmark. This summary benchmark could be much smaller (perhaps even ~100x smaller) than the current version of SOS-Bench and contain key/relevant examples/questions from each one of the three criteria in HHH. That would make the benchmark more usable and the contribution would be more expressive.

Minor points:
(i) Table 1 could be a correlation matrix.
Yes
No ethics review needed.
I also have some minor clarifying questions and suggestions:

1. The paper is missing many in-text references (e.g. for the datasets in Tables 4 and 5), making it slightly challenging for the reader to form context when analyzing the presented results.
2. How is `Loss` defined in Table 3?
3. In Table 5, is Arena score based on an 'overall judgement'. If so, it would be interesting to also compare against fine-grained evaluations of [completeness, conciseness, style, safety, correctness] to see if there are any correlations there with WK, IF, S.
4. The authors should reference pluralistic alignment (which concerns a similar effect as those analyzed) in S4.1 and Table 2.
5. The authors mention the Bradley-Terry model somewhat ad hoc. Did the authors intend to imply anything about the BT model through the presented results/analyses?
The research question the authors tackle is timely and relevant, especially given the increase in recent works that evaluate alignment largely using LLM-as-a-judge. 

The work investigates questions that are in the community's consciousness and provides experimental evidence and analyses.

The paper reads well and is interesting. The presentation interleaves empirical analysis with intuition and justifications.
This paper investigates the reliability of benchmarks that employ LLM-as-a-judge evaluation of model alignment. Specifically, it aims to investigate:
1. Correlation of LLM-judge preferences with more traditional (input, output) benchmarks, where the ground truth is known
2. Implicit biases of LLM-as-a-judge preferences when evaluating pluralistic objectives, that the authors explicitly disentangle into [completeness, correctness, safety, conciseness, style]

The paper also presents SOS-Bench, an amalgamation of existing traditional benchmarks, which is partitioned in a way to evaluate the HHH principles.
One of the main focus of this work is empirical analysis of the reliability of LLM-as-a-judge benchmarks. As such, the main questions on my mind were whether the empirical findings were **robust** and **systematic**, and adequately controlled for potential confounding factors.

Below are my **major concerns**:
1. **LLM judges [S4]**: The authors evaluate results on one judge model (either gpt4 or gpt4o-mini) and one judge benchmark (Arena-Hard-Auto). I understand the effort and costs involved in evaluating multiple judges and benchmarks, but some consistency would strengthen the findings here. Nonetheless, I found this to be the most convincing set of results in the paper.
2. **Novelty of SOSBench [S5]**: Based on my understanding, SOS-Bench is an amalgamation of 19 existing benchmarks, binned together to evaluate three aspects of world knowledge, instruction following, and safety. I am not confident that this can be claimed as a novel contribution, especially since reporting evaluations on multiple benchmarks is common practice in alignment papers/ML research in general. If the author's contributions here are in collating and integrating them, I would consider it more as valuable engineering work rather than a research contribution.
3. **Implicit bias from LLM vs implicit bias from benchmark selection [S5]**: The authors could consider being more nuanced and mindful when discussing different approaches to evaluating alignment. Specifically, the principles of alignment are often pluralistic and ill-defined, e.g. 'safety' or 'helpfulness'. Distilling such principles into an explicitly specified metric or evaluating them indirectly (through traditional benchmarks) is also susceptible to its pathologies. For example, the authors mentioned `BLEU`, which has several well-documented biases. Additionally, the authors have introduced their own biases when using the performance of *world knowledge* as a surrogate for honesty, *instruction following* for helpfulness, and *safety* for harmlessness. While these might be reasonable surrogates, the point is, that the authors have injected their own implicit/explicit biases into the selection of benchmarks and thus evaluation metrics.
4. **Robustness of findings [S6]**: I have concerns about the robustness of the findings and the systematicity of experimental design. I will list my questions in detail below:
* `Data scaling improves alignment in the SFT stage of post-training.` The authors have controlled for differences in pretraining by starting from the same seed model. Is this paragraph (and Fig 2) only comparing models after SFT or after SFT + preference alignment? Analyzed by itself, the results appear to indicate that models are better aligned when SFT is performed with more data. This alone is not surprising. The authors also mention that SFT scale is more important than **curation** and **other important factors**. This relative impacts (of these other important factors) are not analyzed/reported, so it is unclear whether scale or a latent confounder has a higher impact on alignment post SFT.
* `Specialist post-training methods underperform generalist methods.` The authors mention 'diversity of prompts'---this would benefit from an explicit definition. Is it the narrow/broadness of the SFT dataset? Here, the comparison does not control for the size of the dataset, so it is not clear whether the observed differences in performance on specialist domains is due to 'scale' or 'prompt diversity'. Additionally, the authors have not investigated the quality of the datasets compared. I tried to have a look and couldn't find some of these datasets (e.g. Code LLAMA 3) online (the implication here is that they might be lower quality). In short, the results are not strongly convincing that 'diversity of prompts' is the primary factor that could explain the difference in downstream performance.
* `Preference optimization trades world knowledge for improved safety and instruction following.` This is a well-documented phenomenon colloquially referred to as the 'alignment tax'. Additionally, while the authors acknowledge that this result could be confounded by the choice of preference dataset, I feel that this as a crucial factor that must be controlled for any empirical insights to be made.
5. **SFT vs PO**: In the abstract, the authors claim `supervised finetuning stage, and not the PO stage, has the greatest impact on alignment'. Perhaps I missed it, but where this substantiated or investigated?

IMO, this work is ambitious (which I applaud), attempting to tackle many questions but unfortunatley compromising on thoroughness and depth. Specifically, the authors try to approach two tasks in one paper: (1) investigating pathologies of LLM-as-a-judge evaluations, and (2) proposing their own benchmark. I can see that the two are related. But my feeling is the impact would be significantly improved if the paper focused more in-depth on analyzing either of the two.
We find that LLM judgments do not correlate with concrete measures of alignment, and that LLM judges have powerful implicit biases, prioritizing style over factuality and safety.
@inproceedings{
anonymous2024style,
title={Style Outweighs Substance: Failure Modes of {LLM} Judges in Alignment Benchmarking},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=MzHNftnAM1},
note={under review}
}
The release of ChatGPT in November 2022 sparked an explosion of interest in post-training and an avalanche of new preference optimization (PO) methods. These methods claim superior alignment by virtue of better correspondence with human pairwise preferences, often measured by LLM-judges. In this work, we attempt to answer the following question -- do LLM-judge preferences translate to progress on other, more concrete metrics for alignment, and if not, why not? We define a concrete metric for alignment, and introduce SOS-Bench (Substance Outweighs Style Benchmark), the largest standardized, reproducible LLM meta-benchmark to date. We find that (1) LLM-judge preferences do not correlate with concrete measures of safety, world knowledge, and instruction following; (2) LLM-judges have powerful implicit biases, prioritizing style over factuality and safety; and (3) the supervised fine-tuning (SFT) stage of post-training, and not the PO stage, has the greatest impact on alignment, with data scaling and prompt diversity as the driving factors.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM large language model alignment post-training benchmarking evaluation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f2c0716659d2716fc753c1818cf8245868a07d7b.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission