Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge
Automated evaluation;LLM as a judge;correlation measures
Yes
No ethics review needed.
How come the metric does not need a single value to approximate human labels but relies on a single "human and machine labels are not treated interchangeably, as the items in a given bin are selected by the human median or majority value"? This seems contradictory to me.
1) The paper is well structured and presented and is very clear and easy to follow and read.  
2) The paper clearly shows the issue related to relying on a high correlation between human and machine-generated outputs, giving cases where the uncertainty in human annotations is high; these correlations seem to be high, but that could also be the case even when the labeling is random. 
3) The study proposed a metric, namely binned JSD, to account for variations and uncertainty in human judgment.
The paper discusses the current landscape of using LLMs as judges for various tasks and presents compelling arguments for why existing correlation metrics might not take into account variations and uncertainty in human judgment.
1) It is not clear to me how this new metric handles the issues raised with traditional metrics. Could the author clarify and show cases of how JSD improves the analysis of LLMs as judges when compared to human judgment? It seems like a promising direction, but I am not convinced due to the limited number of datasets and support of the authors' claim. 

2) The type of human annotation is limited. Although the paper presented very well the type of collected human annotations, I limited these in their evaluations. Given the space and the breed of the paper, I suggest adding a few more datasets (Please check paper [1] for some guidance on what dataset to choose.)

3) The study considered four datasets with variant tasks and annotations that have sufficient human annotators. I think the number of annotations was well considered, but the number of the dataset could be improved, and metal analysis could have been better presented instead of large tables per dataset. 


[1] Bavaresco, Anna, et al. "Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks." arXiv preprint arXiv:2406.18403 (2024).
Yes
No ethics review needed.
Could the author provide some detail in the selection of different thresholds across the tables? It would be useful to clarify whether these thresholds are dataset-specific or if there's a general principle behind the thresholds selection.

In the paper, different metrics  are used and the human and machine labels are compared with average, with median or with majority labels. Are all these comparisons needed? Do they capture multiple aspects of the outputs?

Is the binned JSD the best metric for the proposed experiments? Is it possible to calculate this metric, or its adaptation,  for all the experiments proposed in the paper?

The bins are used to mimic human perception. Beyond the aggregation of perception, can they capture variation in human perception?
The topic is new, and since human-labelled data are difficult to retrieve, an evaluation of machine-labelled data is very interesting.

The tests have been done on different LLMs 

Multiple tests have been performed.
The paper describes the analysis of different measures in  evaluating  LLM responses.
A measure, specifically a binned Jensen- Shannon divergence is proposed.

This measure for ordinal perception data is justified by the author since the evaluation does not need a single gold label and the human and the machine are not interchangeable. This last condition breaks a necessary condition for the Krippendorff-alpha coefficient.

A distinction is done among Nominal values, ordinal values and continuous values. The Questions are not equally proposed for the three types of data, raising some difficulty in reading the paper.

RQ1:How does uncertainty in human labels impact correlation metrics when
we measure the efficacy of automatic evaluation methods? (Sec. 3.1)

The authors state that the uncertainty in human labels is high the human-machine majority labels are similar. The meaning appears to be that if there is no concordance among labeller, in this case the LLM judge is ok. It is the LLM-judge just adding noise to the labelling process?


RQ2: How can we measure human-to-machine(HM) agreement that accounts for human uncertainty as a result of variation in human perception?
Human to machine agreement is a measure of uncertainty in human perception. For this question, the comparison with different agreement percentage is tested. For this task is proposed the binned Jensed inequality.


RQ3: How can we visualize the underlying data to draw meaningful insights when we compare the results from automatic and human evaluations?

The authors compare ordinal and perceptual based ratings between human and machines
The work's presentation is not clear. The research questions help to  interpret the experiments but not all the results are clear.

Some terms, like H^W R^W (^ indicates apex), are defined in the caption of caption 1. Probably they should be defined in the text and used in the table.

It is unclear how the partitions are decided. In the experiments:in table 1 the thresholds are  0, 0.8, 1; in table 2 the threhsolds are 0.6 and 1; in table 3 are 0.6, 0.8, 1.0. It is not clear if the thresholds are experiment dependent or there is a rationale behind the threshold selection.

The random classifier test, reported in table 1 should be better described. The table reports unique =2 or unique =3 with a percentage. Unique term is not present in the description and should be explained for the clear presentation of the experiment. If the MNLI and SNLi
dataset have only a limited set of labels it should be specified in the description.

In Table 2 the terms H^mu M^mu are not specified. 


In Figure 3 are shown the human perception vs the machine labels binned by human median rating. The JS value is reported.
The highest values of JS are for \bar{H}=1 and  \bar{H}=5. Looking at the histograms, the histograms of human and machine with \bar{H}=3 (and  in some measure \bar{H}=4) are very similar, but the JS values are sensibly lower.
The authors state that humans tend to be more certain when they assign extreme rating and the machine rarely provide extreme values. 
Could  the explanation of the experiment take into account also this aspect? If there is a different interpretation of this discrepancy ( similar histograms but lower JS value) it would be useful to make this point more evident.
Yes
No ethics review needed.
1. Could you provide more specific implementation guidance or pseudocode for this metric, perhaps in an appendix? This would help ensure reproducibility and clarity for those looking to apply it.
2. The paper attributes label uncertainty to genuine perceptual differences, but could the authors discuss other potential sources, such as cultural or contextual biases among annotators? How might such biases affect the evaluation results, and could additional stratification methods help account for them? This would make the framework more applicable across diverse datasets and ensure its robustness in various contexts.
This paper makes a valuable contribution by tackling the often-overlooked role of human uncertainty in evaluating generative models. It’s original in its approach, introducing the binned Jensen-Shannon Divergence metric to better capture the nuances of human perception and using tools like perception charts to bring new depth to evaluations. The quality of the work shows through in its thorough methodology, with experiments across multiple datasets that lend strong support to the findings. The paper is also clear, with a well-structured flow and visuals that help explain complex ideas. Most importantly, the paper has real significance: its framework could reshape how we evaluate generative models, especially in areas where human judgment isn’t always straightforward. Finally, the paper’s significance lies in its potential to reshape evaluation practices for generative models, especially in applications where human judgment is inherently subjective, such as content generation, recommendation systems, and interactive AI. By emphasizing the role of human uncertainty and offering practical tools to account for it, this work highlights a crucial aspect often ignored in model evaluation. This framework could lead to more accurate and context-sensitive evaluations, particularly for models that interact with or respond to human preferences.
This paper offers a well-supported framework that deepens our understanding of human-machine evaluations, bridging the gap between traditional metrics and the complexities of human perception. Its contributions could have a lasting impact, inspiring future research and improving evaluation standards across the field of generative modeling.
This paper explores how current methods for evaluating generative models often fall short by relying too heavily on correlation metrics like Krippendorff’s α and Randolph’s κ. These metrics, while common, can mask important nuances in human judgment, especially in cases where human responses vary widely. The authors show that when there’s a high degree of variation in human evaluations, machine judgments might seem to align well, but as human consensus strengthens, this alignment breaks down, revealing gaps in machine understanding. To address these issues, the paper proposes a more robust evaluation framework that includes stratifying results by the level of human agreement and introducing a new metric, the binned Jensen-Shannon Divergence, to better capture perception-based evaluations. Additionally, the authors suggest using visual tools like perception charts to more clearly illustrate where machine judgments align or diverge from human benchmarks. By combining multiple metrics and visualization methods, this approach aims to provide a more accurate and comprehensive understanding of automated evaluations, especially in areas where human judgments are inherently uncertain.
There are some potential for improvement in paper such as expanding on the technical implementation of the binned Jensen-Shannon Divergence metric would make it more accessible to practitioners, potentially by providing step-by-step instructions or pseudocode. Additionally, testing the framework on a broader range of generative models beyond text (such as image or audio) would demonstrate its versatility. The perception charts are helpful, but they primarily show aggregate trends, which may obscure individual item-level discrepancies; adding item-level visualizations or error bars could improve clarity. To connect more concretely with real-world applications, the paper could benefit from case studies or examples where the framework enhances specific generative tasks, such as in recommender systems. Moreover, a side-by-side comparison with existing metrics like Krippendorff’s α would better illustrate the added value of the proposed metric. Including confidence intervals or statistical significance testing could also add rigor to the findings. Finally, considering potential biases in human label uncertainty, such as cultural or contextual differences among annotators, would make the framework more robust across diverse datasets. Together, these enhancements would increase the framework’s clarity, practical utility, and adoption potential.
Yes
No ethics review needed.
- How do you decide what's "high" or "low" uncertainty in your stratification, and did you try other thresholds?
- Why did you choose Jensen-Shannon Divergence for human perception, and can you show this is better than existing metrics?
- How did you adapt Krippendorff’s α and similar metrics to account for systematic machine errors, not just random ones?
- Why use previous prompts without optimizing them for each model, wouldn't  this affect fairness in comparisons?
- Can you explain how you created the synthetic data for high uncertainty and whether it really reflects real-world data?
- Could you provide concrete examples where machine evaluations diverge significantly from human judgments?
- How would these findings help practitioners improve real-world model evaluations?
Originality: This paper takes an original approach by addressing human uncertainty in generative model evaluation. Introducing stratified results by human variability and the new JSb metric for perception-based tasks adds fresh methods to handle subjectivity in evaluations. The perception charts also offer an innovative way to visualize nuanced performance differences.

Quality: The work is methodologically sound, with comprehensive experiments across datasets like SummEval and MNLI. The use of both real and synthetic data strengthens the empirical basis, showcasing the impact of human judgment noise on evaluation reliability.

Clarity: The paper is well-organized, clearly defining key concepts such as HH vs. HM correlation. Explanations of JSb and perception charts are straightforward, helping readers understand the new evaluation tools effectively.

Significance: This work fills an important gap by addressing subjective variability in human evaluations. Its proposed methods (if widely adopted) could lead to more accurate and relevant model evaluations, especially in perception-driven tasks across AI.
This paper examines how human uncertainty affects evaluating generative models, noting that standard metrics like Krippendorff’s α may misrepresent machine accuracy when human judgments vary. The authors propose three main contributions: stratifying evaluation results by human uncertainty levels, introducing binned Jensen-Shannon Divergence (JSb) to better measure alignment with human perception, and creating perception charts to visualize evaluation performance more effectively. These tools aim to provide a clearer, more accurate picture of machine evaluation performance amidst human variability.
- The choice to stratify results based on “high” and “low” human uncertainty needs clearer justification. A discussion or empirical test on how these thresholds were set would make the stratification process more robust and reproducible.

- The introduction of Jensen-Shannon Divergence as a measure of perception-based tasks is promising, but its effectiveness is not thoroughly proven. Including a comparison with other potential metrics, such as Earth Mover’s Distance or Wasserstein Distance, would better validate the claim that JSb captures human perception more accurately.

-While reusing prior prompts is convenient, this may introduce biases or inconsistencies across models. Optimizing prompts specifically for each model would yield more accurate comparisons, especially given the importance of prompt sensitivity in LLM performance. Adding prompt-tuning experiments for each model could further solidify the findings.

-The synthetic data used to simulate high uncertainty scenarios lacks details on its generation process. More transparency on how closely this data reflects real-world scenarios, including its validation process, would help verify the relevance of the findings. 

- The shifts in ∆ (difference between HH and HM correlations) across different uncertainty levels are intriguing but underexplored. An in-depth analysis of these shifts—perhaps with concrete examples of where machines diverge from human judgment—would help understand the implications of these results more clearly. Visual examples showing alignment and divergence between human and machine judgments could greatly enhance interpretability.

- The experiments rely primarily on four datasets, which, while varied, still represent a narrow slice of possible applications.
Impact of noise when measuring effectiveness of automated evaluation
@inproceedings{
anonymous2024beyond,
title={Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and {LLM}-as-a-judge},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=E8gYIrbP00},
note={under review}
}
The effectiveness of automatic evaluation of generative models is typically measured by comparing it to human evaluation using correlation metrics. 
However, metrics like Krippendorff's $\alpha$ and Randolph's $\kappa$, originally designed to measure the reliability of human labeling, make  assumptions about human behavior and the labeling process. 
In this paper, we show how *relying on a single aggregate correlation score* can obscure fundamental differences between human behavior and automatic evaluation methods, including LLM-as-a-Judge. 
Specifically, we demonstrate that when the proportion of samples with variation or uncertainty in human labels (gathered during human evaluation) is relatively high, machine labels (generated by automatic evaluation methods) may superficially appear to have similar or better correlation with the human majority label compared to human-to-human (HH) correlation. 
This can create the misleading impression that automatic evaluation is accurate enough to approximate the human majority label. 
However, as the proportion of samples with consistent human labels increases, the correlation between machine labels and human majority labels declines, falling below HH correlation. 
Based on these findings, we first propose stratifying results by human label uncertainty to provide a more robust analysis of automatic evaluation performance. Second, recognizing that uncertainty and variation are inherent in perception-based human evaluations, such as those involving attitudes or preferences, we introduce a new metric - *binned Jensen-Shannon Divergence for perception* for such scenarios to better measure the effectiveness of  automatic evaluations. Third, we present visualization techniques -- *perception charts*, to compare the strengths and limitations of automatic evaluation and to contextualize correlation measures appropriately
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Automated evaluation LLM as a judge correlation measures
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/30a3854eb2cf360738668e252c60fad9ab2a5b40.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission