Benchmarking LLMs' Judgments with No Gold Standard
Benchmarking;Peer Review;Mutual Information;Data Contamination;Large Language Models
Yes
na
No ethics review needed.
Using LLM to estimate the conditional probility seems problematic:  While P(x,y)=P(x)*P(y|x)=P(y)*P(x|y), P_LLM(x)*P_LLM(x|y) != P_LLM(y)*P_LLM(x|y).  I wonder if this will cause problems in the estimation.
Using mutual information for evaluation subjective generative response is novel as my understanding.

The paper give theoretic guarantees of the proposed method.

Experiments show the method works well on the task of evaluating reviews for academic papers or student proposals.
The paper introduces GEM, a novel metric for evaluating the performance of large language models in generating informative judgments using given references but not necessary gold standards, and presents GRE-bench, a benchmark for assessing the quality of peer reviews generated by LLMs.

GEM is designed to assess the informativeness of judgments in scenarios where gold standard references are unavailable, expanding the scope of benchmarking to subjective tasks like peer review.
Only one type of evaluation tasks has been assessed: paper review evaluation.  It would be more convicing if the paper can conduct experiments of more subjective evaluation tasks, such as summarization, dialog, etc.  

The paper uses Llama 3.1 8B as the evaluation LM.  I would like to see how different LMs effect the performance of the evaluation.

Figure 4 is not mentioned in the paper.
Yes
No ethics review needed.
This is a kind of optional question out of my curiosity. 

It seems that GEM is a relative scoring/evaluation framework. If so, what would be the pros and cons of GEM compared to other relative evaluation frameworks, such as the ELO rating system commonly used in generation task leaderboards, like those in Chatbot arenas?
The paper is clearly written and rigorously formulated. The research topic is very timely and important, given the increasing use of LLMs and the challenges of evaluating their performance in subjective tasks.
This paper presents GEM (Generative Estimator for Mutual Information), a new evaluation metric for assessing the quality of text responses generated by LLMs in tasks that lack a clear gold standard reference.  GEM aims to provide an accurate and manipulation-resistant metric by estimating the mutual information between candidate responses and a set of reference responses, which are not required to be perfect. The main idea is to measure the semantic informativeness of candidate responses by filtering out aspects like language style and syntax, focusing on how much information the candidate's response reveals about the references.
I have some questions and concerns:

1. Impact of Suboptimal Human References: I am concerned about GEM's reliability when human references are of lower quality than LLM outputs. Could the LLM used for estimating probability distributions learn to favor responses that resemble flawed human references, even if they are less informative, potentially leading to inaccurate scores? 

2. Performance stability according to the choice of evaluation-LM and preprocessing LLM: How sensitive is GEM to the choice of LLMs used for evaluation (Llama-3.1) and preprocessing (GPT-4)? Would using different LLMs, especially open-source ones, impact the results? LLMs are released and updated frequently, and I wonder how we can deal with such frequent updates to make the GEM score consistent and reliable over time.
Yes
No ethics review needed.
- Missing citations: 
    - [Finding Blind Spots in Evaluator LLMs with Interpretable Checklists](https://arxiv.org/abs/2406.13439)
    - [Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?](https://aclanthology.org/2024.findings-eacl.71/)
- In Appendix D3, I see that the whole paper is given in text format, but most of the scientific literature also contains images that are quite important and referenced multiple times. What happens in that case? Similarly, how are tables being managed? Also, are these papers converted from PDF to text format, if so, are they being verified for OCR errors? The papers presented are usually 10 pages in ICLR and 8 pages in *CL, so are you sure the paper is not beyond the token limit of the LLM? Lastly, which LLM was used for the prompt in D3?
- For ICLR dataset, it is mentioned that one reviewer is the candidate and the remaining 2 are reference. Aren't only 2 references too less? I feel in this case the sample bias will be quite high. I urge the authors to clear this for me. 
- For the rephrase manipulation, were any sanity checks conducted to make sure that the semantics did not change? Since we are operating at very long-context lengths (document-level), this could be an issue for smaller models like llama-3.1-8b. GPT-4o is quite large and powerful as compared to llama-3-8b and has much better instruction following capabilities, so was this disparity maintained purposefully? If not, I feel llama-3.1-405B or any of the other larger models should be used in place of it which are good at instruction-following.
- For the GRE-Bench, where the LLM is generating reviews, were all 3 human reviews taken as references?
- Line 472 is very redundant and but obvious. Smaller models do not have good instruction following abilities. 
- The word Table has been misspelled as Tabel at a couple of places. Please fix it. 
- In Figure-4 (and Table-6), I am quite dubious about the parameter counts being mentioned there, as most of closed-source models do not reveal anything, especially for GPT models. Also, the color-scheme in Fig-4 is not comfortable to the eye to make clear demarcations across the bars. Please fix that as well. 
- In Prompt D2, the model is provided with one review, which resembles a 1-shot example in a way. In case of generative tasks, there have been cases when the model tries to follow the verbatim or structure of the provided few shots, and the final result lacks creativity. Was this checked in any way? 
- What were the generation parameters in each prompting case?
- How were the outputs checked for hallucinations as the generations are quite long?
- In F1, was the same prompt format as in D2 used while finetuning the model?
- There should be some comparison of llama-3.1-8b-Instruct and its finetuned version (as mentioned in F1) to see the gains due to finetuning. Also, finetuning an already finetuned model is slightly tricky as the distribution shift may cause the model to collapse. The general trend is to use a base model directly for this case.
- The paper is well-written, and the problem statements is well-motivated. The authors provide cogent and coherent arguments throughout the paper. I personally think this is a very valid issue for LLM-based evaluators as not always we have a single human-verified gold-standard reference, but a set of good-enough responses to aggregate and use as a ground-truth. The paper also has a good mathematical rigor to prove its hypothesis. 
- The benchmark created in the dataset is a plus point as it can help future works. 
- I also appreciate the finding that GEM is robust to perturbations as lately there has been a good discussion around LLM having blind spots in evaluation and missing out on subtle disturbances to the input.
The paper mainly discusses benchmarking LLM's judgments in the lack of gold standard reference answer, but with a pool of vanilla reference answers by estimating the mutual information between a candidate and reference. The authors also show that this technique is also robust against perturbations and manipulations. Lastly, they also present a benchmark/test set to evaluate how good LLMs can generate high-quality reviews for academic paper.
- I do not see any blaring errors in the paper, however, I am a suspicious about the experimental setup as such. Bottom line, the problem statement discusses a **very challenging** problem: _LLMs generating "judicious" reviews for very long and complex scientific papers_. I am not sure if most of the existing LLMs are good enough for such a humongous task and if they do provide good results which can be analyses with some guarantee, as the LLMs are required to have a decent knowledge of the domain as well as a good understanding of the hypothesis in the paper, which even humans struggle to understand in some cases. In a way, this feels like just applying an LLM to a task to see how it fares. But the authors to talk about previous works that have discussed using LLMs as reviewers, and I see that most of them have shown negative results in such cases.
- Related to the above point, I have several questions for the authors, which I hope will be answered to give me some more clarity on the nitty-gritty details of the work.
Yes
No ethics review needed.
1. I'm not sure I understand the setup of the first result, am I right in understanding that what you do here is generate k candidates per review, and then get the average GEM score for each candidate by averaging over all other generated candidates when used as references? And then you make the classification by choosing the candidate that resulted in the highest GEM score? How is the classification made? If I'm correct in understanding this, then I don't see how the following sentence from the conclusion applies: "...while our method does not require gold-standard references, it requires reference that includes original human-generated information." Since the way I understand it now, it doesn't really need any human-generated references to generate these scores.

2. Why do you think LMExaminer performs so much better in Sentence Deletion and Deletion & Completion? I feel like that's quite a significant result that hasn't been discussed.

3. "Figure 5 shows that, within each LLM family, such as the Llama-3 or Claude series, models with larger parameter sizes tend to perform better. Moreover, newer versions tend to outperform their predecessors with similar model sizes, as seen in comparisons between Llama-3.1 and Llama-3. These observations further validate the effectiveness of our GEM and GEM-S metrics in accurately evaluating LLMs."
Can GRE-Bench only work with the GEM metrics? Do you think you would observe something similar if you used different metrics?
The paper presents a valuable step towards doing away with the reliance on gold-standard references in evaluation. It's original and creative in the way it makes use of mutual information in the context of generative models.

The method is described rigorously and thoroughly. The flow and structure of the paper makes sense and conveys their ideas well.

The authors also seem to be highly aware and open about the limitations of the method. The paper actively critiques itself and presents answers and discussions on these critiques.
This work is motivated by the importance of finding a benchmarking method that is not reliant on gold-standard references. The proposed method(s) also attempt to circumvent known issues about using generative models as evaluators, where they are often sensitive to degradation strategies and score-increasing manipulations. The method is empirically compared against a wide range of widely known evaluation metrics to showcase its effectiveness.

The authors also introduce a new benchmark for review generation which is built upon their method that inherits all its benefits. The benchmark is also claimed to combat data contamination by being updated with new review data as it comes out.
While the paper is generally highly verbose discussion-wise, I find the experiments and results parts of the paper to be a bit lacking in discussion and analysis. My concerns on this matter are made more specific in the "Questions" part of this review.

Some minor grammar/spelling mistakes: lines 96-97 "..the score can relies solely...", line 455 "Table" is misspelled.
@inproceedings{
anonymous2024benchmarking,
title={Benchmarking {LLM}s' Judgments with No Gold Standard},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=uE84MGbKD7},
note={under review}
}
We introduce the GEM (Generative Estimator for Mutual Information), an evaluation metric for assessing language generation by large language models (LLMs), particularly in generating informative judgments, without the need for a gold standard reference. GEM broadens the scenarios where we can benchmark LLM generation performance-from traditional ones, like machine translation and summarization, where gold standard references are readily available, to subjective tasks without clear gold standards, such as academic peer review.

GEM uses a generative model to estimate mutual information between candidate and reference responses, without requiring the reference to be a gold standard. In experiments on two human-annotated datasets, GEM demonstrates competitive correlations with human scores compared to the state-of-the-art GPT-4o Examiner, and outperforms all other baselines. Additionally, GEM is more robust against strategic manipulation, such as rephrasing or elongation, which can artificially inflate scores under a GPT-4o Examiner. 

We also present GRE-bench (Generating Review Evaluation Benchmark) which evaluates LLMs based on how well they can generate high-quality peer reviews for academic research papers.  Because GRE-bench is based upon GEM, it inherits its robustness properties.  Additionally, GRE-bench circumvents data contamination problems (or data leakage) by using the continuous influx of new open-access research papers and peer reviews each year. We show GRE-bench results of various popular LLMs on their peer review capabilities using the ICLR2023 dataset.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Benchmarking Peer Review Mutual Information Data Contamination Large Language Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ebe1ebd130d2f9e7797cce98d36753e442342e2e.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Benchmarking LLMs' Judgments with No Gold Standard
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission