Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions
LLM Evaluation;LLM Agents
Yes
No ethics review needed.
See the weakness section above.
1. The motivation of the paper is good and clear: current LLM evaluation methods face major limitations. Static benchmarks lack flexibility and may suffer from data contamination, human evaluation requires extensive manual effort, and single-judge evaluations risk bias. This highlights the need for an automated, reliable evaluation method that aligns well with human preferences.

2. The proposed method is validated by comprehensive experiments on 15 modern LLMs, achieves a 92.14% correlation with human preferences, and demonstrates extensibility across languages and domains.

3. The paper is well-structured, well-written, and clearly explained; it is a pleasure to read.
This paper proposes a novel framework for automatically evaluating LLMs using LLM-powered agents. The framework consists of three main components: question generation, multi-round peer battles, and committee discussions. The authors validate their approach through extensive experiments with 15 recent LLMs, achieving a 92.14% correlation with human preferences without manual effort. The paper also reveals some of the intriguing LLM behaviors in competitive peer battles.
The application of multi-agent debate in automatic evaluation is not particularly novel, and this paper lacks discussion of a closely related work, ChatEval [1].

There are no formal guarantees on the reliability of the evaluation.

I cannot identify further weaknesses in the paper, but there are several questions I would like the authors to address to enhance the work:

(1) How sensitive are the results to the choice and number of initial committee members and their ranking initialization? Would including low-capability LLMs in the judge committees reduce overall performance?

(2) Can you provide examples of cases where Auto-Arena's evaluation significantly diverged from human preferences? What insights did these cases provide?

(3) How do you ensure that the questions generated by LLMs are sufficiently challenging to evaluate a variety of LLMs as they continue to evolve?

[1] ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Yes
No ethics review needed.
What would happen if you use one weak model or a small model in the framework? how would the results change if that model is in peer-battle and when it is acting as judge and when it is creating questions?
-> The architecture of the new framework is novel

-> Using questioning as part of the framework is something new and would help in coming works, as questioning is an important aspect in terms of reasoning

-> The results are well above the existing baseline results

-> Detailed analysis of contamination results

-> Evaluation is very big problem in the LLM research field and evaluations like this help in making it easier for evaluating LLMs as more 
and more models are released

-> The paper is very well written
The paper proposes a new framework name "Auto-Arena", the core idea is make evaluations dynamic and rely fully on static datasets for evaluating LLMs as new models and datasets gets introduced. The authors make use of different set models in agent framework to make the models act like peers in a debate and then use LLM as jusge to judge the conversation or the debate the previously occurred between the models. The results are good when compared to the existing baselines and detailed analysis is done in terms of contaminations, ablation study etc.
-> The Limitations section is missing, while the work is good there are limitation of this work as well

-> The results would have been more robust if there was human in loop evaluations rather than comparing with static baselines

-> The evaluation for the questions is limited to only 30 questions, there should have more questions taken into consideration for the evaluation of the question as there questions are the most important part for this framework, they lay the foundation. Also more detailed analysis of these questions in the main paper would be better, given the importance of this part.

-> There is need for exploring the models for unseen data and how the models can debate or judge on the topic they have not seen before, you would need all the models to have equal time-stamp cutoffs to consider for peer-review or judge the evaluations
Yes
No ethics review needed.
1. As I mentioned earlier, the paper states that Auto-Arena is cheaper than other methods but still costs more than traditional dataset-based evaluations, estimating around $5 per model evaluation. Could you provide a more detailed breakdown of this cost? Specifically, how much goes to computational resources, model inference, multi-round battles, and committee discussions? In addition to the computational cost, what about the time required for these evaluations?

2. Why did you choose stronger models as judges instead of the initial participants in the debate? Wouldn’t using the original debaters as judges improve the quality of candidate answers?

3. Auto-Arena is presented as a dynamic, automated alternative to static benchmarks, but it still relies heavily on the strength of the foundational model itself. Have you considered incorporating external information sources, similar to how Bing does, to enhance performance?
1. The paper provides solid experimental validation for Auto-Arena, showing it aligns with human preferences at a correlation of 92.14%. In terms of experimental design, the authors included multiple benchmarks and ablation studies, which effectively demonstrate how multi-round battles and committee discussions enhance evaluation accuracy and consistency.

2. In the appendix, the authors mention that their method is cheaper than comparable approaches (though still more expensive than dataset-based evaluation), estimating around $5 to evaluate a single model. However, they don't break down this calculation, which seems like a pretty relevant detail. Could you elaborate on that? I think it would really strengthen the paper to clarify the cost calculation.
Auto-Arena uses LLM-driven agents to handle the entire evaluation process through debates and discussions, cutting down the need for manually labeled data. This framework closely aligns with human preferences and is flexible enough to adapt across different domains and languages, showing robust performance in both English and Chinese evaluations. The process has three main stages:
1. An LLM examiner dynamically creates varied questions to prevent data contamination and simulate real-world use cases.
2. Two LLM candidates debate across multiple rounds on each question, showcasing their abilities in reasoning, interaction, and strategy.
3. A panel of top-performing LLMs jointly assesses the debates to determine the winner, enhancing fairness and reducing biases in the evaluation process.
1. Using LLMs to evaluate other LLMs has always been considered somewhat unreliable, as their effectiveness as judges is closely tied to the foundational model’s performance. In classic tasks where these models perform well as judges, it’s often due to data leakage (i.e., the model has likely seen the answers before). The authors claim that Auto-Arena can be “easily extended to other domains and languages,” but there’s a lack of experimental support for this. Testing on newer tasks and data—ones that no LLM agent has seen—would better demonstrate if it can still perform well in unfamiliar contexts.

2. Although Auto-Arena aligns closely with human preferences, the paper could improve by analyzing instances where LLM judgments differ from human judges, especially in nuanced or high-stakes cases. Examples or further analysis in these areas could highlight where LLM judges may still need refinement, helping the framework align more closely with human evaluation standards.
Yes
No ethics review needed.
See above weaknesses.
- Auto-Arena provides a scalable, automated evaluation of LLMs, reducing reliance on manual efforts.
- Works across different languages and tasks, making it versatile.
- By using the peer-battle mechanism, the framework offers a competitive context that reveals subtle model strengths and weaknesses.
The paper proposes Auto-Arena, an evaluation framework that automates the evaluation of large language models (LLMs) through a multi-stage process involving question generation, peer battles, and committee discussions. This approach mimics human-like evaluation while addressing limitations of static datasets and manual assessments. Auto-Arena achieves high correlation with human preferences, enhancing the reliability of LLM evaluation. It also proves adaptable to different languages and domains, making it suitable for diverse LLMs and scalable leaderboard creation.
- The code provided for review has not been anonymized properly and reveals the author's name: https://anonymous.4open.science/r/Auto-Arena-Code/analysis_scripts/result_analysis.ipynb
- The paper only involves baselines experiments for automated LLM evaluations using the Auto-Arena framework. While the study introduces a modified evaluation method, the results primarily showcase this framework's effectiveness compared to existing evaluation methods, rather than demonstrating significant new findings about model capabilities, which makes the contributions seem incremental. For example, it would be nice to see the individual model performances on each domain according to Auto-Arena, and some analysis on it.
- It seems like Auto-Arena involves a lot more agents interacting (used in both peer debate and committee discussions), making it more resource-intensive than existing evaluation methods. Does the increase in accuracy justify the added computational costs? How does Auto-Arena compare in cost-efficiency to existing evaluation methods? It would be nice to see a more detailed cost breakdown and comparison, including computational resources used, to better assess the cost-efficiency trade-off between Auto-Arena and existing evaluation methods.
A completely automated LLM Evaluation method that engages LLM agents in question generation, peer battles, and committee review.
@inproceedings{
anonymous2024autoarena,
title={Auto-Arena: Automating {LLM} Evaluations with Agent Peer Battles and Committee Discussions},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=pMp5njgeLx},
note={under review}
}
As LLMs continuously evolve, there is an urgent need for a reliable evaluation method that delivers trustworthy results promptly. Currently, static benchmarks suffer from inflexibility and unreliability, leading users to prefer human voting platforms like Chatbot Arena. However, human evaluations require significant manual effort. To address this, we propose the Auto-Arena, an innovative framework that automates the entire evaluation process using LLM-powered agents. Firstly, an LLM examiner generates questions. Then, two LLM candidates engage in a multi-round peer battle based on individual questions, aiming at revealing their true performance differences. Finally, a committee of LLM judges collaboratively discusses and decides the winner, reducing bias and enhancing fairness. During the peer battles, we observe intriguing scenarios where the LLM candidates display competitive behaviors and even learn from the opponents. In our extensive experiments involving 15 recent LLMs, Auto-Arena shows a 92.14% correlation with human preferences, surpassing all previous expert-annotated benchmarks without any manual efforts. As a result, Auto-Arena offers a promising alternative to current human evaluation platforms for evaluating LLMs automatically.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM Evaluation LLM Agents
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/0c2fbea6653d7251038da43692129eb8782476b1.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission