A Statistical Framework for Ranking LLM-based Chatbots
Large Language Models (LLMs);Paired Comparison;Statistical Ranking;Human Preferences;Chatbot Arena;Logistic Regression
Yes
No ethics review needed.
1. Is this method only useful to chatbot arena or it is useful to all Elo ratings?
2. We need another figure to see the ranking difference of original chatbot arena and this methods. Also the figure 1 showing the result of PCA analysis is confusing.
1. The proposed method can account for ties in an axiomatic framework. This improves not only tie prediction but also enhances win-loss inference.
2. The mathematical analysis is comprehensive and thorough.
3. This work has an open source python package which is easy to use.
This paper presents a statistical framework for ranking LLM-based chatbots. It addresses the limitations of existing methods like the Elo rating system by incorporating ties and covariance structures. The proposed apply well-established statistical models to properly account for ties within an axiomatic framework and introducing factor analysis. Experiments on the Chatbot Arena dataset show improved accuracy and insights into chatbot rankings and relationships.
1. For model ranking, it has no "ground truth". Therefore, it is hard to convince others under this method, the ranking is more accurate.
2. The evaluation is highly dependent on the Chatbot Arena dataset which makes this work a slight improvement on chatbot arena. Therefore, the impact of this work is limited.
3. As for chatbot arena, a simple enough ranking rule is more important if the users are common users. This work will make the rule too complicated for them to understand and then reduce the impact of chatbot arena.
Yes
No ethics review needed.
1. In the treatment of pairs and unique pairs, have the authors considered introducing semantic clustering to their model, as opposed to solely using statistical techniques? Additionally, could they clarify the reasoning behind not exploring semantic relationships within pairs?
2. Have the authors considered methods other than PCA to further analyze the correlations between models in the results? The current approach does not sufficiently reveal the underlying correlations between LLMs in an intuitive way.
1. The authors identify symmetry issues in the likelihood function of traditional models, which could lead to instability in parameter estimation. To address this, they propose symmetry constraints that ensure stable parameter estimation, thereby enhancing the model’s optimization performance and interpretability;
2. They effectively address the issue of ties, which is a limitation of the existing Elo system, and make optimizations to handle this;
3. They provide a Python package that allows for the reproduction of the paper's results and supports further research.
The authors propose an improved statistical model for ranking large language models (LLMs) on the Chatbot Arena dataset to enhance the traditional Elo rating system. Paper points out that the Elo system has limitations in handling ties and capturing relationships between models. To address these issues, paper introduces the Rao & Kupper and Davidson models, as well as a novel factor model to better capture the complexity of ties between different models, thereby improving prediction performance. Finally, they provide a Python package, “Leaderbot,” to reproduce the statistical model and support further experiments.
1. The authors use multiple models to rank models, showing that high-ranking models exhibit greater consistency than lower-ranking ones. However, they do not provide further analysis, such as examining the specific characteristics of models that initially show ranking inconsistencies;
2. The authors’ work focuses on optimizing the ranking model but lacks subsequent analysis. Additional insights, such as a more in-depth examination of correlations between LLMs or an analysis of the differences between Leaderbot rankings and Chatbot Arena Elo rankings in relation to model characteristics, would enhance this work.
Yes
no
No ethics review needed.
1. The paper mentions that high-ranking models show greater consistency than lower-ranked ones. Could you expand on this observation? For example, are high-ranking models generally larger or more sophisticated in architecture? And at what ranking position does this shift in consistency typically occur?
2. How is the effectiveness of handling ties quantified across different statistical models? Is there an analysis of which models are more favorable to certain LLMs, and if so, what might explain these differences?

If these questions can be addressed, the paper could potentially be rated one point higher.
1. **Pioneering Approach**: This paper tackles a novel and important problem in LLM evaluation, presenting a unique perspective on using advanced statistical models to refine the ranking process in chatbot comparisons. Its approach to systematically modeling ties and latent structures sets a new precedent for evaluating LLM-based chatbots and offers fresh insights that go beyond traditional ranking methods.
2. **Innovative Use of Thurstonian Representations**: By integrating Thurstonian models and introducing covariance structures, the paper offers a groundbreaking method for capturing relationships among LLMs. This enables a deeper analysis of model consistency and performance that extends beyond simple rankings, marking a notable advancement in chatbot evaluation techniques.
This paper addresses a novel problem in the evaluation of large language model (LLM)-based chatbots by proposing an advanced statistical framework that builds on existing methods used in the Chatbot Arena setting. The framework integrates well-established models, such as Rao-Kupper and Davidson, to account for ties in a rigorous axiomatic framework, and introduces Thurstonian representations to model covariance structures between competitors. These additions allow for more nuanced insights into chatbot rankings and performance consistency. A Python package, leaderbot, is also provided for reproducibility and ease of experimentation.
1. While the paper introduces an alternative framework, it does not clearly discuss why Arena’s Elo-based approach is insufficient beyond the issue of ties. Additional insight into Arena’s limitations in capturing competitive dynamics or certain statistical shortcomings would strengthen the argument for this new model.
2. The authors mention consistency in high-ranking models and variability in lower-ranking models, but do not explore further distinctions within these groups. For example, identifying specific characteristics (e.g., model size) associated with high consistency could provide more actionable insights.
We introduce a rigorous statistical framework for ranking large language models (LLMs) using crowdsourced comparisons, improving accuracy for ties, wins, and losses beyond current methods like Elo.
@inproceedings{
anonymous2024a,
title={A Statistical Framework for Ranking {LLM}-based Chatbots},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=rAoEub6Nw2},
note={under review}
}
Evaluating large language models (LLMs) effectively is essential for advancing their development and ensuring alignment with human preferences. Platforms like Chatbot Arena have made significant strides by gathering millions of votes through crowdsourced pairwise comparisons to rank LLMs, offering valuable data for assessing model performance. However, the statistical methods employed rely on simplistic approaches, such as the Elo rating system, which inadequately handles ties in competitions and overlooks the underlying relationships between competing models. In this paper, we introduce a more rigorous statistical framework that builds upon the data from Chatbot Arena while correcting these methodological shortcomings. We apply well-established statistical models to properly account for ties within an axiomatic framework. Additionally, we introduce a novel factor analysis that captures the complexity of ties across pairs of competitors, significantly improving the overall model performance. These improvements not only enhance the handling of ties but also increase the accuracy of win and loss predictions compared to previous methods. Additionally, we incorporate Thurstonian representations to model covariance structures between competitors, allowing for deeper insights beyond rankings. We also address previously unrecognized symmetries in the likelihood function that can hinder optimization and propose constraints to ensure stable parameter estimation. Finally, we provide a Python package, leaderbot, to facilitate reproducibility. Our experiments demonstrate significant improvements in accuracy for both ties and win-loss outcomes, offering a robust alternative to existing methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models (LLMs) Paired Comparison Statistical Ranking Human Preferences Chatbot Arena Logistic Regression
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/050db151c0803e943ffbb3e92f4badb075f19116.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
A Statistical Framework for Ranking LLM-based Chatbots
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission