<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2406.11939] From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline</title><meta property="og:description" content="The rapid evolution of language models has necessitated the development of more challenging benchmarks. Current static benchmarks often struggle to consistently distinguish between the capabilities of different models …">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2406.11939">

<!--Generated on Sat Jul  6 00:31:56 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<div id="p1" class="ltx_para">
<span id="p1.1" class="ltx_ERROR undefined">\pdfcolInitStack</span>
<p id="p1.2" class="ltx_p">tcb@breakable</p>
</div>
<h1 class="ltx_title ltx_title_document">From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Tianle Li<sup id="id3.3.id1" class="ltx_sup">∗</sup>  Wei-Lin Chiang<sup id="id4.4.id2" class="ltx_sup">∗</sup>  Evan Frick  Lisa Dunlap  Tianhao Wu 
<br class="ltx_break"><span id="id5.5.id3" class="ltx_text ltx_font_bold">Banghua Zhu</span>  <span id="id6.6.id4" class="ltx_text ltx_font_bold">Joseph E. González</span>  <span id="id7.7.id5" class="ltx_text ltx_font_bold">Ion Stoica</span> 
<br class="ltx_break">UC Berkeley 
<br class="ltx_break">
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id8.id1" class="ltx_p">The rapid evolution of language models has necessitated the development of more challenging benchmarks. Current static benchmarks often struggle to consistently distinguish between the capabilities of different models and fail to align with real-world user preferences. On the other hand, live crowd-sourced platforms like the Chatbot Arena collect a wide range of natural prompts and user feedback. However, these prompts vary in sophistication and the feedback cannot be applied offline to new models. In order to ensure that benchmarks keep up with the pace of LLM development, we address how one can evaluate benchmarks on their ability to confidently separate models and their alignment with human preference. Under these principles, we developed BenchBuilder, a living benchmark that filters high-quality prompts from live data sources to enable offline evaluation on fresh, challenging prompts. BenchBuilder identifies seven indicators of a high-quality prompt, such as the requirement for domain knowledge, and utilizes an LLM annotator to select a high-quality subset of prompts from various topic clusters. The LLM evaluation process employs an LLM judge to ensure a fully automated, high-quality, and constantly updating benchmark. We apply BenchBuilder on prompts from the Chatbot Arena to create Arena-Hard-Auto v0.1: 500 challenging user prompts from a wide range of tasks. Arena-Hard-Auto v0.1 offers 3x tighter confidence intervals than MT-Bench and achieves a state-of-the-art 89.1% agreement with human preference rankings, all at a cost of only $25 and without human labelers. The BenchBuilder pipeline enhances evaluation benchmarks and provides a valuable tool for developers, enabling them to extract high-quality benchmarks from extensive data with minimal effort.</p>
<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><sup id="footnote1.1" class="ltx_sup">∗</sup>Equal contribution. Correspondence to: Tianle Li (tianleli@berkeley.edu), Wei-Lin Chiang (weichiang@berkeley.edu)</span></span></span>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">The proliferation of Large Language Models (LLMs) has spurred advancements, leading to models trained on vast datasets with significantly enhanced capabilities.
However, this innovation also presents challenges in designing effective evaluation benchmarks.
Many existing benchmarks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib45" title="" class="ltx_ref">45</a>, <a href="#bib.bib34" title="" class="ltx_ref">34</a>, <a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib11" title="" class="ltx_ref">11</a>, <a href="#bib.bib3" title="" class="ltx_ref">3</a>, <a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite> are unable to keep up with the rapidly advancing capabilities of these models, becoming saturated and unable to differentiate between state-of-the-art models.
Additionally, many benchmarks are designed for automatic evaluation with restrictive prompts, such as close-ended or multiple-choice questions, which do not accurately reflect a model’s performance in real-world interactions.
Recent developments in benchmark evaluation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib50" title="" class="ltx_ref">50</a>, <a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite> have made it feasible to evaluate more open-ended prompts, yet generating high-quality prompts at scale remains challenging. Furthermore, static benchmarks are prone to over-fitting and potential test leakage over time, reducing their effectiveness in ranking models.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">To address these issues, an effective benchmark must confidently differentiate model capabilities, reflect real-world human interactions, and update frequently to stay relevant and avoid overfitting.
Live crowdsourced evaluation platforms, like the Chatbot Arena <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite>, address some of these concerns by collecting real-world user prompts with preference labels. However, without quality control, these platforms can include low-quality prompts that do not adequately differentiate top models.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">In this work, we introduce BenchBuilder, a data pipeline designed to automatically build high-quality benchmarks from live crowdsourced data source such as the Chatbot Arena.
BenchBuilder identifies diverse benchmark prompts that can confidently separate model performance. It uses seven key indicators of high-quality prompts, such as specificity and domain knowledge, and employs an LLM to annotate prompts for these indicators.
The resulting benchmark, Arena-Hard-Auto v0.1, uses an LLM judge <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite> to estimate human preference against a baseline model, making BenchBuilder and Arena-Hard-Auto v0.1 a fully automated system.
We further explore how to measure the quality of benchmarks by introducing metrics to assess a benchmark’s ability to confidently separate models and its agreement with human preferences.
Comparing Arena-Hard-Auto v0.1 to current leading benchmarks such as AlpacaEval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite> and MT-Bench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite>, we demonstrate that Arena-Hard-Auto v0.1 offers significantly stronger separability with tighter confidence intervals. Additionally, Arena-Hard-Auto v0.1 achieves 89.1% agreement with Chatbot Arena rankings, making it an excellent predictor of downstream performance.
Arena-Hard-Auto v0.1 provides a cost-effective and informative evaluation benchmark, and the BenchBuilder pipeline serves as a valuable tool for developers seeking to automatically generate high-quality benchmarks from vast data sources.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">To summarize, our work makes the following contributions:</p>
<ol id="S1.I1" class="ltx_enumerate">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We propose a novel data curation pipeline, BenchBuilder, to automatically construct high-quality benchmarks from crowdsourced data.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">We propose metrics to capture desired properties in a benchmark, and validate Arena-Hard-Auto v0.1 achieves significantly better separability and agreement than existing benchmarks.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S1.I1.i3.p1" class="ltx_para ltx_noindent">
<p id="S1.I1.i3.p1.1" class="ltx_p">We open-source both the Arena-Hard-Auto v0.1 benchmark and BenchBuilder pipeline<span id="footnote1a" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Our code is available at: <a target="_blank" href="https://github.com/lm-sys/arena-hard-auto" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/lm-sys/arena-hard-auto</a></span></span></span>.</p>
</div>
</li>
</ol>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related works</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p"><span id="S2.p1.1.1" class="ltx_text ltx_font_bold">LLM benchmarks.</span> We briefly review widely used LLM benchmarks. Most existing benchmarks are static and ground-truth-based (e.g., multi-choice question answering). They cover a wide range of domains, including math, science, coding, and reasoning. Common ones include MMLU <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>, MATH <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite>, GSM-8K <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>,
HumanEval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite>,
DROP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite>,
BigBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>, HellaSwag <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite>, AGIEval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite>, GPQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite>, as well as comprehensive collection like HELM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite>.
Many have considered task-based evaluation such as IFEval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite>, SWE-Bench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>, or AgentBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref">28</a>]</cite>. As LLMs become widely adopted in open-ended scenarios involving interaction with humans (e.g., chatbot), many have considered human evaluation using domain experts or crowd raters such as Amazon Mechanical Turk <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>, <a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite> to examine models’ response quality.
As an alternative to human labeling, previous work has shown that LLM-as-a-judge can be effective human preference proxies (e.g., AlpacaFarm <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>, MT-bench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite>, AlpacaEval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite>).</p>
</div>
<div id="S2.p2" class="ltx_para ltx_noindent">
<p id="S2.p2.1" class="ltx_p"><span id="S2.p2.1.1" class="ltx_text ltx_font_bold">Benchmark leakage.</span> A fundamental limitation of static benchmarks is the potential risk of test set leakage (i.e., contamination). Existing works <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>, <a href="#bib.bib36" title="" class="ltx_ref">36</a>, <a href="#bib.bib48" title="" class="ltx_ref">48</a>, <a href="#bib.bib39" title="" class="ltx_ref">39</a>]</cite> have suggested a growing risk of contamination, which undermines the reliability of benchmarks over time, motivating the need for benchmarks that are more frequently updated.</p>
</div>
<div id="S2.p3" class="ltx_para ltx_noindent">
<p id="S2.p3.1" class="ltx_p"><span id="S2.p3.1.1" class="ltx_text ltx_font_bold">Live benchmarks.</span> DynaBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite> identifies these
challenges and recommends creating living and continuously evolving benchmarks. Recent works LiveCodeBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite>, R2E <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite>, as well as the community based live evaluation, Chatbot Arena <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite>. However, none of these focus on developing a pipeline for automatic benchmark curation.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>How do you measure benchmarks?</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.4" class="ltx_p">Given a benchmark consists of a dataset <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S3.p1.1.m1.1a"><mi id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><ci id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">D</annotation></semantics></math> of prompts and scoring function <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S3.p1.2.m2.1a"><mi id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><ci id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">f</annotation></semantics></math> to measure the performance of model <math id="S3.p1.3.m3.1" class="ltx_Math" alttext="\pi_{i}" display="inline"><semantics id="S3.p1.3.m3.1a"><msub id="S3.p1.3.m3.1.1" xref="S3.p1.3.m3.1.1.cmml"><mi id="S3.p1.3.m3.1.1.2" xref="S3.p1.3.m3.1.1.2.cmml">π</mi><mi id="S3.p1.3.m3.1.1.3" xref="S3.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.3.m3.1b"><apply id="S3.p1.3.m3.1.1.cmml" xref="S3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.p1.3.m3.1.1.1.cmml" xref="S3.p1.3.m3.1.1">subscript</csymbol><ci id="S3.p1.3.m3.1.1.2.cmml" xref="S3.p1.3.m3.1.1.2">𝜋</ci><ci id="S3.p1.3.m3.1.1.3.cmml" xref="S3.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m3.1c">\pi_{i}</annotation></semantics></math> on prompt <math id="S3.p1.4.m4.1" class="ltx_Math" alttext="\mathrm{p}" display="inline"><semantics id="S3.p1.4.m4.1a"><mi mathvariant="normal" id="S3.p1.4.m4.1.1" xref="S3.p1.4.m4.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S3.p1.4.m4.1b"><ci id="S3.p1.4.m4.1.1.cmml" xref="S3.p1.4.m4.1.1">p</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.4.m4.1c">\mathrm{p}</annotation></semantics></math>, we outline two key properties that an LLM chatbot benchmark approximating human preference should possess to provide a meaningful measurement of capabilities between models:</p>
</div>
<div id="S3.p2" class="ltx_para ltx_noindent">
<ol id="S3.I1" class="ltx_enumerate">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p"><span id="S3.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Separability:</span> the benchmark should separate models with high confidence.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S3.I1.i2.p1" class="ltx_para ltx_noindent">
<p id="S3.I1.i2.p1.1" class="ltx_p"><span id="S3.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Alignment with Human Preference:</span> <math id="S3.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S3.I1.i2.p1.1.m1.1a"><mi id="S3.I1.i2.p1.1.m1.1.1" xref="S3.I1.i2.p1.1.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.1.m1.1b"><ci id="S3.I1.i2.p1.1.m1.1.1.cmml" xref="S3.I1.i2.p1.1.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.1.m1.1c">f</annotation></semantics></math> should agree with human preference.</p>
</div>
</li>
</ol>
</div>
<div id="S3.p3" class="ltx_para ltx_noindent">
<p id="S3.p3.1" class="ltx_p">While previous works have focused on alignment, separability is also a crucial consideration when comparing models of similar quality (e.g., different checkpoints from the same training run). However, achieving high-confidence separability is challenging due to limitations in prompt design and inherent variances in LLM evaluations. Overly simplistic prompts fail to distinguish between models, while the randomness in human and LLM judgments leads to inconsistent predictions. As a result, it is often difficult to confidently determine if a model’s apparent performance reflects a genuine difference in capability or merely noisy observations, highlighting a need for methods to verify whether a benchmark can reliably separate similar models.</p>
</div>
<div id="S3.p4" class="ltx_para ltx_noindent">
<p id="S3.p4.1" class="ltx_p">Statistical measures like Pearson <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite> and Spearman Correlations, commonly used in benchmarks such as AlpacaEval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite> to measure correlation to human preference ranking, may fail to adequately address model separability and ranking instability. In addition, these measures only provide a coarse signal of ranking correlation without quantifying the magnitude of performance differences between model pairs.</p>
</div>
<div id="S3.p5" class="ltx_para ltx_noindent">
<p id="S3.p5.1" class="ltx_p">To address these issues, we develop three novel metrics: <span id="S3.p5.1.1" class="ltx_text ltx_font_italic">Separability with Confidence</span>, <span id="S3.p5.1.2" class="ltx_text ltx_font_italic">Agreement with Confidence</span>, and <span id="S3.p5.1.3" class="ltx_text ltx_font_italic">Pair Rank Brier Score</span>.</p>
</div>
<div id="S3.p6" class="ltx_para ltx_noindent">
<p id="S3.p6.1" class="ltx_p"><span id="S3.p6.1.1" class="ltx_text ltx_font_bold">Separability with Confidence</span>
quantifies the benchmark’s confidence by measuring its consistency in predicting the winner of a model pair across random seeds through bootstrapping. This is done by calculating the percentage of model pairs that have non-overlapping confidence intervals of their benchmark scores. A higher percentage indicates that the benchmark is more confident in distinguishing between the performance of different models, as the confidence intervals of their scores do not overlap.</p>
</div>
<div id="S3.p7" class="ltx_para ltx_noindent">
<p id="S3.p7.1" class="ltx_p"><span id="S3.p7.1.1" class="ltx_text ltx_font_bold">Agreement with Confidence Interval</span> measures how well benchmarks A and B confidently distinguish between two models with the same ordering. Given two models <math id="S3.p7.1.m1.2" class="ltx_Math" alttext="\pi_{1},\pi_{2}" display="inline"><semantics id="S3.p7.1.m1.2a"><mrow id="S3.p7.1.m1.2.2.2" xref="S3.p7.1.m1.2.2.3.cmml"><msub id="S3.p7.1.m1.1.1.1.1" xref="S3.p7.1.m1.1.1.1.1.cmml"><mi id="S3.p7.1.m1.1.1.1.1.2" xref="S3.p7.1.m1.1.1.1.1.2.cmml">π</mi><mn id="S3.p7.1.m1.1.1.1.1.3" xref="S3.p7.1.m1.1.1.1.1.3.cmml">1</mn></msub><mo id="S3.p7.1.m1.2.2.2.3" xref="S3.p7.1.m1.2.2.3.cmml">,</mo><msub id="S3.p7.1.m1.2.2.2.2" xref="S3.p7.1.m1.2.2.2.2.cmml"><mi id="S3.p7.1.m1.2.2.2.2.2" xref="S3.p7.1.m1.2.2.2.2.2.cmml">π</mi><mn id="S3.p7.1.m1.2.2.2.2.3" xref="S3.p7.1.m1.2.2.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.p7.1.m1.2b"><list id="S3.p7.1.m1.2.2.3.cmml" xref="S3.p7.1.m1.2.2.2"><apply id="S3.p7.1.m1.1.1.1.1.cmml" xref="S3.p7.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.p7.1.m1.1.1.1.1.1.cmml" xref="S3.p7.1.m1.1.1.1.1">subscript</csymbol><ci id="S3.p7.1.m1.1.1.1.1.2.cmml" xref="S3.p7.1.m1.1.1.1.1.2">𝜋</ci><cn type="integer" id="S3.p7.1.m1.1.1.1.1.3.cmml" xref="S3.p7.1.m1.1.1.1.1.3">1</cn></apply><apply id="S3.p7.1.m1.2.2.2.2.cmml" xref="S3.p7.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S3.p7.1.m1.2.2.2.2.1.cmml" xref="S3.p7.1.m1.2.2.2.2">subscript</csymbol><ci id="S3.p7.1.m1.2.2.2.2.2.cmml" xref="S3.p7.1.m1.2.2.2.2.2">𝜋</ci><cn type="integer" id="S3.p7.1.m1.2.2.2.2.3.cmml" xref="S3.p7.1.m1.2.2.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S3.p7.1.m1.2c">\pi_{1},\pi_{2}</annotation></semantics></math> to evaluated on A and B, we assign the following scores:</p>
<ol id="S3.I2" class="ltx_enumerate">
<li id="S3.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S3.I2.i1.p1" class="ltx_para">
<p id="S3.I2.i1.p1.1" class="ltx_p">If both A and B can confidently separate <math id="S3.I2.i1.p1.1.m1.2" class="ltx_Math" alttext="\pi_{1},\pi_{2}" display="inline"><semantics id="S3.I2.i1.p1.1.m1.2a"><mrow id="S3.I2.i1.p1.1.m1.2.2.2" xref="S3.I2.i1.p1.1.m1.2.2.3.cmml"><msub id="S3.I2.i1.p1.1.m1.1.1.1.1" xref="S3.I2.i1.p1.1.m1.1.1.1.1.cmml"><mi id="S3.I2.i1.p1.1.m1.1.1.1.1.2" xref="S3.I2.i1.p1.1.m1.1.1.1.1.2.cmml">π</mi><mn id="S3.I2.i1.p1.1.m1.1.1.1.1.3" xref="S3.I2.i1.p1.1.m1.1.1.1.1.3.cmml">1</mn></msub><mo id="S3.I2.i1.p1.1.m1.2.2.2.3" xref="S3.I2.i1.p1.1.m1.2.2.3.cmml">,</mo><msub id="S3.I2.i1.p1.1.m1.2.2.2.2" xref="S3.I2.i1.p1.1.m1.2.2.2.2.cmml"><mi id="S3.I2.i1.p1.1.m1.2.2.2.2.2" xref="S3.I2.i1.p1.1.m1.2.2.2.2.2.cmml">π</mi><mn id="S3.I2.i1.p1.1.m1.2.2.2.2.3" xref="S3.I2.i1.p1.1.m1.2.2.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.I2.i1.p1.1.m1.2b"><list id="S3.I2.i1.p1.1.m1.2.2.3.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2"><apply id="S3.I2.i1.p1.1.m1.1.1.1.1.cmml" xref="S3.I2.i1.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.I2.i1.p1.1.m1.1.1.1.1.1.cmml" xref="S3.I2.i1.p1.1.m1.1.1.1.1">subscript</csymbol><ci id="S3.I2.i1.p1.1.m1.1.1.1.1.2.cmml" xref="S3.I2.i1.p1.1.m1.1.1.1.1.2">𝜋</ci><cn type="integer" id="S3.I2.i1.p1.1.m1.1.1.1.1.3.cmml" xref="S3.I2.i1.p1.1.m1.1.1.1.1.3">1</cn></apply><apply id="S3.I2.i1.p1.1.m1.2.2.2.2.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S3.I2.i1.p1.1.m1.2.2.2.2.1.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.2">subscript</csymbol><ci id="S3.I2.i1.p1.1.m1.2.2.2.2.2.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.2.2">𝜋</ci><cn type="integer" id="S3.I2.i1.p1.1.m1.2.2.2.2.3.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i1.p1.1.m1.2c">\pi_{1},\pi_{2}</annotation></semantics></math>, we assign a score of 1 if the rankings match across A and B, otherwise -1.</p>
</div>
</li>
<li id="S3.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S3.I2.i2.p1" class="ltx_para ltx_noindent">
<p id="S3.I2.i2.p1.1" class="ltx_p">If either A or B cannot separate <math id="S3.I2.i2.p1.1.m1.2" class="ltx_Math" alttext="\pi_{1},\pi_{2}" display="inline"><semantics id="S3.I2.i2.p1.1.m1.2a"><mrow id="S3.I2.i2.p1.1.m1.2.2.2" xref="S3.I2.i2.p1.1.m1.2.2.3.cmml"><msub id="S3.I2.i2.p1.1.m1.1.1.1.1" xref="S3.I2.i2.p1.1.m1.1.1.1.1.cmml"><mi id="S3.I2.i2.p1.1.m1.1.1.1.1.2" xref="S3.I2.i2.p1.1.m1.1.1.1.1.2.cmml">π</mi><mn id="S3.I2.i2.p1.1.m1.1.1.1.1.3" xref="S3.I2.i2.p1.1.m1.1.1.1.1.3.cmml">1</mn></msub><mo id="S3.I2.i2.p1.1.m1.2.2.2.3" xref="S3.I2.i2.p1.1.m1.2.2.3.cmml">,</mo><msub id="S3.I2.i2.p1.1.m1.2.2.2.2" xref="S3.I2.i2.p1.1.m1.2.2.2.2.cmml"><mi id="S3.I2.i2.p1.1.m1.2.2.2.2.2" xref="S3.I2.i2.p1.1.m1.2.2.2.2.2.cmml">π</mi><mn id="S3.I2.i2.p1.1.m1.2.2.2.2.3" xref="S3.I2.i2.p1.1.m1.2.2.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.I2.i2.p1.1.m1.2b"><list id="S3.I2.i2.p1.1.m1.2.2.3.cmml" xref="S3.I2.i2.p1.1.m1.2.2.2"><apply id="S3.I2.i2.p1.1.m1.1.1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.I2.i2.p1.1.m1.1.1.1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.1.1.1.1">subscript</csymbol><ci id="S3.I2.i2.p1.1.m1.1.1.1.1.2.cmml" xref="S3.I2.i2.p1.1.m1.1.1.1.1.2">𝜋</ci><cn type="integer" id="S3.I2.i2.p1.1.m1.1.1.1.1.3.cmml" xref="S3.I2.i2.p1.1.m1.1.1.1.1.3">1</cn></apply><apply id="S3.I2.i2.p1.1.m1.2.2.2.2.cmml" xref="S3.I2.i2.p1.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S3.I2.i2.p1.1.m1.2.2.2.2.1.cmml" xref="S3.I2.i2.p1.1.m1.2.2.2.2">subscript</csymbol><ci id="S3.I2.i2.p1.1.m1.2.2.2.2.2.cmml" xref="S3.I2.i2.p1.1.m1.2.2.2.2.2">𝜋</ci><cn type="integer" id="S3.I2.i2.p1.1.m1.2.2.2.2.3.cmml" xref="S3.I2.i2.p1.1.m1.2.2.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i2.p1.1.m1.2c">\pi_{1},\pi_{2}</annotation></semantics></math> with confidence, we assign a score of 0.</p>
</div>
</li>
</ol>
</div>
<div id="S3.p8" class="ltx_para ltx_noindent">
<p id="S3.p8.1" class="ltx_p">The final agreement score across models is the average score across all unique model pairs. An agreement score of 1 implies that benchmark A confidently agrees with benchmark B on the preference of every unique model pair. Conversely, a score of -1 indicates that A confidently disagrees with B on every unique model pair.</p>
</div>
<div id="S3.p9" class="ltx_para ltx_noindent">
<p id="S3.p9.5" class="ltx_p"><span id="S3.p9.5.1" class="ltx_text ltx_font_bold">Pair Rank Brier Score</span> further assesses an LLM benchmark’s capability to predict the ranking of a pair of competing models by rewarding confidence in correct predictions while penalizing confidence when incorrect. Consider two models <math id="S3.p9.1.m1.1" class="ltx_Math" alttext="\pi_{1}&gt;\pi_{2}" display="inline"><semantics id="S3.p9.1.m1.1a"><mrow id="S3.p9.1.m1.1.1" xref="S3.p9.1.m1.1.1.cmml"><msub id="S3.p9.1.m1.1.1.2" xref="S3.p9.1.m1.1.1.2.cmml"><mi id="S3.p9.1.m1.1.1.2.2" xref="S3.p9.1.m1.1.1.2.2.cmml">π</mi><mn id="S3.p9.1.m1.1.1.2.3" xref="S3.p9.1.m1.1.1.2.3.cmml">1</mn></msub><mo id="S3.p9.1.m1.1.1.1" xref="S3.p9.1.m1.1.1.1.cmml">&gt;</mo><msub id="S3.p9.1.m1.1.1.3" xref="S3.p9.1.m1.1.1.3.cmml"><mi id="S3.p9.1.m1.1.1.3.2" xref="S3.p9.1.m1.1.1.3.2.cmml">π</mi><mn id="S3.p9.1.m1.1.1.3.3" xref="S3.p9.1.m1.1.1.3.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.p9.1.m1.1b"><apply id="S3.p9.1.m1.1.1.cmml" xref="S3.p9.1.m1.1.1"><gt id="S3.p9.1.m1.1.1.1.cmml" xref="S3.p9.1.m1.1.1.1"></gt><apply id="S3.p9.1.m1.1.1.2.cmml" xref="S3.p9.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.p9.1.m1.1.1.2.1.cmml" xref="S3.p9.1.m1.1.1.2">subscript</csymbol><ci id="S3.p9.1.m1.1.1.2.2.cmml" xref="S3.p9.1.m1.1.1.2.2">𝜋</ci><cn type="integer" id="S3.p9.1.m1.1.1.2.3.cmml" xref="S3.p9.1.m1.1.1.2.3">1</cn></apply><apply id="S3.p9.1.m1.1.1.3.cmml" xref="S3.p9.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.p9.1.m1.1.1.3.1.cmml" xref="S3.p9.1.m1.1.1.3">subscript</csymbol><ci id="S3.p9.1.m1.1.1.3.2.cmml" xref="S3.p9.1.m1.1.1.3.2">𝜋</ci><cn type="integer" id="S3.p9.1.m1.1.1.3.3.cmml" xref="S3.p9.1.m1.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p9.1.m1.1c">\pi_{1}&gt;\pi_{2}</annotation></semantics></math> with disparate quality. Although two benchmarks A and B predict the same ranking <math id="S3.p9.2.m2.1" class="ltx_Math" alttext="\pi_{1}&gt;\pi_{2}" display="inline"><semantics id="S3.p9.2.m2.1a"><mrow id="S3.p9.2.m2.1.1" xref="S3.p9.2.m2.1.1.cmml"><msub id="S3.p9.2.m2.1.1.2" xref="S3.p9.2.m2.1.1.2.cmml"><mi id="S3.p9.2.m2.1.1.2.2" xref="S3.p9.2.m2.1.1.2.2.cmml">π</mi><mn id="S3.p9.2.m2.1.1.2.3" xref="S3.p9.2.m2.1.1.2.3.cmml">1</mn></msub><mo id="S3.p9.2.m2.1.1.1" xref="S3.p9.2.m2.1.1.1.cmml">&gt;</mo><msub id="S3.p9.2.m2.1.1.3" xref="S3.p9.2.m2.1.1.3.cmml"><mi id="S3.p9.2.m2.1.1.3.2" xref="S3.p9.2.m2.1.1.3.2.cmml">π</mi><mn id="S3.p9.2.m2.1.1.3.3" xref="S3.p9.2.m2.1.1.3.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.p9.2.m2.1b"><apply id="S3.p9.2.m2.1.1.cmml" xref="S3.p9.2.m2.1.1"><gt id="S3.p9.2.m2.1.1.1.cmml" xref="S3.p9.2.m2.1.1.1"></gt><apply id="S3.p9.2.m2.1.1.2.cmml" xref="S3.p9.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.p9.2.m2.1.1.2.1.cmml" xref="S3.p9.2.m2.1.1.2">subscript</csymbol><ci id="S3.p9.2.m2.1.1.2.2.cmml" xref="S3.p9.2.m2.1.1.2.2">𝜋</ci><cn type="integer" id="S3.p9.2.m2.1.1.2.3.cmml" xref="S3.p9.2.m2.1.1.2.3">1</cn></apply><apply id="S3.p9.2.m2.1.1.3.cmml" xref="S3.p9.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.p9.2.m2.1.1.3.1.cmml" xref="S3.p9.2.m2.1.1.3">subscript</csymbol><ci id="S3.p9.2.m2.1.1.3.2.cmml" xref="S3.p9.2.m2.1.1.3.2">𝜋</ci><cn type="integer" id="S3.p9.2.m2.1.1.3.3.cmml" xref="S3.p9.2.m2.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p9.2.m2.1c">\pi_{1}&gt;\pi_{2}</annotation></semantics></math>, they predict <math id="S3.p9.3.m3.1" class="ltx_Math" alttext="P(\pi_{1}&gt;\pi_{2})" display="inline"><semantics id="S3.p9.3.m3.1a"><mrow id="S3.p9.3.m3.1.1" xref="S3.p9.3.m3.1.1.cmml"><mi id="S3.p9.3.m3.1.1.3" xref="S3.p9.3.m3.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="S3.p9.3.m3.1.1.2" xref="S3.p9.3.m3.1.1.2.cmml">​</mo><mrow id="S3.p9.3.m3.1.1.1.1" xref="S3.p9.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.p9.3.m3.1.1.1.1.2" xref="S3.p9.3.m3.1.1.1.1.1.cmml">(</mo><mrow id="S3.p9.3.m3.1.1.1.1.1" xref="S3.p9.3.m3.1.1.1.1.1.cmml"><msub id="S3.p9.3.m3.1.1.1.1.1.2" xref="S3.p9.3.m3.1.1.1.1.1.2.cmml"><mi id="S3.p9.3.m3.1.1.1.1.1.2.2" xref="S3.p9.3.m3.1.1.1.1.1.2.2.cmml">π</mi><mn id="S3.p9.3.m3.1.1.1.1.1.2.3" xref="S3.p9.3.m3.1.1.1.1.1.2.3.cmml">1</mn></msub><mo id="S3.p9.3.m3.1.1.1.1.1.1" xref="S3.p9.3.m3.1.1.1.1.1.1.cmml">&gt;</mo><msub id="S3.p9.3.m3.1.1.1.1.1.3" xref="S3.p9.3.m3.1.1.1.1.1.3.cmml"><mi id="S3.p9.3.m3.1.1.1.1.1.3.2" xref="S3.p9.3.m3.1.1.1.1.1.3.2.cmml">π</mi><mn id="S3.p9.3.m3.1.1.1.1.1.3.3" xref="S3.p9.3.m3.1.1.1.1.1.3.3.cmml">2</mn></msub></mrow><mo stretchy="false" id="S3.p9.3.m3.1.1.1.1.3" xref="S3.p9.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p9.3.m3.1b"><apply id="S3.p9.3.m3.1.1.cmml" xref="S3.p9.3.m3.1.1"><times id="S3.p9.3.m3.1.1.2.cmml" xref="S3.p9.3.m3.1.1.2"></times><ci id="S3.p9.3.m3.1.1.3.cmml" xref="S3.p9.3.m3.1.1.3">𝑃</ci><apply id="S3.p9.3.m3.1.1.1.1.1.cmml" xref="S3.p9.3.m3.1.1.1.1"><gt id="S3.p9.3.m3.1.1.1.1.1.1.cmml" xref="S3.p9.3.m3.1.1.1.1.1.1"></gt><apply id="S3.p9.3.m3.1.1.1.1.1.2.cmml" xref="S3.p9.3.m3.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.p9.3.m3.1.1.1.1.1.2.1.cmml" xref="S3.p9.3.m3.1.1.1.1.1.2">subscript</csymbol><ci id="S3.p9.3.m3.1.1.1.1.1.2.2.cmml" xref="S3.p9.3.m3.1.1.1.1.1.2.2">𝜋</ci><cn type="integer" id="S3.p9.3.m3.1.1.1.1.1.2.3.cmml" xref="S3.p9.3.m3.1.1.1.1.1.2.3">1</cn></apply><apply id="S3.p9.3.m3.1.1.1.1.1.3.cmml" xref="S3.p9.3.m3.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.p9.3.m3.1.1.1.1.1.3.1.cmml" xref="S3.p9.3.m3.1.1.1.1.1.3">subscript</csymbol><ci id="S3.p9.3.m3.1.1.1.1.1.3.2.cmml" xref="S3.p9.3.m3.1.1.1.1.1.3.2">𝜋</ci><cn type="integer" id="S3.p9.3.m3.1.1.1.1.1.3.3.cmml" xref="S3.p9.3.m3.1.1.1.1.1.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p9.3.m3.1c">P(\pi_{1}&gt;\pi_{2})</annotation></semantics></math> as <math id="S3.p9.4.m4.1" class="ltx_Math" alttext=".60" display="inline"><semantics id="S3.p9.4.m4.1a"><mn id="S3.p9.4.m4.1.1" xref="S3.p9.4.m4.1.1.cmml">.60</mn><annotation-xml encoding="MathML-Content" id="S3.p9.4.m4.1b"><cn type="float" id="S3.p9.4.m4.1.1.cmml" xref="S3.p9.4.m4.1.1">.60</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p9.4.m4.1c">.60</annotation></semantics></math> and <math id="S3.p9.5.m5.1" class="ltx_Math" alttext=".90" display="inline"><semantics id="S3.p9.5.m5.1a"><mn id="S3.p9.5.m5.1.1" xref="S3.p9.5.m5.1.1.cmml">.90</mn><annotation-xml encoding="MathML-Content" id="S3.p9.5.m5.1b"><cn type="float" id="S3.p9.5.m5.1.1.cmml" xref="S3.p9.5.m5.1.1">.90</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p9.5.m5.1c">.90</annotation></semantics></math>, respectively (undetectable by Spearman correlation). These benchmarks would result in very different Brier scores, reflecting their ability to quantify the magnitude of performance difference between the models. If both benchmarks give the wrong prediction of the winner, we prefer the benchmark with a less confident prediction. Brier score weighs a benchmark’s accuracy and its ability to quantify the appropriate level of uncertainty in its predictions.</p>
</div>
<div id="S3.p10" class="ltx_para ltx_noindent">
<p id="S3.p10.9" class="ltx_p">For each model pair, the benchmark’s forecast probability <math id="S3.p10.1.m1.1" class="ltx_Math" alttext="f(\pi_{i}&lt;\pi_{j})" display="inline"><semantics id="S3.p10.1.m1.1a"><mrow id="S3.p10.1.m1.1.1" xref="S3.p10.1.m1.1.1.cmml"><mi id="S3.p10.1.m1.1.1.3" xref="S3.p10.1.m1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.p10.1.m1.1.1.2" xref="S3.p10.1.m1.1.1.2.cmml">​</mo><mrow id="S3.p10.1.m1.1.1.1.1" xref="S3.p10.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.p10.1.m1.1.1.1.1.2" xref="S3.p10.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S3.p10.1.m1.1.1.1.1.1" xref="S3.p10.1.m1.1.1.1.1.1.cmml"><msub id="S3.p10.1.m1.1.1.1.1.1.2" xref="S3.p10.1.m1.1.1.1.1.1.2.cmml"><mi id="S3.p10.1.m1.1.1.1.1.1.2.2" xref="S3.p10.1.m1.1.1.1.1.1.2.2.cmml">π</mi><mi id="S3.p10.1.m1.1.1.1.1.1.2.3" xref="S3.p10.1.m1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo id="S3.p10.1.m1.1.1.1.1.1.1" xref="S3.p10.1.m1.1.1.1.1.1.1.cmml">&lt;</mo><msub id="S3.p10.1.m1.1.1.1.1.1.3" xref="S3.p10.1.m1.1.1.1.1.1.3.cmml"><mi id="S3.p10.1.m1.1.1.1.1.1.3.2" xref="S3.p10.1.m1.1.1.1.1.1.3.2.cmml">π</mi><mi id="S3.p10.1.m1.1.1.1.1.1.3.3" xref="S3.p10.1.m1.1.1.1.1.1.3.3.cmml">j</mi></msub></mrow><mo stretchy="false" id="S3.p10.1.m1.1.1.1.1.3" xref="S3.p10.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p10.1.m1.1b"><apply id="S3.p10.1.m1.1.1.cmml" xref="S3.p10.1.m1.1.1"><times id="S3.p10.1.m1.1.1.2.cmml" xref="S3.p10.1.m1.1.1.2"></times><ci id="S3.p10.1.m1.1.1.3.cmml" xref="S3.p10.1.m1.1.1.3">𝑓</ci><apply id="S3.p10.1.m1.1.1.1.1.1.cmml" xref="S3.p10.1.m1.1.1.1.1"><lt id="S3.p10.1.m1.1.1.1.1.1.1.cmml" xref="S3.p10.1.m1.1.1.1.1.1.1"></lt><apply id="S3.p10.1.m1.1.1.1.1.1.2.cmml" xref="S3.p10.1.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.p10.1.m1.1.1.1.1.1.2.1.cmml" xref="S3.p10.1.m1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.p10.1.m1.1.1.1.1.1.2.2.cmml" xref="S3.p10.1.m1.1.1.1.1.1.2.2">𝜋</ci><ci id="S3.p10.1.m1.1.1.1.1.1.2.3.cmml" xref="S3.p10.1.m1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S3.p10.1.m1.1.1.1.1.1.3.cmml" xref="S3.p10.1.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.p10.1.m1.1.1.1.1.1.3.1.cmml" xref="S3.p10.1.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.p10.1.m1.1.1.1.1.1.3.2.cmml" xref="S3.p10.1.m1.1.1.1.1.1.3.2">𝜋</ci><ci id="S3.p10.1.m1.1.1.1.1.1.3.3.cmml" xref="S3.p10.1.m1.1.1.1.1.1.3.3">𝑗</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p10.1.m1.1c">f(\pi_{i}&lt;\pi_{j})</annotation></semantics></math> can be estimated as <math id="S3.p10.2.m2.1" class="ltx_Math" alttext="P(f(\pi_{i})&lt;f(\pi_{j}))" display="inline"><semantics id="S3.p10.2.m2.1a"><mrow id="S3.p10.2.m2.1.1" xref="S3.p10.2.m2.1.1.cmml"><mi id="S3.p10.2.m2.1.1.3" xref="S3.p10.2.m2.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="S3.p10.2.m2.1.1.2" xref="S3.p10.2.m2.1.1.2.cmml">​</mo><mrow id="S3.p10.2.m2.1.1.1.1" xref="S3.p10.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.p10.2.m2.1.1.1.1.2" xref="S3.p10.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="S3.p10.2.m2.1.1.1.1.1" xref="S3.p10.2.m2.1.1.1.1.1.cmml"><mrow id="S3.p10.2.m2.1.1.1.1.1.1" xref="S3.p10.2.m2.1.1.1.1.1.1.cmml"><mi id="S3.p10.2.m2.1.1.1.1.1.1.3" xref="S3.p10.2.m2.1.1.1.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.p10.2.m2.1.1.1.1.1.1.2" xref="S3.p10.2.m2.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.p10.2.m2.1.1.1.1.1.1.1.1" xref="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.p10.2.m2.1.1.1.1.1.1.1.1.2" xref="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.p10.2.m2.1.1.1.1.1.1.1.1.1" xref="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.2" xref="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.2.cmml">π</mi><mi id="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.3" xref="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.p10.2.m2.1.1.1.1.1.1.1.1.3" xref="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.p10.2.m2.1.1.1.1.1.3" xref="S3.p10.2.m2.1.1.1.1.1.3.cmml">&lt;</mo><mrow id="S3.p10.2.m2.1.1.1.1.1.2" xref="S3.p10.2.m2.1.1.1.1.1.2.cmml"><mi id="S3.p10.2.m2.1.1.1.1.1.2.3" xref="S3.p10.2.m2.1.1.1.1.1.2.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.p10.2.m2.1.1.1.1.1.2.2" xref="S3.p10.2.m2.1.1.1.1.1.2.2.cmml">​</mo><mrow id="S3.p10.2.m2.1.1.1.1.1.2.1.1" xref="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="S3.p10.2.m2.1.1.1.1.1.2.1.1.2" xref="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.cmml">(</mo><msub id="S3.p10.2.m2.1.1.1.1.1.2.1.1.1" xref="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.cmml"><mi id="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.2" xref="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.2.cmml">π</mi><mi id="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.3" xref="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.3.cmml">j</mi></msub><mo stretchy="false" id="S3.p10.2.m2.1.1.1.1.1.2.1.1.3" xref="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.p10.2.m2.1.1.1.1.3" xref="S3.p10.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p10.2.m2.1b"><apply id="S3.p10.2.m2.1.1.cmml" xref="S3.p10.2.m2.1.1"><times id="S3.p10.2.m2.1.1.2.cmml" xref="S3.p10.2.m2.1.1.2"></times><ci id="S3.p10.2.m2.1.1.3.cmml" xref="S3.p10.2.m2.1.1.3">𝑃</ci><apply id="S3.p10.2.m2.1.1.1.1.1.cmml" xref="S3.p10.2.m2.1.1.1.1"><lt id="S3.p10.2.m2.1.1.1.1.1.3.cmml" xref="S3.p10.2.m2.1.1.1.1.1.3"></lt><apply id="S3.p10.2.m2.1.1.1.1.1.1.cmml" xref="S3.p10.2.m2.1.1.1.1.1.1"><times id="S3.p10.2.m2.1.1.1.1.1.1.2.cmml" xref="S3.p10.2.m2.1.1.1.1.1.1.2"></times><ci id="S3.p10.2.m2.1.1.1.1.1.1.3.cmml" xref="S3.p10.2.m2.1.1.1.1.1.1.3">𝑓</ci><apply id="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.p10.2.m2.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.p10.2.m2.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.2">𝜋</ci><ci id="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.p10.2.m2.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply><apply id="S3.p10.2.m2.1.1.1.1.1.2.cmml" xref="S3.p10.2.m2.1.1.1.1.1.2"><times id="S3.p10.2.m2.1.1.1.1.1.2.2.cmml" xref="S3.p10.2.m2.1.1.1.1.1.2.2"></times><ci id="S3.p10.2.m2.1.1.1.1.1.2.3.cmml" xref="S3.p10.2.m2.1.1.1.1.1.2.3">𝑓</ci><apply id="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.cmml" xref="S3.p10.2.m2.1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.1.cmml" xref="S3.p10.2.m2.1.1.1.1.1.2.1.1">subscript</csymbol><ci id="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.2.cmml" xref="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.2">𝜋</ci><ci id="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.3.cmml" xref="S3.p10.2.m2.1.1.1.1.1.2.1.1.1.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p10.2.m2.1c">P(f(\pi_{i})&lt;f(\pi_{j}))</annotation></semantics></math> with <math id="S3.p10.3.m3.3" class="ltx_Math" alttext="f(\pi_{i})\sim\mathcal{N}(\mu_{i},\sigma_{i}^{2})" display="inline"><semantics id="S3.p10.3.m3.3a"><mrow id="S3.p10.3.m3.3.3" xref="S3.p10.3.m3.3.3.cmml"><mrow id="S3.p10.3.m3.1.1.1" xref="S3.p10.3.m3.1.1.1.cmml"><mi id="S3.p10.3.m3.1.1.1.3" xref="S3.p10.3.m3.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.p10.3.m3.1.1.1.2" xref="S3.p10.3.m3.1.1.1.2.cmml">​</mo><mrow id="S3.p10.3.m3.1.1.1.1.1" xref="S3.p10.3.m3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.p10.3.m3.1.1.1.1.1.2" xref="S3.p10.3.m3.1.1.1.1.1.1.cmml">(</mo><msub id="S3.p10.3.m3.1.1.1.1.1.1" xref="S3.p10.3.m3.1.1.1.1.1.1.cmml"><mi id="S3.p10.3.m3.1.1.1.1.1.1.2" xref="S3.p10.3.m3.1.1.1.1.1.1.2.cmml">π</mi><mi id="S3.p10.3.m3.1.1.1.1.1.1.3" xref="S3.p10.3.m3.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.p10.3.m3.1.1.1.1.1.3" xref="S3.p10.3.m3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.p10.3.m3.3.3.4" xref="S3.p10.3.m3.3.3.4.cmml">∼</mo><mrow id="S3.p10.3.m3.3.3.3" xref="S3.p10.3.m3.3.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p10.3.m3.3.3.3.4" xref="S3.p10.3.m3.3.3.3.4.cmml">𝒩</mi><mo lspace="0em" rspace="0em" id="S3.p10.3.m3.3.3.3.3" xref="S3.p10.3.m3.3.3.3.3.cmml">​</mo><mrow id="S3.p10.3.m3.3.3.3.2.2" xref="S3.p10.3.m3.3.3.3.2.3.cmml"><mo stretchy="false" id="S3.p10.3.m3.3.3.3.2.2.3" xref="S3.p10.3.m3.3.3.3.2.3.cmml">(</mo><msub id="S3.p10.3.m3.2.2.2.1.1.1" xref="S3.p10.3.m3.2.2.2.1.1.1.cmml"><mi id="S3.p10.3.m3.2.2.2.1.1.1.2" xref="S3.p10.3.m3.2.2.2.1.1.1.2.cmml">μ</mi><mi id="S3.p10.3.m3.2.2.2.1.1.1.3" xref="S3.p10.3.m3.2.2.2.1.1.1.3.cmml">i</mi></msub><mo id="S3.p10.3.m3.3.3.3.2.2.4" xref="S3.p10.3.m3.3.3.3.2.3.cmml">,</mo><msubsup id="S3.p10.3.m3.3.3.3.2.2.2" xref="S3.p10.3.m3.3.3.3.2.2.2.cmml"><mi id="S3.p10.3.m3.3.3.3.2.2.2.2.2" xref="S3.p10.3.m3.3.3.3.2.2.2.2.2.cmml">σ</mi><mi id="S3.p10.3.m3.3.3.3.2.2.2.2.3" xref="S3.p10.3.m3.3.3.3.2.2.2.2.3.cmml">i</mi><mn id="S3.p10.3.m3.3.3.3.2.2.2.3" xref="S3.p10.3.m3.3.3.3.2.2.2.3.cmml">2</mn></msubsup><mo stretchy="false" id="S3.p10.3.m3.3.3.3.2.2.5" xref="S3.p10.3.m3.3.3.3.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p10.3.m3.3b"><apply id="S3.p10.3.m3.3.3.cmml" xref="S3.p10.3.m3.3.3"><csymbol cd="latexml" id="S3.p10.3.m3.3.3.4.cmml" xref="S3.p10.3.m3.3.3.4">similar-to</csymbol><apply id="S3.p10.3.m3.1.1.1.cmml" xref="S3.p10.3.m3.1.1.1"><times id="S3.p10.3.m3.1.1.1.2.cmml" xref="S3.p10.3.m3.1.1.1.2"></times><ci id="S3.p10.3.m3.1.1.1.3.cmml" xref="S3.p10.3.m3.1.1.1.3">𝑓</ci><apply id="S3.p10.3.m3.1.1.1.1.1.1.cmml" xref="S3.p10.3.m3.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.p10.3.m3.1.1.1.1.1.1.1.cmml" xref="S3.p10.3.m3.1.1.1.1.1">subscript</csymbol><ci id="S3.p10.3.m3.1.1.1.1.1.1.2.cmml" xref="S3.p10.3.m3.1.1.1.1.1.1.2">𝜋</ci><ci id="S3.p10.3.m3.1.1.1.1.1.1.3.cmml" xref="S3.p10.3.m3.1.1.1.1.1.1.3">𝑖</ci></apply></apply><apply id="S3.p10.3.m3.3.3.3.cmml" xref="S3.p10.3.m3.3.3.3"><times id="S3.p10.3.m3.3.3.3.3.cmml" xref="S3.p10.3.m3.3.3.3.3"></times><ci id="S3.p10.3.m3.3.3.3.4.cmml" xref="S3.p10.3.m3.3.3.3.4">𝒩</ci><interval closure="open" id="S3.p10.3.m3.3.3.3.2.3.cmml" xref="S3.p10.3.m3.3.3.3.2.2"><apply id="S3.p10.3.m3.2.2.2.1.1.1.cmml" xref="S3.p10.3.m3.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.p10.3.m3.2.2.2.1.1.1.1.cmml" xref="S3.p10.3.m3.2.2.2.1.1.1">subscript</csymbol><ci id="S3.p10.3.m3.2.2.2.1.1.1.2.cmml" xref="S3.p10.3.m3.2.2.2.1.1.1.2">𝜇</ci><ci id="S3.p10.3.m3.2.2.2.1.1.1.3.cmml" xref="S3.p10.3.m3.2.2.2.1.1.1.3">𝑖</ci></apply><apply id="S3.p10.3.m3.3.3.3.2.2.2.cmml" xref="S3.p10.3.m3.3.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.p10.3.m3.3.3.3.2.2.2.1.cmml" xref="S3.p10.3.m3.3.3.3.2.2.2">superscript</csymbol><apply id="S3.p10.3.m3.3.3.3.2.2.2.2.cmml" xref="S3.p10.3.m3.3.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.p10.3.m3.3.3.3.2.2.2.2.1.cmml" xref="S3.p10.3.m3.3.3.3.2.2.2">subscript</csymbol><ci id="S3.p10.3.m3.3.3.3.2.2.2.2.2.cmml" xref="S3.p10.3.m3.3.3.3.2.2.2.2.2">𝜎</ci><ci id="S3.p10.3.m3.3.3.3.2.2.2.2.3.cmml" xref="S3.p10.3.m3.3.3.3.2.2.2.2.3">𝑖</ci></apply><cn type="integer" id="S3.p10.3.m3.3.3.3.2.2.2.3.cmml" xref="S3.p10.3.m3.3.3.3.2.2.2.3">2</cn></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p10.3.m3.3c">f(\pi_{i})\sim\mathcal{N}(\mu_{i},\sigma_{i}^{2})</annotation></semantics></math> and <math id="S3.p10.4.m4.3" class="ltx_Math" alttext="f(\pi_{j})\sim\mathcal{N}(\mu_{j},\sigma_{j}^{2})" display="inline"><semantics id="S3.p10.4.m4.3a"><mrow id="S3.p10.4.m4.3.3" xref="S3.p10.4.m4.3.3.cmml"><mrow id="S3.p10.4.m4.1.1.1" xref="S3.p10.4.m4.1.1.1.cmml"><mi id="S3.p10.4.m4.1.1.1.3" xref="S3.p10.4.m4.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.p10.4.m4.1.1.1.2" xref="S3.p10.4.m4.1.1.1.2.cmml">​</mo><mrow id="S3.p10.4.m4.1.1.1.1.1" xref="S3.p10.4.m4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.p10.4.m4.1.1.1.1.1.2" xref="S3.p10.4.m4.1.1.1.1.1.1.cmml">(</mo><msub id="S3.p10.4.m4.1.1.1.1.1.1" xref="S3.p10.4.m4.1.1.1.1.1.1.cmml"><mi id="S3.p10.4.m4.1.1.1.1.1.1.2" xref="S3.p10.4.m4.1.1.1.1.1.1.2.cmml">π</mi><mi id="S3.p10.4.m4.1.1.1.1.1.1.3" xref="S3.p10.4.m4.1.1.1.1.1.1.3.cmml">j</mi></msub><mo stretchy="false" id="S3.p10.4.m4.1.1.1.1.1.3" xref="S3.p10.4.m4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.p10.4.m4.3.3.4" xref="S3.p10.4.m4.3.3.4.cmml">∼</mo><mrow id="S3.p10.4.m4.3.3.3" xref="S3.p10.4.m4.3.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p10.4.m4.3.3.3.4" xref="S3.p10.4.m4.3.3.3.4.cmml">𝒩</mi><mo lspace="0em" rspace="0em" id="S3.p10.4.m4.3.3.3.3" xref="S3.p10.4.m4.3.3.3.3.cmml">​</mo><mrow id="S3.p10.4.m4.3.3.3.2.2" xref="S3.p10.4.m4.3.3.3.2.3.cmml"><mo stretchy="false" id="S3.p10.4.m4.3.3.3.2.2.3" xref="S3.p10.4.m4.3.3.3.2.3.cmml">(</mo><msub id="S3.p10.4.m4.2.2.2.1.1.1" xref="S3.p10.4.m4.2.2.2.1.1.1.cmml"><mi id="S3.p10.4.m4.2.2.2.1.1.1.2" xref="S3.p10.4.m4.2.2.2.1.1.1.2.cmml">μ</mi><mi id="S3.p10.4.m4.2.2.2.1.1.1.3" xref="S3.p10.4.m4.2.2.2.1.1.1.3.cmml">j</mi></msub><mo id="S3.p10.4.m4.3.3.3.2.2.4" xref="S3.p10.4.m4.3.3.3.2.3.cmml">,</mo><msubsup id="S3.p10.4.m4.3.3.3.2.2.2" xref="S3.p10.4.m4.3.3.3.2.2.2.cmml"><mi id="S3.p10.4.m4.3.3.3.2.2.2.2.2" xref="S3.p10.4.m4.3.3.3.2.2.2.2.2.cmml">σ</mi><mi id="S3.p10.4.m4.3.3.3.2.2.2.2.3" xref="S3.p10.4.m4.3.3.3.2.2.2.2.3.cmml">j</mi><mn id="S3.p10.4.m4.3.3.3.2.2.2.3" xref="S3.p10.4.m4.3.3.3.2.2.2.3.cmml">2</mn></msubsup><mo stretchy="false" id="S3.p10.4.m4.3.3.3.2.2.5" xref="S3.p10.4.m4.3.3.3.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p10.4.m4.3b"><apply id="S3.p10.4.m4.3.3.cmml" xref="S3.p10.4.m4.3.3"><csymbol cd="latexml" id="S3.p10.4.m4.3.3.4.cmml" xref="S3.p10.4.m4.3.3.4">similar-to</csymbol><apply id="S3.p10.4.m4.1.1.1.cmml" xref="S3.p10.4.m4.1.1.1"><times id="S3.p10.4.m4.1.1.1.2.cmml" xref="S3.p10.4.m4.1.1.1.2"></times><ci id="S3.p10.4.m4.1.1.1.3.cmml" xref="S3.p10.4.m4.1.1.1.3">𝑓</ci><apply id="S3.p10.4.m4.1.1.1.1.1.1.cmml" xref="S3.p10.4.m4.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.p10.4.m4.1.1.1.1.1.1.1.cmml" xref="S3.p10.4.m4.1.1.1.1.1">subscript</csymbol><ci id="S3.p10.4.m4.1.1.1.1.1.1.2.cmml" xref="S3.p10.4.m4.1.1.1.1.1.1.2">𝜋</ci><ci id="S3.p10.4.m4.1.1.1.1.1.1.3.cmml" xref="S3.p10.4.m4.1.1.1.1.1.1.3">𝑗</ci></apply></apply><apply id="S3.p10.4.m4.3.3.3.cmml" xref="S3.p10.4.m4.3.3.3"><times id="S3.p10.4.m4.3.3.3.3.cmml" xref="S3.p10.4.m4.3.3.3.3"></times><ci id="S3.p10.4.m4.3.3.3.4.cmml" xref="S3.p10.4.m4.3.3.3.4">𝒩</ci><interval closure="open" id="S3.p10.4.m4.3.3.3.2.3.cmml" xref="S3.p10.4.m4.3.3.3.2.2"><apply id="S3.p10.4.m4.2.2.2.1.1.1.cmml" xref="S3.p10.4.m4.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.p10.4.m4.2.2.2.1.1.1.1.cmml" xref="S3.p10.4.m4.2.2.2.1.1.1">subscript</csymbol><ci id="S3.p10.4.m4.2.2.2.1.1.1.2.cmml" xref="S3.p10.4.m4.2.2.2.1.1.1.2">𝜇</ci><ci id="S3.p10.4.m4.2.2.2.1.1.1.3.cmml" xref="S3.p10.4.m4.2.2.2.1.1.1.3">𝑗</ci></apply><apply id="S3.p10.4.m4.3.3.3.2.2.2.cmml" xref="S3.p10.4.m4.3.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.p10.4.m4.3.3.3.2.2.2.1.cmml" xref="S3.p10.4.m4.3.3.3.2.2.2">superscript</csymbol><apply id="S3.p10.4.m4.3.3.3.2.2.2.2.cmml" xref="S3.p10.4.m4.3.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.p10.4.m4.3.3.3.2.2.2.2.1.cmml" xref="S3.p10.4.m4.3.3.3.2.2.2">subscript</csymbol><ci id="S3.p10.4.m4.3.3.3.2.2.2.2.2.cmml" xref="S3.p10.4.m4.3.3.3.2.2.2.2.2">𝜎</ci><ci id="S3.p10.4.m4.3.3.3.2.2.2.2.3.cmml" xref="S3.p10.4.m4.3.3.3.2.2.2.2.3">𝑗</ci></apply><cn type="integer" id="S3.p10.4.m4.3.3.3.2.2.2.3.cmml" xref="S3.p10.4.m4.3.3.3.2.2.2.3">2</cn></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p10.4.m4.3c">f(\pi_{j})\sim\mathcal{N}(\mu_{j},\sigma_{j}^{2})</annotation></semantics></math>, where <math id="S3.p10.5.m5.1" class="ltx_Math" alttext="\mu_{i}" display="inline"><semantics id="S3.p10.5.m5.1a"><msub id="S3.p10.5.m5.1.1" xref="S3.p10.5.m5.1.1.cmml"><mi id="S3.p10.5.m5.1.1.2" xref="S3.p10.5.m5.1.1.2.cmml">μ</mi><mi id="S3.p10.5.m5.1.1.3" xref="S3.p10.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p10.5.m5.1b"><apply id="S3.p10.5.m5.1.1.cmml" xref="S3.p10.5.m5.1.1"><csymbol cd="ambiguous" id="S3.p10.5.m5.1.1.1.cmml" xref="S3.p10.5.m5.1.1">subscript</csymbol><ci id="S3.p10.5.m5.1.1.2.cmml" xref="S3.p10.5.m5.1.1.2">𝜇</ci><ci id="S3.p10.5.m5.1.1.3.cmml" xref="S3.p10.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p10.5.m5.1c">\mu_{i}</annotation></semantics></math> and <math id="S3.p10.6.m6.1" class="ltx_Math" alttext="\sigma^{2}_{i}" display="inline"><semantics id="S3.p10.6.m6.1a"><msubsup id="S3.p10.6.m6.1.1" xref="S3.p10.6.m6.1.1.cmml"><mi id="S3.p10.6.m6.1.1.2.2" xref="S3.p10.6.m6.1.1.2.2.cmml">σ</mi><mi id="S3.p10.6.m6.1.1.3" xref="S3.p10.6.m6.1.1.3.cmml">i</mi><mn id="S3.p10.6.m6.1.1.2.3" xref="S3.p10.6.m6.1.1.2.3.cmml">2</mn></msubsup><annotation-xml encoding="MathML-Content" id="S3.p10.6.m6.1b"><apply id="S3.p10.6.m6.1.1.cmml" xref="S3.p10.6.m6.1.1"><csymbol cd="ambiguous" id="S3.p10.6.m6.1.1.1.cmml" xref="S3.p10.6.m6.1.1">subscript</csymbol><apply id="S3.p10.6.m6.1.1.2.cmml" xref="S3.p10.6.m6.1.1"><csymbol cd="ambiguous" id="S3.p10.6.m6.1.1.2.1.cmml" xref="S3.p10.6.m6.1.1">superscript</csymbol><ci id="S3.p10.6.m6.1.1.2.2.cmml" xref="S3.p10.6.m6.1.1.2.2">𝜎</ci><cn type="integer" id="S3.p10.6.m6.1.1.2.3.cmml" xref="S3.p10.6.m6.1.1.2.3">2</cn></apply><ci id="S3.p10.6.m6.1.1.3.cmml" xref="S3.p10.6.m6.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p10.6.m6.1c">\sigma^{2}_{i}</annotation></semantics></math> are the bootstrapped mean and variance by evaluating multiple times. The Brier Score Loss is then calculated over the benchmark’s prediction for each model pair with respect to the ground truth probability, where Brier Score is defined as <math id="S3.p10.7.m7.3" class="ltx_Math" alttext="\frac{1}{N}\sum_{\{i,j\}}(f(\pi_{j}&lt;\pi_{i})-O(\pi_{j}&lt;\pi_{i}))^{2}" display="inline"><semantics id="S3.p10.7.m7.3a"><mrow id="S3.p10.7.m7.3.3" xref="S3.p10.7.m7.3.3.cmml"><mfrac id="S3.p10.7.m7.3.3.3" xref="S3.p10.7.m7.3.3.3.cmml"><mn id="S3.p10.7.m7.3.3.3.2" xref="S3.p10.7.m7.3.3.3.2.cmml">1</mn><mi id="S3.p10.7.m7.3.3.3.3" xref="S3.p10.7.m7.3.3.3.3.cmml">N</mi></mfrac><mo lspace="0em" rspace="0em" id="S3.p10.7.m7.3.3.2" xref="S3.p10.7.m7.3.3.2.cmml">​</mo><mrow id="S3.p10.7.m7.3.3.1" xref="S3.p10.7.m7.3.3.1.cmml"><msub id="S3.p10.7.m7.3.3.1.2" xref="S3.p10.7.m7.3.3.1.2.cmml"><mo rspace="0em" id="S3.p10.7.m7.3.3.1.2.2" xref="S3.p10.7.m7.3.3.1.2.2.cmml">∑</mo><mrow id="S3.p10.7.m7.2.2.2.4" xref="S3.p10.7.m7.2.2.2.3.cmml"><mo stretchy="false" id="S3.p10.7.m7.2.2.2.4.1" xref="S3.p10.7.m7.2.2.2.3.cmml">{</mo><mi id="S3.p10.7.m7.1.1.1.1" xref="S3.p10.7.m7.1.1.1.1.cmml">i</mi><mo id="S3.p10.7.m7.2.2.2.4.2" xref="S3.p10.7.m7.2.2.2.3.cmml">,</mo><mi id="S3.p10.7.m7.2.2.2.2" xref="S3.p10.7.m7.2.2.2.2.cmml">j</mi><mo stretchy="false" id="S3.p10.7.m7.2.2.2.4.3" xref="S3.p10.7.m7.2.2.2.3.cmml">}</mo></mrow></msub><msup id="S3.p10.7.m7.3.3.1.1" xref="S3.p10.7.m7.3.3.1.1.cmml"><mrow id="S3.p10.7.m7.3.3.1.1.1.1" xref="S3.p10.7.m7.3.3.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.p10.7.m7.3.3.1.1.1.1.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.cmml">(</mo><mrow id="S3.p10.7.m7.3.3.1.1.1.1.1" xref="S3.p10.7.m7.3.3.1.1.1.1.1.cmml"><mrow id="S3.p10.7.m7.3.3.1.1.1.1.1.1" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.cmml"><mi id="S3.p10.7.m7.3.3.1.1.1.1.1.1.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.p10.7.m7.3.3.1.1.1.1.1.1.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.cmml"><msub id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.2.cmml">π</mi><mi id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.3.cmml">j</mi></msub><mo id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.1" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.1.cmml">&lt;</mo><msub id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.2.cmml">π</mi><mi id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.p10.7.m7.3.3.1.1.1.1.1.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.3.cmml">−</mo><mrow id="S3.p10.7.m7.3.3.1.1.1.1.1.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.cmml"><mi id="S3.p10.7.m7.3.3.1.1.1.1.1.2.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S3.p10.7.m7.3.3.1.1.1.1.1.2.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.2.cmml">​</mo><mrow id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.cmml">(</mo><mrow id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.cmml"><msub id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.cmml"><mi id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.2.cmml">π</mi><mi id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.3.cmml">j</mi></msub><mo id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.1" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.1.cmml">&lt;</mo><msub id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.cmml"><mi id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.2" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.2.cmml">π</mi><mi id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.p10.7.m7.3.3.1.1.1.1.3" xref="S3.p10.7.m7.3.3.1.1.1.1.1.cmml">)</mo></mrow><mn id="S3.p10.7.m7.3.3.1.1.3" xref="S3.p10.7.m7.3.3.1.1.3.cmml">2</mn></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p10.7.m7.3b"><apply id="S3.p10.7.m7.3.3.cmml" xref="S3.p10.7.m7.3.3"><times id="S3.p10.7.m7.3.3.2.cmml" xref="S3.p10.7.m7.3.3.2"></times><apply id="S3.p10.7.m7.3.3.3.cmml" xref="S3.p10.7.m7.3.3.3"><divide id="S3.p10.7.m7.3.3.3.1.cmml" xref="S3.p10.7.m7.3.3.3"></divide><cn type="integer" id="S3.p10.7.m7.3.3.3.2.cmml" xref="S3.p10.7.m7.3.3.3.2">1</cn><ci id="S3.p10.7.m7.3.3.3.3.cmml" xref="S3.p10.7.m7.3.3.3.3">𝑁</ci></apply><apply id="S3.p10.7.m7.3.3.1.cmml" xref="S3.p10.7.m7.3.3.1"><apply id="S3.p10.7.m7.3.3.1.2.cmml" xref="S3.p10.7.m7.3.3.1.2"><csymbol cd="ambiguous" id="S3.p10.7.m7.3.3.1.2.1.cmml" xref="S3.p10.7.m7.3.3.1.2">subscript</csymbol><sum id="S3.p10.7.m7.3.3.1.2.2.cmml" xref="S3.p10.7.m7.3.3.1.2.2"></sum><set id="S3.p10.7.m7.2.2.2.3.cmml" xref="S3.p10.7.m7.2.2.2.4"><ci id="S3.p10.7.m7.1.1.1.1.cmml" xref="S3.p10.7.m7.1.1.1.1">𝑖</ci><ci id="S3.p10.7.m7.2.2.2.2.cmml" xref="S3.p10.7.m7.2.2.2.2">𝑗</ci></set></apply><apply id="S3.p10.7.m7.3.3.1.1.cmml" xref="S3.p10.7.m7.3.3.1.1"><csymbol cd="ambiguous" id="S3.p10.7.m7.3.3.1.1.2.cmml" xref="S3.p10.7.m7.3.3.1.1">superscript</csymbol><apply id="S3.p10.7.m7.3.3.1.1.1.1.1.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1"><minus id="S3.p10.7.m7.3.3.1.1.1.1.1.3.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.3"></minus><apply id="S3.p10.7.m7.3.3.1.1.1.1.1.1.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1"><times id="S3.p10.7.m7.3.3.1.1.1.1.1.1.2.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.2"></times><ci id="S3.p10.7.m7.3.3.1.1.1.1.1.1.3.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.3">𝑓</ci><apply id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1"><lt id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.1"></lt><apply id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.2">𝜋</ci><ci id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.2.3">𝑗</ci></apply><apply id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.2">𝜋</ci><ci id="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply><apply id="S3.p10.7.m7.3.3.1.1.1.1.1.2.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2"><times id="S3.p10.7.m7.3.3.1.1.1.1.1.2.2.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.2"></times><ci id="S3.p10.7.m7.3.3.1.1.1.1.1.2.3.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.3">𝑂</ci><apply id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1"><lt id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.1.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.1"></lt><apply id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2"><csymbol cd="ambiguous" id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.1.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2">subscript</csymbol><ci id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.2.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.2">𝜋</ci><ci id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.3.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.2.3">𝑗</ci></apply><apply id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.1.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3">subscript</csymbol><ci id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.2.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.2">𝜋</ci><ci id="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.3.cmml" xref="S3.p10.7.m7.3.3.1.1.1.1.1.2.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply><cn type="integer" id="S3.p10.7.m7.3.3.1.1.3.cmml" xref="S3.p10.7.m7.3.3.1.1.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p10.7.m7.3c">\frac{1}{N}\sum_{\{i,j\}}(f(\pi_{j}&lt;\pi_{i})-O(\pi_{j}&lt;\pi_{i}))^{2}</annotation></semantics></math>, where <math id="S3.p10.8.m8.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.p10.8.m8.1a"><mi id="S3.p10.8.m8.1.1" xref="S3.p10.8.m8.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.p10.8.m8.1b"><ci id="S3.p10.8.m8.1.1.cmml" xref="S3.p10.8.m8.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p10.8.m8.1c">N</annotation></semantics></math> is the number of model pairs and <math id="S3.p10.9.m9.1" class="ltx_Math" alttext="{O}" display="inline"><semantics id="S3.p10.9.m9.1a"><mi id="S3.p10.9.m9.1.1" xref="S3.p10.9.m9.1.1.cmml">O</mi><annotation-xml encoding="MathML-Content" id="S3.p10.9.m9.1b"><ci id="S3.p10.9.m9.1.1.cmml" xref="S3.p10.9.m9.1.1">𝑂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p10.9.m9.1c">{O}</annotation></semantics></math> is the ground truth outcome.</p>
</div>
<div id="S3.p11" class="ltx_para ltx_noindent">
<p id="S3.p11.1" class="ltx_p">Separability with Confidence, Agreement with Confidence Interval, and Pair Rank Brier Score each provide useful perspectives for evaluating the effectiveness of LLM benchmarks. However, no single metric is intended to be individually sufficient. Ultimately, these metrics are complementary and should be considered together holistically. We claim that together, these metrics offer a robust framework for assessing benchmark performance, balancing the need for clear differentiation with alignment to human preferences.</p>
</div>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>The BenchBuilder Pipeline and Arena-Hard-Auto v0.1 Dataset</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>BenchBuilder: A pipeline for converting messy data into high-quality benchmarks</h3>

<figure id="S4.F1" class="ltx_figure"><img src="/html/2406.11939/assets/x1.png" id="S4.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="178" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span><span id="S4.F1.2.1" class="ltx_text ltx_font_bold">BenchBuilder Pipeline.</span> Starting with a live data source of crowdsourced user prompts, we first cluster their embeddings to form topic clusters. An LLM annotator then assigns quality scores based on the required skills. Clusters with low quality scores are filtered out, and we sample from the remaining high-quality clusters to create a diverse and challenging dataset of benchmark prompts.</figcaption>
</figure>
<div id="S4.SS1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.p1.1" class="ltx_p">BenchBuilder automatically extracts diverse, high quality prompts from a dataset of 200,000 user queries collected via Chatbot Arena. This is done by assigning a quality score to each prompt and by sampling prompts with a high quality score evenly across a wide range of topics. An outline of our data creation pipeline is shown in Figure <a href="#S4.F1" title="Figure 1 ‣ 4.1 BenchBuilder: A pipeline for converting messy data into high-quality benchmarks ‣ 4 The BenchBuilder Pipeline and Arena-Hard-Auto v0.1 Dataset ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para ltx_noindent">
<svg id="S4.SS1.p2.pic1" class="ltx_picture" height="175.28" overflow="visible" version="1.1" width="603.54"><g transform="translate(0,175.28) matrix(1 0 0 -1 0 0) translate(0,3.54)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g stroke-opacity="0.5" fill="#F2F2F2" fill-opacity="0.5" stroke="#F2F2F2" color="#F2F2F2"><path d="M 3.54 4.13 L 3.54 160.52 C 3.54 164.76 6.98 168.2 11.22 168.2 L 595.87 168.2 C 600.11 168.2 603.54 164.76 603.54 160.52 L 603.54 4.13 C 603.54 -0.11 600.11 -3.54 595.87 -3.54 L 11.22 -3.54 C 6.98 -3.54 3.54 -0.11 3.54 4.13 Z" style="stroke:none"></path></g><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 165.84 C 0 169.1 2.64 171.74 5.91 171.74 L 594.09 171.74 C 597.36 171.74 600 169.1 600 165.84 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 155.5 L 598.03 155.5 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 17.72 157.47)"><foreignObject width="564.57" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S4.SS1.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:408.0pt;">
<span id="S4.SS1.p2.pic1.1.1.1.1.1.1" class="ltx_p">Key Prompt Qualities</span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 17.72 13.78)"><foreignObject width="564.57" height="129.91" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S4.SS1.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:408.0pt;">
<span id="S4.I1" class="ltx_itemize">
<span id="S4.I1.i1" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I1.i1.p1" class="ltx_para">
<span id="S4.I1.i1.p1.1" class="ltx_p"><span id="S4.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Specificity:</span> Does the prompt ask for a specific, well-defined output without leaving any ambiguity?</span>
</span></span>
<span id="S4.I1.i2" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I1.i2.p1" class="ltx_para">
<span id="S4.I1.i2.p1.1" class="ltx_p"><span id="S4.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Domain Knowledge:</span> Does the prompt test the AI’s knowledge and understanding in a specific domain or set of domains?</span>
</span></span>
<span id="S4.I1.i3" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I1.i3.p1" class="ltx_para">
<span id="S4.I1.i3.p1.1" class="ltx_p"><span id="S4.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Complexity:</span> Does the prompt have multiple components, variables, or levels of depth and nuance?
</span>
</span></span>
<span id="S4.I1.i4" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I1.i4.p1" class="ltx_para">
<span id="S4.I1.i4.p1.1" class="ltx_p"><span id="S4.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">Problem-Solving:</span> Does the prompt require active problem-solving: analyzing and clearly defining the problem and systematically devising and implementing a solution?</span>
</span></span>
<span id="S4.I1.i5" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I1.i5.p1" class="ltx_para">
<span id="S4.I1.i5.p1.1" class="ltx_p"><span id="S4.I1.i5.p1.1.1" class="ltx_text ltx_font_bold">Creativity:</span> Does the prompt require a creative approach or solution?</span>
</span></span>
<span id="S4.I1.i6" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I1.i6.p1" class="ltx_para">
<span id="S4.I1.i6.p1.1" class="ltx_p"><span id="S4.I1.i6.p1.1.1" class="ltx_text ltx_font_bold">Technical Accuracy:</span> Does the prompt require an answer with a high degree of technical accuracy, correctness and precision?</span>
</span></span>
<span id="S4.I1.i7" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I1.i7.p1" class="ltx_para ltx_noindent">
<span id="S4.I1.i7.p1.1" class="ltx_p"><span id="S4.I1.i7.p1.1.1" class="ltx_text ltx_font_bold">Real-world Application:</span> Does the prompt relate to real-world applications?</span>
</span></span>
</span>
</span></foreignObject></g></g></svg>
</div>
<div id="S4.SS1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS1.p3.2" class="ltx_p">To discover high quality prompts, we define seven qualities of a prompt which signify the skills required to successfully answer, including specificity, domain knowledge, complexity, and more shown in the figure above.
These qualities are used by an LLM annotator to assign a “quality score” to a given prompt <math id="S4.SS1.p3.1.m1.1" class="ltx_Math" alttext="\mathrm{p}" display="inline"><semantics id="S4.SS1.p3.1.m1.1a"><mi mathvariant="normal" id="S4.SS1.p3.1.m1.1.1" xref="S4.SS1.p3.1.m1.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.1.m1.1b"><ci id="S4.SS1.p3.1.m1.1.1.cmml" xref="S4.SS1.p3.1.m1.1.1">p</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.1.m1.1c">\mathrm{p}</annotation></semantics></math>, which indicates how many of the above qualities are seen in <math id="S4.SS1.p3.2.m2.1" class="ltx_Math" alttext="\mathrm{p}" display="inline"><semantics id="S4.SS1.p3.2.m2.1a"><mi mathvariant="normal" id="S4.SS1.p3.2.m2.1.1" xref="S4.SS1.p3.2.m2.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.2.m2.1b"><ci id="S4.SS1.p3.2.m2.1.1.cmml" xref="S4.SS1.p3.2.m2.1.1">p</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.2.m2.1c">\mathrm{p}</annotation></semantics></math>. Full instruction prompt is in Section <a href="#S11" title="11 Prompts ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>.</p>
</div>
<div id="S4.SS1.p4" class="ltx_para ltx_noindent">
<p id="S4.SS1.p4.1" class="ltx_p">To ensure that the filtered prompts represent a wide range of tasks, we adopt a topic modeling pipeline in BERTopic by first converting each prompt with OpenAI’s embedding model (text-embedding-3-small <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>), reducing dimension with UMAP, and using a hierarchical-based clustering algorithm (HDBSCAN) to identify clusters which are then summarized using GPT-4-Turbo <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>]</cite>.</p>
</div>
<div id="S4.SS1.p5" class="ltx_para ltx_noindent">
<p id="S4.SS1.p5.1" class="ltx_p">As some topic clusters contain mostly trivial or ill defined prompt like “hi”, we select the topic clusters with mean quality score &gt;=6 out of 7 and sample prompts equally across each of these clusters. The resulting dataset consists of mostly well-defined, technical problem-solving queries as required in the above key criteria. Dataset statistics and further description of the filtering process used for Arena-Hard-Auto v0.1 along are in the following section.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Arena-Hard-Auto v0.1: A set of challenging, high-quality prompts from Chatbot Arena</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.1" class="ltx_p">We utilize the BenchBuilder pipeline to create Arena-Hard-Auto v0.1, a set of 500 high-quality prompts taken from 200,000 prompts from Chatbot Arena, an open source platform for LLM evaluation. We first remove any duplicate prompts, multi-turn conversations, or non-English conversations, and obtain over 4,000 topic clusters and 75,000 queries covering a wide range of domains.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.p2.1" class="ltx_p">Then, we use the GPT-3.5-Turbo as judge to assign a “quality score” to each prompt and remove prompts which either have a score &lt; 5 or which belong to a topic cluster with a mean score &lt; 3. We present the statistics in Table <a href="#S6.T1" title="Table 1 ‣ 6 Experimental Results ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. Notably, around 18% out of 75,000 queries are higher quality according to GPT-3.5-Turbo. We then use GPT-4-Turbo as judge to assign a quality score and ensuring the quality of the remaining prompts, resulting with over 500 high quality clusters. We randomly sample 500 prompts from these clusters, creating Arena-Hard-Auto v0.1. Additionally, we confirm our final dataset does not contain any personally identifiable information or offensive content.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS2.p3.1" class="ltx_p">In Figure <a href="#S4.F3" title="Figure 3 ‣ 4.2 Arena-Hard-Auto v0.1: A set of challenging, high-quality prompts from Chatbot Arena ‣ 4 The BenchBuilder Pipeline and Arena-Hard-Auto v0.1 Dataset ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we show examples of topic clusters ranging from low to high mean scores provided by GPT-3.5-Turbo. We can observe clusters with higher scores often correlate to challenging topics or tasks for LLMs like game development or mathematical proofs. On the other hand, clusters with lower scores point to trivial or ambiguous questions like "Flirty Texting Strategies". We present several example prompts and their respective topic cluster below.</p>
</div>
<figure id="S4.SS2.1" class="ltx_float ltx_float_tcbfloat">
<p id="S4.SS2.1.1" class="ltx_p">[htb] <svg id="S4.SS2.1.1.pic1" class="ltx_picture" height="4391.45" overflow="visible" version="1.1" width="603.54"><g transform="translate(0,4391.45) matrix(1 0 0 -1 0 0) translate(0,3.54)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g stroke-opacity="0.5" fill="#F2F2F2" fill-opacity="0.5" stroke="#F2F2F2" color="#F2F2F2"><path d="M 3.54 -3.54 L 3.54 4384.36 L 603.54 4384.36 L 603.54 -3.54 Z" style="stroke:none"></path></g><g fill="#FFFF99" fill-opacity="1.0"><path d="M 0 0 L 0 4387.91 L 600 4387.91 L 600 0 Z" style="stroke:none"></path></g><g fill="#FFFFE6" fill-opacity="1.0"><path d="M 1.97 1.97 L 1.97 4385.94 L 598.03 4385.94 L 598.03 1.97 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.84 13.78)"><foreignObject width="580.32" height="4360.35" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S4.SS2.1.1.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:419.4pt;">
<span id="S4.SS2.1.1.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S4.SS2.1.1.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Cluster 1: Greetings and Well-Being Inquiry</span> (Mean Score: 2.7)</span>
<span id="S4.SS2.1.1.pic1.1.1.1.1.1.2" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,WW8sIHdoYXQgdXAgbXkgYnJvdGhlciAoUXVhbGl0aWVzOiBOb25lKQ==" download="">⬇</a></span>
<span id="lstnumberx1" class="ltx_listingline"><span id="lstnumberx1.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Yo</span><span id="lstnumberx1.2" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx1.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">what</span><span id="lstnumberx1.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">up</span><span id="lstnumberx1.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">my</span><span id="lstnumberx1.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">brother</span><span id="lstnumberx1.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.12" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx1.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Qualities</span><span id="lstnumberx1.14" class="ltx_text ltx_font_typewriter">:</span><span id="lstnumberx1.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span><span id="lstnumberx1.17" class="ltx_text ltx_font_typewriter">)</span>
</span>
</span>
<span id="S4.SS2.1.1.pic1.1.1.1.1.1.3" class="ltx_p"><span id="S4.SS2.1.1.pic1.1.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Cluster 2: US Presidents Query</span> (Mean Score: 3.2)</span>
<span id="S4.SS2.1.1.pic1.1.1.1.1.1.4" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,V2hvIHdhcyB0aGUgcHJlc2lkZW50IG9mIHRoZSBVUyBpbiAxOTc1IChRdWFsaXRpZXM6IFNwZWNpZmljaXR5LCBEb21haW4tS25vd2xlZGdlLCBUZWNobmljYWwgQWNjdXJhY3ksIFJlYWwtV29ybGQp" download="">⬇</a></span>
<span id="lstnumberx2" class="ltx_listingline"><span id="lstnumberx2.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Who</span><span id="lstnumberx2.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx2.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx2.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">president</span><span id="lstnumberx2.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx2.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx2.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">US</span><span id="lstnumberx2.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx2.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.17" class="ltx_text ltx_font_typewriter">1975</span><span id="lstnumberx2.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.19" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx2.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Qualities</span><span id="lstnumberx2.21" class="ltx_text ltx_font_typewriter">:</span><span id="lstnumberx2.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Specificity</span><span id="lstnumberx2.24" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx2.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Domain</span><span id="lstnumberx2.27" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx2.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Knowledge</span><span id="lstnumberx2.29" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx2.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Technical</span><span id="lstnumberx2.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Accuracy</span><span id="lstnumberx2.34" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx2.35" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.36" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Real</span><span id="lstnumberx2.37" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx2.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter">World</span><span id="lstnumberx2.39" class="ltx_text ltx_font_typewriter">)</span>
</span>
</span>
<span id="S4.SS2.1.1.pic1.1.1.1.1.1.5" class="ltx_p"><span id="S4.SS2.1.1.pic1.1.1.1.1.1.5.1" class="ltx_text ltx_font_bold">Cluster 3: Physics Problem Solving</span> (Mean Score: 5.0)</span>
<span id="S4.SS2.1.1.pic1.1.1.1.1.1.6" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,QSA1MCwwMDAga2cgYWlycGxhbmUgaW5pdGlhbGx5IGZseWluZyBhdCBhIHNwZWVkIG9mIDYwLjAgbS9zIGFjY2VsZXJhdGVzIGF0IDUuMCBtL3MyIGZvciA2MDAgbWV0ZXJzLiBXaGF0IGlzIGl0cyB2ZWxvY2l0eSBhZnRlciB0aGlzIGFjY2VsZXJhdGlvbj8gV2hhdCBpcyB0aGUgbmV0IGZvcmNlIHRoYXQgY2F1c2VkIHRoaXMgYWNjZWxlcmF0aW9uPyAoUXVhbGl0aWVzOiBTcGVjaWZpY2l0eSwgRG9tYWluLUtub3dsZWRnZSwgQ29tcGxleGl0eSwgUHJvYmxlbS1Tb2x2aW5nLCBUZWNobmljYWwgQWNjdXJhY3ksIFJlYWwtV29ybGQp" download="">⬇</a></span>
<span id="lstnumberx3" class="ltx_listingline"><span id="lstnumberx3.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">A</span><span id="lstnumberx3.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.3" class="ltx_text ltx_font_typewriter">50,000</span><span id="lstnumberx3.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">kg</span><span id="lstnumberx3.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">airplane</span><span id="lstnumberx3.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">initially</span><span id="lstnumberx3.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">flying</span><span id="lstnumberx3.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">at</span><span id="lstnumberx3.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx3.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">speed</span><span id="lstnumberx3.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx3.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.21" class="ltx_text ltx_font_typewriter">60.0</span><span id="lstnumberx3.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">m</span><span id="lstnumberx3.24" class="ltx_text ltx_font_typewriter">/</span><span id="lstnumberx3.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx3.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">accelerates</span><span id="lstnumberx3.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">at</span><span id="lstnumberx3.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.31" class="ltx_text ltx_font_typewriter">5.0</span><span id="lstnumberx3.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">m</span><span id="lstnumberx3.34" class="ltx_text ltx_font_typewriter">/</span><span id="lstnumberx3.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s2</span><span id="lstnumberx3.36" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.37" class="ltx_text ltx_lst_identifier ltx_font_typewriter">for</span><span id="lstnumberx3.38" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.39" class="ltx_text ltx_font_typewriter">600</span><span id="lstnumberx3.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.41" class="ltx_text ltx_lst_identifier ltx_font_typewriter">meters</span><span id="lstnumberx3.42" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx3.43" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter">What</span><span id="lstnumberx3.45" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx3.47" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter">its</span><span id="lstnumberx3.49" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter">velocity</span><span id="lstnumberx3.51" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter">after</span><span id="lstnumberx3.53" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter">this</span><span id="lstnumberx3.55" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter">acceleration</span><span id="lstnumberx3.57" class="ltx_text ltx_font_typewriter">?</span><span id="lstnumberx3.58" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.59" class="ltx_text ltx_lst_identifier ltx_font_typewriter">What</span><span id="lstnumberx3.60" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.61" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx3.62" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.63" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx3.64" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter">net</span><span id="lstnumberx3.66" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.67" class="ltx_text ltx_lst_identifier ltx_font_typewriter">force</span><span id="lstnumberx3.68" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.69" class="ltx_text ltx_lst_identifier ltx_font_typewriter">that</span><span id="lstnumberx3.70" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.71" class="ltx_text ltx_lst_identifier ltx_font_typewriter">caused</span><span id="lstnumberx3.72" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.73" class="ltx_text ltx_lst_identifier ltx_font_typewriter">this</span><span id="lstnumberx3.74" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.75" class="ltx_text ltx_lst_identifier ltx_font_typewriter">acceleration</span><span id="lstnumberx3.76" class="ltx_text ltx_font_typewriter">?</span><span id="lstnumberx3.77" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.78" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx3.79" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Qualities</span><span id="lstnumberx3.80" class="ltx_text ltx_font_typewriter">:</span><span id="lstnumberx3.81" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.82" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Specificity</span><span id="lstnumberx3.83" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx3.84" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.85" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Domain</span><span id="lstnumberx3.86" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx3.87" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Knowledge</span><span id="lstnumberx3.88" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx3.89" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.90" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Complexity</span><span id="lstnumberx3.91" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx3.92" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.93" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Problem</span><span id="lstnumberx3.94" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx3.95" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Solving</span><span id="lstnumberx3.96" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx3.97" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.98" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Technical</span><span id="lstnumberx3.99" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.100" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Accuracy</span><span id="lstnumberx3.101" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx3.102" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.103" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Real</span><span id="lstnumberx3.104" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx3.105" class="ltx_text ltx_lst_identifier ltx_font_typewriter">World</span><span id="lstnumberx3.106" class="ltx_text ltx_font_typewriter">)</span>
</span>
</span>
<span id="S4.SS2.1.1.pic1.1.1.1.1.1.7" class="ltx_p"><span id="S4.SS2.1.1.pic1.1.1.1.1.1.7.1" class="ltx_text ltx_font_bold">Cluster 4: OpenCV Image Processing Technique</span> (Mean Score: 5.5)</span>
<span id="S4.SS2.1.1.pic1.1.1.1.1.1.8" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,eW91IGFyZSBnaXZlbiBhIHRhc2sgdG8gZGV0ZWN0IG51bWJlciBvZiBmYWNlcyBpbiBlYWNoIGZyYW1lIG9mIGFueSB2aWRlbyB1c2luZyBweXRvcmNoIGFuZCBkaXNwbGF5IHRoZSBudW1iZXIgaW4gdGhlIGZpbmFsIGVkaXRlZCB2aWRlby4gKFF1YWxpdGllczogQWxsKQ==" download="">⬇</a></span>
<span id="lstnumberx4" class="ltx_listingline"><span id="lstnumberx4.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">you</span><span id="lstnumberx4.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx4.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx4.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx4.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">task</span><span id="lstnumberx4.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx4.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">detect</span><span id="lstnumberx4.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx4.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx4.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">faces</span><span id="lstnumberx4.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx4.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">each</span><span id="lstnumberx4.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">frame</span><span id="lstnumberx4.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx4.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">any</span><span id="lstnumberx4.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">video</span><span id="lstnumberx4.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">using</span><span id="lstnumberx4.34" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter">pytorch</span><span id="lstnumberx4.36" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.37" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx4.38" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.39" class="ltx_text ltx_lst_identifier ltx_font_typewriter">display</span><span id="lstnumberx4.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.41" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx4.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx4.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx4.46" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.47" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx4.48" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.49" class="ltx_text ltx_lst_identifier ltx_font_typewriter">final</span><span id="lstnumberx4.50" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.51" class="ltx_text ltx_lst_identifier ltx_font_typewriter">edited</span><span id="lstnumberx4.52" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.53" class="ltx_text ltx_lst_identifier ltx_font_typewriter">video</span><span id="lstnumberx4.54" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx4.55" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.56" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx4.57" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Qualities</span><span id="lstnumberx4.58" class="ltx_text ltx_font_typewriter">:</span><span id="lstnumberx4.59" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter">All</span><span id="lstnumberx4.61" class="ltx_text ltx_font_typewriter">)</span>
</span>
</span>
</span></foreignObject></g></g></svg></p>
</figure>
<div id="S4.SS2.p4" class="ltx_para ltx_noindent">
<p id="S4.SS2.p4.1" class="ltx_p">To see whether “quality score” assigned during BenchBuilder’s pipeline correlates with separability and agreement, we sample 50 prompts per score and compare the responses from GPT-4 and Llama-2-70b-Chat <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite>, with GPT-4-Turbo as judge. In Figure <a href="#S4.F4" title="Figure 4 ‣ 4.2 Arena-Hard-Auto v0.1: A set of challenging, high-quality prompts from Chatbot Arena ‣ 4 The BenchBuilder Pipeline and Arena-Hard-Auto v0.1 Dataset ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (Left), we observe a strong correlation between high potential score and the win-rate of GPT-4-Turbo over Llama-2-70b-Chat. A similar trend is also observed in other model pairs such as Claude Sonnet vs Haiku and Mistral-Large <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite> vs Mixtral <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>.</p>
</div>
<div id="S4.SS2.p5" class="ltx_para ltx_noindent">
<p id="S4.SS2.p5.1" class="ltx_p">Since qualities assigned by GPT-4-Turbo are critical to ensure the final prompt selection is high-quality, we verified its accuracy as annotator. We construct “ground truth” labels for 200 randomly sampled queries by collecting majority votes from GPT-4o <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>, Claude-3-Opus <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>, and Gemini-1.5-Pro <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib39" title="" class="ltx_ref">39</a>]</cite>. GPT-4-Turbo has 85.6% accuracy against “ground truth” labels, validating its ability as annotator.</p>
</div>
<figure id="S4.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F3.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_bottom" style="width:208.1pt;"><img src="/html/2406.11939/assets/x2.png" id="S4.F3.1.g1" class="ltx_graphics ltx_img_square" width="461" height="461" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Mean score of various topic clusters in descending order. Higher-scoring clusters correlate to challenging topics, while lower scores indicate trivial or ambiguous questions. A more complete topic cluster plot is in <a href="#S10.F6" title="Figure 6 ‣ 10 Appendix ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 6</span></a>.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F3.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_bottom" style="width:208.1pt;"><img src="/html/2406.11939/assets/x3.png" id="S4.F3.2.g1" class="ltx_graphics ltx_img_square" width="461" height="461" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Comparison between MT-bench and Arena-Hard-Auto v0.1. The latter offers significantly better separability between models and tighter confidence intervals. GPT-4-0314 has no variance in Arena-Hard-Auto v0.1 because it is the anchor model.</figcaption>
</figure>
</div>
</div>
</figure>
<figure id="S4.F4" class="ltx_figure"><img src="" id="S4.F4.g1" class="ltx_graphics ltx_centering ltx_missing ltx_missing_image" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Win-rate of three model pairs
(GPT-4-0613 vs Llama-2-70b-chat, Claude-3-Sonnet-20240229 vs Claude-3-Haiku-20240307, and Mistral-Large vs Mixtral-8x7b-Instruct-v0.1)
over “quality score”. We randomly sample 50 queries for each quality score 0-7 and bootstrap a win-rate and confidence interval between model pairs on each score interval of 2. We observe a similar trend of win-rate between model pairs becomes increasingly separable as the quality score increases.</figcaption>
</figure>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Evaluation with LLM-as-a-Judge</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">Evaluating models on challenging queries such as Arena-Hard-Auto v0.1 is a non-trivial task. Most queries involve deep domain knowledge and problem solving skills, requiring expert-level judgment to evaluate the answer quality. Unfortunately, it is prohibitively expensive and time consuming to use human judges. Following LLM-as-a-Judge <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite> and AlpacaFarm <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>, we employ LLM-as-a-judge framework to approximate human preference.</p>
</div>
<div id="S5.p2" class="ltx_para ltx_noindent">
<p id="S5.p2.12" class="ltx_p">To obtain a score for model <math id="S5.p2.1.m1.1" class="ltx_Math" alttext="\pi_{i}" display="inline"><semantics id="S5.p2.1.m1.1a"><msub id="S5.p2.1.m1.1.1" xref="S5.p2.1.m1.1.1.cmml"><mi id="S5.p2.1.m1.1.1.2" xref="S5.p2.1.m1.1.1.2.cmml">π</mi><mi id="S5.p2.1.m1.1.1.3" xref="S5.p2.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S5.p2.1.m1.1b"><apply id="S5.p2.1.m1.1.1.cmml" xref="S5.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S5.p2.1.m1.1.1.1.cmml" xref="S5.p2.1.m1.1.1">subscript</csymbol><ci id="S5.p2.1.m1.1.1.2.cmml" xref="S5.p2.1.m1.1.1.2">𝜋</ci><ci id="S5.p2.1.m1.1.1.3.cmml" xref="S5.p2.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.1.m1.1c">\pi_{i}</annotation></semantics></math> on prompt <math id="S5.p2.2.m2.1" class="ltx_Math" alttext="\mathrm{p}" display="inline"><semantics id="S5.p2.2.m2.1a"><mi mathvariant="normal" id="S5.p2.2.m2.1.1" xref="S5.p2.2.m2.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S5.p2.2.m2.1b"><ci id="S5.p2.2.m2.1.1.cmml" xref="S5.p2.2.m2.1.1">p</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.2.m2.1c">\mathrm{p}</annotation></semantics></math>, we consider the pairwise comparison setup against a strong baseline model <math id="S5.p2.3.m3.1" class="ltx_Math" alttext="\pi_{base}" display="inline"><semantics id="S5.p2.3.m3.1a"><msub id="S5.p2.3.m3.1.1" xref="S5.p2.3.m3.1.1.cmml"><mi id="S5.p2.3.m3.1.1.2" xref="S5.p2.3.m3.1.1.2.cmml">π</mi><mrow id="S5.p2.3.m3.1.1.3" xref="S5.p2.3.m3.1.1.3.cmml"><mi id="S5.p2.3.m3.1.1.3.2" xref="S5.p2.3.m3.1.1.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="S5.p2.3.m3.1.1.3.1" xref="S5.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S5.p2.3.m3.1.1.3.3" xref="S5.p2.3.m3.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S5.p2.3.m3.1.1.3.1a" xref="S5.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S5.p2.3.m3.1.1.3.4" xref="S5.p2.3.m3.1.1.3.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="S5.p2.3.m3.1.1.3.1b" xref="S5.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S5.p2.3.m3.1.1.3.5" xref="S5.p2.3.m3.1.1.3.5.cmml">e</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.p2.3.m3.1b"><apply id="S5.p2.3.m3.1.1.cmml" xref="S5.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S5.p2.3.m3.1.1.1.cmml" xref="S5.p2.3.m3.1.1">subscript</csymbol><ci id="S5.p2.3.m3.1.1.2.cmml" xref="S5.p2.3.m3.1.1.2">𝜋</ci><apply id="S5.p2.3.m3.1.1.3.cmml" xref="S5.p2.3.m3.1.1.3"><times id="S5.p2.3.m3.1.1.3.1.cmml" xref="S5.p2.3.m3.1.1.3.1"></times><ci id="S5.p2.3.m3.1.1.3.2.cmml" xref="S5.p2.3.m3.1.1.3.2">𝑏</ci><ci id="S5.p2.3.m3.1.1.3.3.cmml" xref="S5.p2.3.m3.1.1.3.3">𝑎</ci><ci id="S5.p2.3.m3.1.1.3.4.cmml" xref="S5.p2.3.m3.1.1.3.4">𝑠</ci><ci id="S5.p2.3.m3.1.1.3.5.cmml" xref="S5.p2.3.m3.1.1.3.5">𝑒</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.3.m3.1c">\pi_{base}</annotation></semantics></math> (GPT-4-0314) and employ a strong judge model (e.g., GPT-4-Turbo or Gemini-1.5-Pro) to measure preference. We obtain a score for each output <math id="S5.p2.4.m4.1" class="ltx_Math" alttext="\pi_{i}(\mathrm{p})" display="inline"><semantics id="S5.p2.4.m4.1a"><mrow id="S5.p2.4.m4.1.2" xref="S5.p2.4.m4.1.2.cmml"><msub id="S5.p2.4.m4.1.2.2" xref="S5.p2.4.m4.1.2.2.cmml"><mi id="S5.p2.4.m4.1.2.2.2" xref="S5.p2.4.m4.1.2.2.2.cmml">π</mi><mi id="S5.p2.4.m4.1.2.2.3" xref="S5.p2.4.m4.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S5.p2.4.m4.1.2.1" xref="S5.p2.4.m4.1.2.1.cmml">​</mo><mrow id="S5.p2.4.m4.1.2.3.2" xref="S5.p2.4.m4.1.2.cmml"><mo stretchy="false" id="S5.p2.4.m4.1.2.3.2.1" xref="S5.p2.4.m4.1.2.cmml">(</mo><mi mathvariant="normal" id="S5.p2.4.m4.1.1" xref="S5.p2.4.m4.1.1.cmml">p</mi><mo stretchy="false" id="S5.p2.4.m4.1.2.3.2.2" xref="S5.p2.4.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.p2.4.m4.1b"><apply id="S5.p2.4.m4.1.2.cmml" xref="S5.p2.4.m4.1.2"><times id="S5.p2.4.m4.1.2.1.cmml" xref="S5.p2.4.m4.1.2.1"></times><apply id="S5.p2.4.m4.1.2.2.cmml" xref="S5.p2.4.m4.1.2.2"><csymbol cd="ambiguous" id="S5.p2.4.m4.1.2.2.1.cmml" xref="S5.p2.4.m4.1.2.2">subscript</csymbol><ci id="S5.p2.4.m4.1.2.2.2.cmml" xref="S5.p2.4.m4.1.2.2.2">𝜋</ci><ci id="S5.p2.4.m4.1.2.2.3.cmml" xref="S5.p2.4.m4.1.2.2.3">𝑖</ci></apply><ci id="S5.p2.4.m4.1.1.cmml" xref="S5.p2.4.m4.1.1">p</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.4.m4.1c">\pi_{i}(\mathrm{p})</annotation></semantics></math> by asking the judge to rank its preference of <math id="S5.p2.5.m5.1" class="ltx_Math" alttext="\pi_{i}(\mathrm{p})" display="inline"><semantics id="S5.p2.5.m5.1a"><mrow id="S5.p2.5.m5.1.2" xref="S5.p2.5.m5.1.2.cmml"><msub id="S5.p2.5.m5.1.2.2" xref="S5.p2.5.m5.1.2.2.cmml"><mi id="S5.p2.5.m5.1.2.2.2" xref="S5.p2.5.m5.1.2.2.2.cmml">π</mi><mi id="S5.p2.5.m5.1.2.2.3" xref="S5.p2.5.m5.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S5.p2.5.m5.1.2.1" xref="S5.p2.5.m5.1.2.1.cmml">​</mo><mrow id="S5.p2.5.m5.1.2.3.2" xref="S5.p2.5.m5.1.2.cmml"><mo stretchy="false" id="S5.p2.5.m5.1.2.3.2.1" xref="S5.p2.5.m5.1.2.cmml">(</mo><mi mathvariant="normal" id="S5.p2.5.m5.1.1" xref="S5.p2.5.m5.1.1.cmml">p</mi><mo stretchy="false" id="S5.p2.5.m5.1.2.3.2.2" xref="S5.p2.5.m5.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.p2.5.m5.1b"><apply id="S5.p2.5.m5.1.2.cmml" xref="S5.p2.5.m5.1.2"><times id="S5.p2.5.m5.1.2.1.cmml" xref="S5.p2.5.m5.1.2.1"></times><apply id="S5.p2.5.m5.1.2.2.cmml" xref="S5.p2.5.m5.1.2.2"><csymbol cd="ambiguous" id="S5.p2.5.m5.1.2.2.1.cmml" xref="S5.p2.5.m5.1.2.2">subscript</csymbol><ci id="S5.p2.5.m5.1.2.2.2.cmml" xref="S5.p2.5.m5.1.2.2.2">𝜋</ci><ci id="S5.p2.5.m5.1.2.2.3.cmml" xref="S5.p2.5.m5.1.2.2.3">𝑖</ci></apply><ci id="S5.p2.5.m5.1.1.cmml" xref="S5.p2.5.m5.1.1">p</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.5.m5.1c">\pi_{i}(\mathrm{p})</annotation></semantics></math> or <math id="S5.p2.6.m6.1" class="ltx_Math" alttext="\pi_{base}(\mathrm{p})" display="inline"><semantics id="S5.p2.6.m6.1a"><mrow id="S5.p2.6.m6.1.2" xref="S5.p2.6.m6.1.2.cmml"><msub id="S5.p2.6.m6.1.2.2" xref="S5.p2.6.m6.1.2.2.cmml"><mi id="S5.p2.6.m6.1.2.2.2" xref="S5.p2.6.m6.1.2.2.2.cmml">π</mi><mrow id="S5.p2.6.m6.1.2.2.3" xref="S5.p2.6.m6.1.2.2.3.cmml"><mi id="S5.p2.6.m6.1.2.2.3.2" xref="S5.p2.6.m6.1.2.2.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="S5.p2.6.m6.1.2.2.3.1" xref="S5.p2.6.m6.1.2.2.3.1.cmml">​</mo><mi id="S5.p2.6.m6.1.2.2.3.3" xref="S5.p2.6.m6.1.2.2.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S5.p2.6.m6.1.2.2.3.1a" xref="S5.p2.6.m6.1.2.2.3.1.cmml">​</mo><mi id="S5.p2.6.m6.1.2.2.3.4" xref="S5.p2.6.m6.1.2.2.3.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="S5.p2.6.m6.1.2.2.3.1b" xref="S5.p2.6.m6.1.2.2.3.1.cmml">​</mo><mi id="S5.p2.6.m6.1.2.2.3.5" xref="S5.p2.6.m6.1.2.2.3.5.cmml">e</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S5.p2.6.m6.1.2.1" xref="S5.p2.6.m6.1.2.1.cmml">​</mo><mrow id="S5.p2.6.m6.1.2.3.2" xref="S5.p2.6.m6.1.2.cmml"><mo stretchy="false" id="S5.p2.6.m6.1.2.3.2.1" xref="S5.p2.6.m6.1.2.cmml">(</mo><mi mathvariant="normal" id="S5.p2.6.m6.1.1" xref="S5.p2.6.m6.1.1.cmml">p</mi><mo stretchy="false" id="S5.p2.6.m6.1.2.3.2.2" xref="S5.p2.6.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.p2.6.m6.1b"><apply id="S5.p2.6.m6.1.2.cmml" xref="S5.p2.6.m6.1.2"><times id="S5.p2.6.m6.1.2.1.cmml" xref="S5.p2.6.m6.1.2.1"></times><apply id="S5.p2.6.m6.1.2.2.cmml" xref="S5.p2.6.m6.1.2.2"><csymbol cd="ambiguous" id="S5.p2.6.m6.1.2.2.1.cmml" xref="S5.p2.6.m6.1.2.2">subscript</csymbol><ci id="S5.p2.6.m6.1.2.2.2.cmml" xref="S5.p2.6.m6.1.2.2.2">𝜋</ci><apply id="S5.p2.6.m6.1.2.2.3.cmml" xref="S5.p2.6.m6.1.2.2.3"><times id="S5.p2.6.m6.1.2.2.3.1.cmml" xref="S5.p2.6.m6.1.2.2.3.1"></times><ci id="S5.p2.6.m6.1.2.2.3.2.cmml" xref="S5.p2.6.m6.1.2.2.3.2">𝑏</ci><ci id="S5.p2.6.m6.1.2.2.3.3.cmml" xref="S5.p2.6.m6.1.2.2.3.3">𝑎</ci><ci id="S5.p2.6.m6.1.2.2.3.4.cmml" xref="S5.p2.6.m6.1.2.2.3.4">𝑠</ci><ci id="S5.p2.6.m6.1.2.2.3.5.cmml" xref="S5.p2.6.m6.1.2.2.3.5">𝑒</ci></apply></apply><ci id="S5.p2.6.m6.1.1.cmml" xref="S5.p2.6.m6.1.1">p</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.6.m6.1c">\pi_{base}(\mathrm{p})</annotation></semantics></math> on a Likert scale <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref">27</a>]</cite> (<math id="S5.p2.7.m7.1" class="ltx_Math" alttext="1=" display="inline"><semantics id="S5.p2.7.m7.1a"><mrow id="S5.p2.7.m7.1.1" xref="S5.p2.7.m7.1.1.cmml"><mn id="S5.p2.7.m7.1.1.2" xref="S5.p2.7.m7.1.1.2.cmml">1</mn><mo id="S5.p2.7.m7.1.1.1" xref="S5.p2.7.m7.1.1.1.cmml">=</mo><mi id="S5.p2.7.m7.1.1.3" xref="S5.p2.7.m7.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="S5.p2.7.m7.1b"><apply id="S5.p2.7.m7.1.1.cmml" xref="S5.p2.7.m7.1.1"><eq id="S5.p2.7.m7.1.1.1.cmml" xref="S5.p2.7.m7.1.1.1"></eq><cn type="integer" id="S5.p2.7.m7.1.1.2.cmml" xref="S5.p2.7.m7.1.1.2">1</cn><csymbol cd="latexml" id="S5.p2.7.m7.1.1.3.cmml" xref="S5.p2.7.m7.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.7.m7.1c">1=</annotation></semantics></math> prefers <math id="S5.p2.8.m8.1" class="ltx_Math" alttext="\pi_{i}(\mathrm{p})" display="inline"><semantics id="S5.p2.8.m8.1a"><mrow id="S5.p2.8.m8.1.2" xref="S5.p2.8.m8.1.2.cmml"><msub id="S5.p2.8.m8.1.2.2" xref="S5.p2.8.m8.1.2.2.cmml"><mi id="S5.p2.8.m8.1.2.2.2" xref="S5.p2.8.m8.1.2.2.2.cmml">π</mi><mi id="S5.p2.8.m8.1.2.2.3" xref="S5.p2.8.m8.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S5.p2.8.m8.1.2.1" xref="S5.p2.8.m8.1.2.1.cmml">​</mo><mrow id="S5.p2.8.m8.1.2.3.2" xref="S5.p2.8.m8.1.2.cmml"><mo stretchy="false" id="S5.p2.8.m8.1.2.3.2.1" xref="S5.p2.8.m8.1.2.cmml">(</mo><mi mathvariant="normal" id="S5.p2.8.m8.1.1" xref="S5.p2.8.m8.1.1.cmml">p</mi><mo stretchy="false" id="S5.p2.8.m8.1.2.3.2.2" xref="S5.p2.8.m8.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.p2.8.m8.1b"><apply id="S5.p2.8.m8.1.2.cmml" xref="S5.p2.8.m8.1.2"><times id="S5.p2.8.m8.1.2.1.cmml" xref="S5.p2.8.m8.1.2.1"></times><apply id="S5.p2.8.m8.1.2.2.cmml" xref="S5.p2.8.m8.1.2.2"><csymbol cd="ambiguous" id="S5.p2.8.m8.1.2.2.1.cmml" xref="S5.p2.8.m8.1.2.2">subscript</csymbol><ci id="S5.p2.8.m8.1.2.2.2.cmml" xref="S5.p2.8.m8.1.2.2.2">𝜋</ci><ci id="S5.p2.8.m8.1.2.2.3.cmml" xref="S5.p2.8.m8.1.2.2.3">𝑖</ci></apply><ci id="S5.p2.8.m8.1.1.cmml" xref="S5.p2.8.m8.1.1">p</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.8.m8.1c">\pi_{i}(\mathrm{p})</annotation></semantics></math> much less than <math id="S5.p2.9.m9.1" class="ltx_Math" alttext="\pi_{base}(\mathrm{p})" display="inline"><semantics id="S5.p2.9.m9.1a"><mrow id="S5.p2.9.m9.1.2" xref="S5.p2.9.m9.1.2.cmml"><msub id="S5.p2.9.m9.1.2.2" xref="S5.p2.9.m9.1.2.2.cmml"><mi id="S5.p2.9.m9.1.2.2.2" xref="S5.p2.9.m9.1.2.2.2.cmml">π</mi><mrow id="S5.p2.9.m9.1.2.2.3" xref="S5.p2.9.m9.1.2.2.3.cmml"><mi id="S5.p2.9.m9.1.2.2.3.2" xref="S5.p2.9.m9.1.2.2.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="S5.p2.9.m9.1.2.2.3.1" xref="S5.p2.9.m9.1.2.2.3.1.cmml">​</mo><mi id="S5.p2.9.m9.1.2.2.3.3" xref="S5.p2.9.m9.1.2.2.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S5.p2.9.m9.1.2.2.3.1a" xref="S5.p2.9.m9.1.2.2.3.1.cmml">​</mo><mi id="S5.p2.9.m9.1.2.2.3.4" xref="S5.p2.9.m9.1.2.2.3.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="S5.p2.9.m9.1.2.2.3.1b" xref="S5.p2.9.m9.1.2.2.3.1.cmml">​</mo><mi id="S5.p2.9.m9.1.2.2.3.5" xref="S5.p2.9.m9.1.2.2.3.5.cmml">e</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S5.p2.9.m9.1.2.1" xref="S5.p2.9.m9.1.2.1.cmml">​</mo><mrow id="S5.p2.9.m9.1.2.3.2" xref="S5.p2.9.m9.1.2.cmml"><mo stretchy="false" id="S5.p2.9.m9.1.2.3.2.1" xref="S5.p2.9.m9.1.2.cmml">(</mo><mi mathvariant="normal" id="S5.p2.9.m9.1.1" xref="S5.p2.9.m9.1.1.cmml">p</mi><mo stretchy="false" id="S5.p2.9.m9.1.2.3.2.2" xref="S5.p2.9.m9.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.p2.9.m9.1b"><apply id="S5.p2.9.m9.1.2.cmml" xref="S5.p2.9.m9.1.2"><times id="S5.p2.9.m9.1.2.1.cmml" xref="S5.p2.9.m9.1.2.1"></times><apply id="S5.p2.9.m9.1.2.2.cmml" xref="S5.p2.9.m9.1.2.2"><csymbol cd="ambiguous" id="S5.p2.9.m9.1.2.2.1.cmml" xref="S5.p2.9.m9.1.2.2">subscript</csymbol><ci id="S5.p2.9.m9.1.2.2.2.cmml" xref="S5.p2.9.m9.1.2.2.2">𝜋</ci><apply id="S5.p2.9.m9.1.2.2.3.cmml" xref="S5.p2.9.m9.1.2.2.3"><times id="S5.p2.9.m9.1.2.2.3.1.cmml" xref="S5.p2.9.m9.1.2.2.3.1"></times><ci id="S5.p2.9.m9.1.2.2.3.2.cmml" xref="S5.p2.9.m9.1.2.2.3.2">𝑏</ci><ci id="S5.p2.9.m9.1.2.2.3.3.cmml" xref="S5.p2.9.m9.1.2.2.3.3">𝑎</ci><ci id="S5.p2.9.m9.1.2.2.3.4.cmml" xref="S5.p2.9.m9.1.2.2.3.4">𝑠</ci><ci id="S5.p2.9.m9.1.2.2.3.5.cmml" xref="S5.p2.9.m9.1.2.2.3.5">𝑒</ci></apply></apply><ci id="S5.p2.9.m9.1.1.cmml" xref="S5.p2.9.m9.1.1">p</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.9.m9.1c">\pi_{base}(\mathrm{p})</annotation></semantics></math>, <math id="S5.p2.10.m10.1" class="ltx_Math" alttext="5=" display="inline"><semantics id="S5.p2.10.m10.1a"><mrow id="S5.p2.10.m10.1.1" xref="S5.p2.10.m10.1.1.cmml"><mn id="S5.p2.10.m10.1.1.2" xref="S5.p2.10.m10.1.1.2.cmml">5</mn><mo id="S5.p2.10.m10.1.1.1" xref="S5.p2.10.m10.1.1.1.cmml">=</mo><mi id="S5.p2.10.m10.1.1.3" xref="S5.p2.10.m10.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="S5.p2.10.m10.1b"><apply id="S5.p2.10.m10.1.1.cmml" xref="S5.p2.10.m10.1.1"><eq id="S5.p2.10.m10.1.1.1.cmml" xref="S5.p2.10.m10.1.1.1"></eq><cn type="integer" id="S5.p2.10.m10.1.1.2.cmml" xref="S5.p2.10.m10.1.1.2">5</cn><csymbol cd="latexml" id="S5.p2.10.m10.1.1.3.cmml" xref="S5.p2.10.m10.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.10.m10.1c">5=</annotation></semantics></math> prefers <math id="S5.p2.11.m11.1" class="ltx_Math" alttext="\pi_{i}(\mathrm{p})" display="inline"><semantics id="S5.p2.11.m11.1a"><mrow id="S5.p2.11.m11.1.2" xref="S5.p2.11.m11.1.2.cmml"><msub id="S5.p2.11.m11.1.2.2" xref="S5.p2.11.m11.1.2.2.cmml"><mi id="S5.p2.11.m11.1.2.2.2" xref="S5.p2.11.m11.1.2.2.2.cmml">π</mi><mi id="S5.p2.11.m11.1.2.2.3" xref="S5.p2.11.m11.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S5.p2.11.m11.1.2.1" xref="S5.p2.11.m11.1.2.1.cmml">​</mo><mrow id="S5.p2.11.m11.1.2.3.2" xref="S5.p2.11.m11.1.2.cmml"><mo stretchy="false" id="S5.p2.11.m11.1.2.3.2.1" xref="S5.p2.11.m11.1.2.cmml">(</mo><mi mathvariant="normal" id="S5.p2.11.m11.1.1" xref="S5.p2.11.m11.1.1.cmml">p</mi><mo stretchy="false" id="S5.p2.11.m11.1.2.3.2.2" xref="S5.p2.11.m11.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.p2.11.m11.1b"><apply id="S5.p2.11.m11.1.2.cmml" xref="S5.p2.11.m11.1.2"><times id="S5.p2.11.m11.1.2.1.cmml" xref="S5.p2.11.m11.1.2.1"></times><apply id="S5.p2.11.m11.1.2.2.cmml" xref="S5.p2.11.m11.1.2.2"><csymbol cd="ambiguous" id="S5.p2.11.m11.1.2.2.1.cmml" xref="S5.p2.11.m11.1.2.2">subscript</csymbol><ci id="S5.p2.11.m11.1.2.2.2.cmml" xref="S5.p2.11.m11.1.2.2.2">𝜋</ci><ci id="S5.p2.11.m11.1.2.2.3.cmml" xref="S5.p2.11.m11.1.2.2.3">𝑖</ci></apply><ci id="S5.p2.11.m11.1.1.cmml" xref="S5.p2.11.m11.1.1">p</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.11.m11.1c">\pi_{i}(\mathrm{p})</annotation></semantics></math> much more than <math id="S5.p2.12.m12.1" class="ltx_Math" alttext="\pi_{base}(\mathrm{p})" display="inline"><semantics id="S5.p2.12.m12.1a"><mrow id="S5.p2.12.m12.1.2" xref="S5.p2.12.m12.1.2.cmml"><msub id="S5.p2.12.m12.1.2.2" xref="S5.p2.12.m12.1.2.2.cmml"><mi id="S5.p2.12.m12.1.2.2.2" xref="S5.p2.12.m12.1.2.2.2.cmml">π</mi><mrow id="S5.p2.12.m12.1.2.2.3" xref="S5.p2.12.m12.1.2.2.3.cmml"><mi id="S5.p2.12.m12.1.2.2.3.2" xref="S5.p2.12.m12.1.2.2.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="S5.p2.12.m12.1.2.2.3.1" xref="S5.p2.12.m12.1.2.2.3.1.cmml">​</mo><mi id="S5.p2.12.m12.1.2.2.3.3" xref="S5.p2.12.m12.1.2.2.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S5.p2.12.m12.1.2.2.3.1a" xref="S5.p2.12.m12.1.2.2.3.1.cmml">​</mo><mi id="S5.p2.12.m12.1.2.2.3.4" xref="S5.p2.12.m12.1.2.2.3.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="S5.p2.12.m12.1.2.2.3.1b" xref="S5.p2.12.m12.1.2.2.3.1.cmml">​</mo><mi id="S5.p2.12.m12.1.2.2.3.5" xref="S5.p2.12.m12.1.2.2.3.5.cmml">e</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S5.p2.12.m12.1.2.1" xref="S5.p2.12.m12.1.2.1.cmml">​</mo><mrow id="S5.p2.12.m12.1.2.3.2" xref="S5.p2.12.m12.1.2.cmml"><mo stretchy="false" id="S5.p2.12.m12.1.2.3.2.1" xref="S5.p2.12.m12.1.2.cmml">(</mo><mi mathvariant="normal" id="S5.p2.12.m12.1.1" xref="S5.p2.12.m12.1.1.cmml">p</mi><mo stretchy="false" id="S5.p2.12.m12.1.2.3.2.2" xref="S5.p2.12.m12.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.p2.12.m12.1b"><apply id="S5.p2.12.m12.1.2.cmml" xref="S5.p2.12.m12.1.2"><times id="S5.p2.12.m12.1.2.1.cmml" xref="S5.p2.12.m12.1.2.1"></times><apply id="S5.p2.12.m12.1.2.2.cmml" xref="S5.p2.12.m12.1.2.2"><csymbol cd="ambiguous" id="S5.p2.12.m12.1.2.2.1.cmml" xref="S5.p2.12.m12.1.2.2">subscript</csymbol><ci id="S5.p2.12.m12.1.2.2.2.cmml" xref="S5.p2.12.m12.1.2.2.2">𝜋</ci><apply id="S5.p2.12.m12.1.2.2.3.cmml" xref="S5.p2.12.m12.1.2.2.3"><times id="S5.p2.12.m12.1.2.2.3.1.cmml" xref="S5.p2.12.m12.1.2.2.3.1"></times><ci id="S5.p2.12.m12.1.2.2.3.2.cmml" xref="S5.p2.12.m12.1.2.2.3.2">𝑏</ci><ci id="S5.p2.12.m12.1.2.2.3.3.cmml" xref="S5.p2.12.m12.1.2.2.3.3">𝑎</ci><ci id="S5.p2.12.m12.1.2.2.3.4.cmml" xref="S5.p2.12.m12.1.2.2.3.4">𝑠</ci><ci id="S5.p2.12.m12.1.2.2.3.5.cmml" xref="S5.p2.12.m12.1.2.2.3.5">𝑒</ci></apply></apply><ci id="S5.p2.12.m12.1.1.cmml" xref="S5.p2.12.m12.1.1">p</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.12.m12.1c">\pi_{base}(\mathrm{p})</annotation></semantics></math>). This way, a model will be penalized more in big losses than small losses, which we find to be effective in separating models. We also employ chain-of-thought (CoT) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite> to prompt the LLM judge to generate its own answer first before giving judgments. The full judge prompt can be found in Section <a href="#S11" title="11 Prompts ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>.</p>
</div>
<div id="S5.p3" class="ltx_para ltx_noindent">
<p id="S5.p3.1" class="ltx_p">To avoid potential position bias, we adopt a two-game setup – per query we swap the models on the first and second position. This results in 1000 judgments per model evaluation. Following Chatbot Arena, we adopt the Bradley-Terry <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite> model to produce model’s the final model scores. We aggregate all pairwise comparisons to the baseline model for all models and bootstrapping the comparisons to retrieve a bootstrapped confidence interval of all models’ win-rate against the baseline, producing a ordered ranking of all models by their win-rates.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Experimental Results</h2>

<figure id="S6.T1" class="ltx_table">
<div id="S6.T1.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:55.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-66.1pt,8.4pt) scale(0.76641268107514,0.76641268107514) ;">
<table id="S6.T1.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S6.T1.1.1.1.1" class="ltx_tr">
<th id="S6.T1.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Quality Score</th>
<th id="S6.T1.1.1.1.1.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">1+</th>
<th id="S6.T1.1.1.1.1.3" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">2+</th>
<th id="S6.T1.1.1.1.1.4" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">3+</th>
<th id="S6.T1.1.1.1.1.5" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">4+</th>
<th id="S6.T1.1.1.1.1.6" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">5+</th>
<th id="S6.T1.1.1.1.1.7" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">6+</th>
<th id="S6.T1.1.1.1.1.8" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">7+</th>
</tr>
<tr id="S6.T1.1.1.2.2" class="ltx_tr">
<th id="S6.T1.1.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t">% of queries</th>
<th id="S6.T1.1.1.2.2.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">95.4</th>
<th id="S6.T1.1.1.2.2.3" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">83.5</th>
<th id="S6.T1.1.1.2.2.4" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">61.9</th>
<th id="S6.T1.1.1.2.2.5" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">48.7</th>
<th id="S6.T1.1.1.2.2.6" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">33.8</th>
<th id="S6.T1.1.1.2.2.7" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">17.9</th>
<th id="S6.T1.1.1.2.2.8" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">0.2</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S6.T1.1.1.3.1" class="ltx_tr">
<th id="S6.T1.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Qualities</th>
<td id="S6.T1.1.1.3.1.2" class="ltx_td ltx_align_right ltx_border_tt">Specificity</td>
<td id="S6.T1.1.1.3.1.3" class="ltx_td ltx_align_right ltx_border_tt">Domain-knowledge</td>
<td id="S6.T1.1.1.3.1.4" class="ltx_td ltx_align_right ltx_border_tt">Complexity</td>
<td id="S6.T1.1.1.3.1.5" class="ltx_td ltx_align_right ltx_border_tt">Problem-solving</td>
<td id="S6.T1.1.1.3.1.6" class="ltx_td ltx_align_right ltx_border_tt">Creativity</td>
<td id="S6.T1.1.1.3.1.7" class="ltx_td ltx_align_right ltx_border_tt">Tech. Accuracy</td>
<td id="S6.T1.1.1.3.1.8" class="ltx_td ltx_align_right ltx_border_tt">Real-world</td>
</tr>
<tr id="S6.T1.1.1.4.2" class="ltx_tr">
<th id="S6.T1.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t">% of queries</th>
<td id="S6.T1.1.1.4.2.2" class="ltx_td ltx_align_right ltx_border_bb ltx_border_t">57.3</td>
<td id="S6.T1.1.1.4.2.3" class="ltx_td ltx_align_right ltx_border_bb ltx_border_t">63.4</td>
<td id="S6.T1.1.1.4.2.4" class="ltx_td ltx_align_right ltx_border_bb ltx_border_t">35.0</td>
<td id="S6.T1.1.1.4.2.5" class="ltx_td ltx_align_right ltx_border_bb ltx_border_t">34.9</td>
<td id="S6.T1.1.1.4.2.6" class="ltx_td ltx_align_right ltx_border_bb ltx_border_t">26.1</td>
<td id="S6.T1.1.1.4.2.7" class="ltx_td ltx_align_right ltx_border_bb ltx_border_t">39.0</td>
<td id="S6.T1.1.1.4.2.8" class="ltx_td ltx_align_right ltx_border_bb ltx_border_t">87.9</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>First row is the percentage of queries with quality scores of the column or more in 75K Chatbot Arena data assigned by GPT-3.5-Turbo. Second row is the percentage of queries in 75K Chatbot Arena labeled by GPT-3.5-Turbo with each of the 7 qualities.</figcaption>
</figure>
<figure id="S6.T2" class="ltx_table">
<div id="S6.T2.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:151.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-15.1pt,5.3pt) scale(0.934748128815995,0.934748128815995) ;">
<table id="S6.T2.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S6.T2.1.1.1.1" class="ltx_tr">
<th id="S6.T2.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<td id="S6.T2.1.1.1.1.2" class="ltx_td ltx_align_right ltx_border_tt">Chatbot Arena</td>
<td id="S6.T2.1.1.1.1.3" class="ltx_td ltx_align_right ltx_border_tt">MT Bench</td>
<td id="S6.T2.1.1.1.1.4" class="ltx_td ltx_align_right ltx_border_tt">AlpacaEval 2.0 LC</td>
<td id="S6.T2.1.1.1.1.5" class="ltx_td ltx_align_right ltx_border_tt">Arena-Hard-Auto v0.1</td>
</tr>
<tr id="S6.T2.1.1.2.2" class="ltx_tr">
<th id="S6.T2.1.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Confiderence Agreement</th>
<td id="S6.T2.1.1.2.2.2" class="ltx_td ltx_align_right ltx_border_t">N/A</td>
<td id="S6.T2.1.1.2.2.3" class="ltx_td ltx_align_right ltx_border_t">26.1%</td>
<td id="S6.T2.1.1.2.2.4" class="ltx_td ltx_align_right ltx_border_t">81.2%</td>
<td id="S6.T2.1.1.2.2.5" class="ltx_td ltx_align_right ltx_border_t"><span id="S6.T2.1.1.2.2.5.1" class="ltx_text ltx_font_bold">89.1%</span></td>
</tr>
<tr id="S6.T2.1.1.3.3" class="ltx_tr">
<th id="S6.T2.1.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Separability</th>
<td id="S6.T2.1.1.3.3.2" class="ltx_td ltx_align_right">85.8%</td>
<td id="S6.T2.1.1.3.3.3" class="ltx_td ltx_align_right">22.6%</td>
<td id="S6.T2.1.1.3.3.4" class="ltx_td ltx_align_right">83.2%</td>
<td id="S6.T2.1.1.3.3.5" class="ltx_td ltx_align_right"><span id="S6.T2.1.1.3.3.5.1" class="ltx_text ltx_font_bold">87.4%</span></td>
</tr>
<tr id="S6.T2.1.1.4.4" class="ltx_tr">
<th id="S6.T2.1.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Spearman Correlation</th>
<td id="S6.T2.1.1.4.4.2" class="ltx_td ltx_align_right">N/A</td>
<td id="S6.T2.1.1.4.4.3" class="ltx_td ltx_align_right">91.3%</td>
<td id="S6.T2.1.1.4.4.4" class="ltx_td ltx_align_right">90.8%</td>
<td id="S6.T2.1.1.4.4.5" class="ltx_td ltx_align_right"><span id="S6.T2.1.1.4.4.5.1" class="ltx_text ltx_font_bold">94.1%</span></td>
</tr>
<tr id="S6.T2.1.1.5.5" class="ltx_tr">
<th id="S6.T2.1.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Brier Score</th>
<td id="S6.T2.1.1.5.5.2" class="ltx_td ltx_align_right">N/A</td>
<td id="S6.T2.1.1.5.5.3" class="ltx_td ltx_align_right">0.09</td>
<td id="S6.T2.1.1.5.5.4" class="ltx_td ltx_align_right">0.11</td>
<td id="S6.T2.1.1.5.5.5" class="ltx_td ltx_align_right"><span id="S6.T2.1.1.5.5.5.1" class="ltx_text ltx_font_bold">0.069</span></td>
</tr>
<tr id="S6.T2.1.1.6.6" class="ltx_tr">
<th id="S6.T2.1.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Real-world</th>
<td id="S6.T2.1.1.6.6.2" class="ltx_td ltx_align_right ltx_border_t">Yes</td>
<td id="S6.T2.1.1.6.6.3" class="ltx_td ltx_align_right ltx_border_t">Mixed</td>
<td id="S6.T2.1.1.6.6.4" class="ltx_td ltx_align_right ltx_border_t">Mixed</td>
<td id="S6.T2.1.1.6.6.5" class="ltx_td ltx_align_right ltx_border_t">Yes</td>
</tr>
<tr id="S6.T2.1.1.7.7" class="ltx_tr">
<th id="S6.T2.1.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Freshness</th>
<td id="S6.T2.1.1.7.7.2" class="ltx_td ltx_align_right">Live</td>
<td id="S6.T2.1.1.7.7.3" class="ltx_td ltx_align_right">Static</td>
<td id="S6.T2.1.1.7.7.4" class="ltx_td ltx_align_right">Static</td>
<td id="S6.T2.1.1.7.7.5" class="ltx_td ltx_align_right">Frequent Updates</td>
</tr>
<tr id="S6.T2.1.1.8.8" class="ltx_tr">
<th id="S6.T2.1.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Eval cost per model</th>
<td id="S6.T2.1.1.8.8.2" class="ltx_td ltx_align_right">very high</td>
<td id="S6.T2.1.1.8.8.3" class="ltx_td ltx_align_right">$10</td>
<td id="S6.T2.1.1.8.8.4" class="ltx_td ltx_align_right">$10</td>
<td id="S6.T2.1.1.8.8.5" class="ltx_td ltx_align_right">$25</td>
</tr>
<tr id="S6.T2.1.1.9.9" class="ltx_tr">
<th id="S6.T2.1.1.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Prompts per model</th>
<td id="S6.T2.1.1.9.9.2" class="ltx_td ltx_align_right ltx_border_bb">10000+</td>
<td id="S6.T2.1.1.9.9.3" class="ltx_td ltx_align_right ltx_border_bb">160</td>
<td id="S6.T2.1.1.9.9.4" class="ltx_td ltx_align_right ltx_border_bb">800</td>
<td id="S6.T2.1.1.9.9.5" class="ltx_td ltx_align_right ltx_border_bb">1000</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>We use a set of top-20 models<sup id="S6.T2.3.1" class="ltx_sup">1</sup> on Chatbot Arena (2024/04/13) that are also present on the AlpacaEval leaderboard to calculate separability and agreement per benchmark. We consider the human preference ranking by Chatbot Arena (English only) as the reference to calculate agreement.</figcaption>
</figure>
<figure id="S6.T3" class="ltx_table">
<div id="S6.T3.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:80.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-25.5pt,4.7pt) scale(0.894675354130338,0.894675354130338) ;">
<table id="S6.T3.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S6.T3.1.1.1.1" class="ltx_tr">
<th id="S6.T3.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Model</th>
<th id="S6.T3.1.1.1.1.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">GPT4-T</th>
<th id="S6.T3.1.1.1.1.3" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">Claude3-Opus</th>
<th id="S6.T3.1.1.1.1.4" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">Gemini1.5-Pro</th>
<th id="S6.T3.1.1.1.1.5" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">Llama3-70B</th>
<th id="S6.T3.1.1.1.1.6" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">Ensemble-as-Judges</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S6.T3.1.1.2.1" class="ltx_tr">
<th id="S6.T3.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Confiderence Agreement</th>
<td id="S6.T3.1.1.2.1.2" class="ltx_td ltx_align_right ltx_border_t">89.1%</td>
<td id="S6.T3.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t">66.7%</td>
<td id="S6.T3.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t">84.8%</td>
<td id="S6.T3.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t">65.6%</td>
<td id="S6.T3.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t"><span id="S6.T3.1.1.2.1.6.1" class="ltx_text ltx_font_bold">91.5%</span></td>
</tr>
<tr id="S6.T3.1.1.3.2" class="ltx_tr">
<th id="S6.T3.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Separability</th>
<td id="S6.T3.1.1.3.2.2" class="ltx_td ltx_align_right">87.4%</td>
<td id="S6.T3.1.1.3.2.3" class="ltx_td ltx_align_right">83.68%</td>
<td id="S6.T3.1.1.3.2.4" class="ltx_td ltx_align_right">82.11%</td>
<td id="S6.T3.1.1.3.2.5" class="ltx_td ltx_align_right">81.6%</td>
<td id="S6.T3.1.1.3.2.6" class="ltx_td ltx_align_right"><span id="S6.T3.1.1.3.2.6.1" class="ltx_text ltx_font_bold">89.5%</span></td>
</tr>
<tr id="S6.T3.1.1.4.3" class="ltx_tr">
<th id="S6.T3.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Spearman Correlation</th>
<td id="S6.T3.1.1.4.3.2" class="ltx_td ltx_align_right">94.1%</td>
<td id="S6.T3.1.1.4.3.3" class="ltx_td ltx_align_right">77.0%</td>
<td id="S6.T3.1.1.4.3.4" class="ltx_td ltx_align_right">95.2%</td>
<td id="S6.T3.1.1.4.3.5" class="ltx_td ltx_align_right">70.5%</td>
<td id="S6.T3.1.1.4.3.6" class="ltx_td ltx_align_right"><span id="S6.T3.1.1.4.3.6.1" class="ltx_text ltx_font_bold">96.5%</span></td>
</tr>
<tr id="S6.T3.1.1.5.4" class="ltx_tr">
<th id="S6.T3.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Brier Score</th>
<td id="S6.T3.1.1.5.4.2" class="ltx_td ltx_align_right ltx_border_bb">0.069</td>
<td id="S6.T3.1.1.5.4.3" class="ltx_td ltx_align_right ltx_border_bb">0.170</td>
<td id="S6.T3.1.1.5.4.4" class="ltx_td ltx_align_right ltx_border_bb"><span id="S6.T3.1.1.5.4.4.1" class="ltx_text ltx_font_bold">0.064</span></td>
<td id="S6.T3.1.1.5.4.5" class="ltx_td ltx_align_right ltx_border_bb">0.196</td>
<td id="S6.T3.1.1.5.4.6" class="ltx_td ltx_align_right ltx_border_bb">0.065</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Statistics of Arena-Hard-Auto v0.1 with four LLM different judges: GPT4-T (gpt-4-1106-preview), Claude-3-Opus, Gemini1.5-Pro (gemini-1.5-pro-0514), Llama3-70B (llama-3-70b-instruct). We compare rankings produced by these judges against Chatbot Arena (English) ranking (as of 2024/04/13). We observe GPT-4T and Gemini1.5-Pro have higher agreement than Claude-3-Opus and Llama-3-70B. Furthermore, the ensemble of GPT4-T and Gemini1.5-Pro shows even higher agreement.</figcaption>
</figure>
<figure id="S6.T4" class="ltx_table">
<div id="S6.T4.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:346.9pt;height:74.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-35.1pt,7.6pt) scale(0.831886568483667,0.831886568483667) ;">
<table id="S6.T4.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S6.T4.1.1.1.1" class="ltx_tr">
<th id="S6.T4.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Model</th>
<th id="S6.T4.1.1.1.1.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">Arena-Hard-Auto v0.1</th>
<th id="S6.T4.1.1.1.1.3" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">Random Sample 1</th>
<th id="S6.T4.1.1.1.1.4" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">Random Sample 2</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S6.T4.1.1.2.1" class="ltx_tr">
<th id="S6.T4.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Confiderence Agreement</th>
<td id="S6.T4.1.1.2.1.2" class="ltx_td ltx_align_right ltx_border_t"><span id="S6.T4.1.1.2.1.2.1" class="ltx_text ltx_font_bold">84.2%</span></td>
<td id="S6.T4.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t">57.5%</td>
<td id="S6.T4.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t">66.1%</td>
</tr>
<tr id="S6.T4.1.1.3.2" class="ltx_tr">
<th id="S6.T4.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Separability</th>
<td id="S6.T4.1.1.3.2.2" class="ltx_td ltx_align_right"><span id="S6.T4.1.1.3.2.2.1" class="ltx_text ltx_font_bold">80.5%</span></td>
<td id="S6.T4.1.1.3.2.3" class="ltx_td ltx_align_right">74.7%</td>
<td id="S6.T4.1.1.3.2.4" class="ltx_td ltx_align_right">76.3%</td>
</tr>
<tr id="S6.T4.1.1.4.3" class="ltx_tr">
<th id="S6.T4.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Spearman Correlation</th>
<td id="S6.T4.1.1.4.3.2" class="ltx_td ltx_align_right"><span id="S6.T4.1.1.4.3.2.1" class="ltx_text ltx_font_bold">94.7%</span></td>
<td id="S6.T4.1.1.4.3.3" class="ltx_td ltx_align_right">64.7%</td>
<td id="S6.T4.1.1.4.3.4" class="ltx_td ltx_align_right">72.5%</td>
</tr>
<tr id="S6.T4.1.1.5.4" class="ltx_tr">
<th id="S6.T4.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Brier Score</th>
<td id="S6.T4.1.1.5.4.2" class="ltx_td ltx_align_right ltx_border_bb"><span id="S6.T4.1.1.5.4.2.1" class="ltx_text ltx_font_bold">0.069</span></td>
<td id="S6.T4.1.1.5.4.3" class="ltx_td ltx_align_right ltx_border_bb">0.215</td>
<td id="S6.T4.1.1.5.4.4" class="ltx_td ltx_align_right ltx_border_bb">0.162</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>We compare Arena-Hard-Auto v0.1 with two sets of 500 prompts randomly sampled from 75K Chatbot Arena user queries. We evaluate the set of top-20 models and compare various statistics across. Each prompt is judged only once by positioning the baseline answer first.</figcaption>
</figure>
<figure id="S6.T5" class="ltx_table">
<table id="S6.T5.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S6.T5.1.1.1" class="ltx_tr">
<th id="S6.T5.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S6.T5.1.1.1.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">Arena-Hard-Auto v0.1</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S6.T5.1.2.1" class="ltx_tr">
<th id="S6.T5.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Confiderence Agreement</th>
<td id="S6.T5.1.2.1.2" class="ltx_td ltx_align_right ltx_border_t">92.0%</td>
</tr>
<tr id="S6.T5.1.3.2" class="ltx_tr">
<th id="S6.T5.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Spearman Correlation</th>
<td id="S6.T5.1.3.2.2" class="ltx_td ltx_align_right">96.4%</td>
</tr>
<tr id="S6.T5.1.4.3" class="ltx_tr">
<th id="S6.T5.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Brier Score</th>
<td id="S6.T5.1.4.3.2" class="ltx_td ltx_align_right ltx_border_bb">0.055</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>We compare Arena-Hard-Auto v0.1 (gpt-4-1106-preview as judge) to Chatbot Arena Category Hard Prompt (English) on the same set of top-20 models*. By comparing Arena-Hard-Auto v0.1 to a similar distribution of queries from Chatbot Arena, we obtain even higher alignment to human preferences.</figcaption>
</figure>
<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.1 </span>Setup and Baselines</h3>

<div id="S6.SS1.p1" class="ltx_para ltx_noindent">
<p id="S6.SS1.p1.1" class="ltx_p">To compare Arena-Hard-Auto v0.1’s <span id="S6.SS1.p1.1.1" class="ltx_text ltx_font_italic">separability</span> and <span id="S6.SS1.p1.1.2" class="ltx_text ltx_font_italic">alignment with humans</span> against other widely used benchmarks, MT-Bench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite> and AlpacaEval 2.0 Length Controlled <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite>, we obtain 95% confidence interval of model performances via applying 100 rounds of bootstrapping on judgment results for each benchmark. For AlpacaEval, we use pre-existing results from their repository. We obtain MT-Bench judgment with no modification to their recommended evaluation setup. For Arena-Hard-Auto v0.1, we employ the system proposed in <a href="#S5" title="5 Evaluation with LLM-as-a-Judge ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">section 5</span></a> by choosing gpt-4-0314 as baseline model for pairwise comparison.</p>
</div>
<div id="S6.SS1.p2" class="ltx_para ltx_noindent">
<p id="S6.SS1.p2.1" class="ltx_p">In order to compare these benchmarks fairly, we use a set of top-20 models<span id="footnotex2" class="ltx_note ltx_role_footnotemark"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">footnotemark: </span></span></span></span> on Chatbot Arena <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite> (2024/04/13) that are also presented on AlpacaEval leaderboard (2024/04/13) as ground truth for human preferences on the model ranking orders. In <a href="#S6.SS3" title="6.3 Comparing to a similar distribution of human preference ‣ 6 Experimental Results ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">subsection 6.3</span></a>, we individually compare Arena-Hard-Auto v0.1 to Chatbot Arena within a more similar distribution of user queries. We compute separability using 95% confidence intervals and consider the human preference ranking by Chatbot Arena as the reference to calculate agreement, Spearman correlation, and Pair Rank Brier Score across AlpacaEval, MT-Bench, and Arena-Hard-Auto v0.1.</p>
</div>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.2 </span>Separability and Agreement Across Benchmarks</h3>

<div id="S6.SS2.p1" class="ltx_para ltx_noindent">
<p id="S6.SS2.p1.1" class="ltx_p">In Table <a href="#S6.T2" title="Table 2 ‣ 6 Experimental Results ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, Arena-Hard-Auto v0.1 shows the highest separability (87.4%) against widely adopted LLM benchmarks and offers highest agreement (89.1%) to Chatbot Arena at $25 cost. In Figure <a href="#S4.F3" title="Figure 3 ‣ 4.2 Arena-Hard-Auto v0.1: A set of challenging, high-quality prompts from Chatbot Arena ‣ 4 The BenchBuilder Pipeline and Arena-Hard-Auto v0.1 Dataset ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we show Arena-Hard-Auto v0.1 offers significantly stronger separability against MT Bench with tighter confidence intervals. With only 500 prompts, Arena-Hard-Auto v0.1 achieve impressive alignment to and even higher separability than Chatbot Arena Ranking, which constitutes for approximately 1 million real-world human preferences.</p>
</div>
<div id="S6.SS2.p2" class="ltx_para ltx_noindent">
<p id="S6.SS2.p2.1" class="ltx_p">Notably, we observe a significant gap between MT-bench’s Spearman Correlation (91.3%) and confidence agreement (22.6%) to Chatbot Arena, an example where Spearman Correlation fails to account for variance of the rankings, and hence cannot adequately measure important ranking granularity of top LLMs. We present a visual comparison between Arena-Hard-Auto v0.1 and MT-Bench in <a href="#S4.F3" title="Figure 3 ‣ 4.2 Arena-Hard-Auto v0.1: A set of challenging, high-quality prompts from Chatbot Arena ‣ 4 The BenchBuilder Pipeline and Arena-Hard-Auto v0.1 Dataset ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 3</span></a>, highlighting Arena-Hard-Auto v0.1’s improved separability.</p>
</div>
</section>
<section id="S6.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.3 </span>Comparing to a similar distribution of human preference</h3>

<div id="S6.SS3.p1" class="ltx_para ltx_noindent">
<p id="S6.SS3.p1.1" class="ltx_p">We set Chatbot Arena Category Hard Prompts (English) as ground truth reference. Chatbot Arena Category Hard Prompt is a human preference ranking leaderboard based on votes from a harder subset of 200,000+ prompts extracted from over 1 million Chatbot Arena queries using a similar method as BenchBuilder. In <a href="#S6.T5" title="Table 5 ‣ 6 Experimental Results ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 5</span></a>, we show Arena-Hard-Auto v0.1 achieving an overall higher alignment (92.0% Confidence Agreement) to human preferences.</p>
</div>
</section>
<section id="S6.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.4 </span>Comparing to Random Subsets of Arena</h3>

<div id="S6.SS4.p1" class="ltx_para ltx_noindent">
<p id="S6.SS4.p1.1" class="ltx_p">To further verify BenchBuilder pipeline’s ability to select high-quality prompts, we compare Arena-Hard-Auto v0.1 to two sets of 500 randomly selected prompts from the 75,000 user queries prior to pipeline extraction. We compare their ranking preferences of the set of top-20 models produced by GPT-4-1106-Preview as judge. In Table <a href="#S6.T4" title="Table 4 ‣ 6 Experimental Results ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we observe Arena-Hard-Auto v0.1 is significantly better than both random baselines, verifying BenchBuilder’s effectiveness at selecting superior prompts for a LLM benchmark.</p>
</div>
</section>
<section id="S6.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.5 </span>Ensemble-as-Judges</h3>

<div id="S6.SS5.p1" class="ltx_para ltx_noindent">
<p id="S6.SS5.p1.1" class="ltx_p">We study four popular LLMs<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>GPT-4-1106-Preview, Claude-3-Opus-20240229, Gemini-1.5-Pro-0514, and Llama-3-70b-instruct</span></span></span> as judge on Arena-Hard-Auto v0.1. We statistically compare their ranking preference of the set of top-20 models<span id="footnotex3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>gpt-4-turbo-2024-04-09, claude-3-opus-20240229, claude-3-sonnet-20240229, gpt-4-0314 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>, gpt-4-0613, mistral-large-2402, qwen1.5-72b-chat <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>, mistral-medium, claude-2.0, gpt-3.5-turbo-0613, claude-2.1, gemini-pro <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>, mixtral-8x7b-instruct-v0.1, gpt-3.5-turbo-0314, yi-34b-chat <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite>, tulu-2-dpo-70b <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>, dbrx-instruct-preview <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite>, vicuna-33b <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib8" title="" class="ltx_ref">8</a>]</cite>, starling-lm-7b-alpha <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>, llama-2-70b-chat</span></span></span> to the Chatbot Arena Leaderboard in Table <a href="#S6.T3" title="Table 3 ‣ 6 Experimental Results ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Out of the four judges, GPT-4-1106-Preview has the highest separability and agreement to human ranking preference. We also observe GPT-4-1106-Preview and Gemini-1.5-Pro have much higher agreement than Claude-3-Opus and Llama-3-70b-instruct, with more than 20% improvement.</p>
</div>
<div id="S6.SS5.p2" class="ltx_para ltx_noindent">
<p id="S6.SS5.p2.1" class="ltx_p">We also experiment combining GPT-4-1106-Preview and Gemini-1.5-Pro as an ensemble judge by combining their judgment results. This achieves an even higher separability and agreement to human ranking preference, with agreement reaching 91.5%, demonstrating effectiveness of judging LLMs with Ensemble-as-Judges.</p>
</div>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Limitations</h2>

<div id="S7.p1" class="ltx_para ltx_noindent">
<p id="S7.p1.1" class="ltx_p">Although our data source comes from diverse distributions, we acknowledge possible biases within our pipeline. For example, the 7 qualities we defined might not be all-encompassing and could potentially bias towards a limited distribution of prompts, such as technical fields. Further, prompt selection process could be biased by LLM annotators. Additionally, for model evaluation, we are employing LLMs as judges, who are known to suffer from verbosity bias <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>, <a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite>. In Table <a href="#S10.T7" title="Table 7 ‣ 10 Appendix ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, we present how verbosity can effect LLM as Judge on Arena-Hard-Auto v0.1. We also acknowledge potential self-bias in LLM judges (e.g., GPT-4 Judge prefers GPT-4 answers) in Table <a href="#S6.T3" title="Table 3 ‣ 6 Experimental Results ‣ From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Arena-Hard-Auto v0.1 also does not evaluate multi-turn or non-English interactions.</p>
</div>
<div id="S7.p2" class="ltx_para ltx_noindent">
<p id="S7.p2.1" class="ltx_p">In the future, we plan to further the exploration of a more generalized and diverse set of prompt qualities, establishing a pipeline to systematically produce benchmark across a more comprehensive and inclusive distribution of prompts while maintaining high separability and alignment to humans. Additionally, we believe our approach of utilizing LLM-as-a-judge can be improved, such as developing a system of Ensemble-as-Judges, similar to Judges with Jury <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>, to reduce biases and improve correlation.</p>
</div>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Conclusions</h2>

<div id="S8.p1" class="ltx_para ltx_noindent">
<p id="S8.p1.1" class="ltx_p">We introduced BenchBuilder, a data curation pipeline that transforms crowdsourced data into high-quality benchmarks by seven key qualities. This pipeline enables building challenging and evolving benchmarks which is crucial for evaluating today’s advanced language models.
Our evaluation metrics, including separability and agreement with confidence, provide a comprehensive assessment of benchmarks.
We show the resulting benchmark, Arena-Hard-Auto v0.1, significantly improves separability and alignment with human preferences over existing benchmarks, achieving 89.1% agreement with Chatbot Arena rankings at only $25 per evaluation.
We expect Arena-Hard-Auto v0.1 to be useful for LLM developers to evaluate their models with confidence and BenchBuilder to be a valuable tool for developers seeking to extract high-quality benchmark from vast amounts of data with minimal human effort.</p>
</div>
</section>
<section id="S9" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">9 </span>Acknowledgment</h2>

<div id="S9.p1" class="ltx_para ltx_noindent">
<p id="S9.p1.1" class="ltx_p">We thank Matei Zaharia, Yann Dubois, Anastasios Angelopoulos, Lianmin Zheng, Lewis Tunstall, Nathan Lambert, Xuechen Li, Naman Jain, Ying Sheng, Maarten Grootendorst for their valuable feedback. We thank Siyuan Zhuang and Dacheng Li for the valuable review and debug of the code. We thank Microsoft AFMR for Azure OpenAI credits support. We also thank Together.ai &amp; Anyscale for open model endpoint support.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[1]</span>
<span class="ltx_bibblock">
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai.

</span>
<span class="ltx_bibblock">Yi: Open foundation models by 01.ai, 2024.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[2]</span>
<span class="ltx_bibblock">
Anthropic.

</span>
<span class="ltx_bibblock">The claude 3 model family: Opus, sonnet, haiku.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf</a>, 2024.

</span>
<span class="ltx_bibblock">(Accessed on 06/05/2024).

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[3]</span>
<span class="ltx_bibblock">
Johan Bos and Katja Markert.

</span>
<span class="ltx_bibblock">Recognising textual entailment with logical inference.

</span>
<span class="ltx_bibblock">In Raymond Mooney, Chris Brew, Lee-Feng Chien, and Katrin Kirchhoff, editors, <span id="bib.bib3.1.1" class="ltx_text ltx_font_italic">Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing</span>, pages 628–635, Vancouver, British Columbia, Canada, October 2005. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[4]</span>
<span class="ltx_bibblock">
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.

</span>
<span class="ltx_bibblock">A large annotated corpus for learning natural language inference.

</span>
<span class="ltx_bibblock">In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors, <span id="bib.bib4.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing</span>, pages 632–642, Lisbon, Portugal, September 2015. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[5]</span>
<span class="ltx_bibblock">
Ralph Allan Bradley and Milton E Terry.

</span>
<span class="ltx_bibblock">Rank analysis of incomplete block designs: I. the method of paired comparisons.

</span>
<span class="ltx_bibblock"><span id="bib.bib5.1.1" class="ltx_text ltx_font_italic">Biometrika</span>, 39(3/4):324–345, 1952.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[6]</span>
<span class="ltx_bibblock">
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al.

</span>
<span class="ltx_bibblock">Extracting training data from large language models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib6.1.1" class="ltx_text ltx_font_italic">30th USENIX Security Symposium (USENIX Security 21)</span>, pages 2633–2650, 2021.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[7]</span>
<span class="ltx_bibblock">
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.

</span>
<span class="ltx_bibblock">Evaluating large language models trained on code.

</span>
<span class="ltx_bibblock"><span id="bib.bib7.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2107.03374</span>, 2021.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[8]</span>
<span class="ltx_bibblock">
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.

</span>
<span class="ltx_bibblock">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[9]</span>
<span class="ltx_bibblock">
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica.

</span>
<span class="ltx_bibblock">Chatbot arena: An open platform for evaluating llms by human preference, 2024.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[10]</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.

</span>
<span class="ltx_bibblock">Training verifiers to solve math word problems.

</span>
<span class="ltx_bibblock"><span id="bib.bib10.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2110.14168</span>, 2021.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[11]</span>
<span class="ltx_bibblock">
Bill Dolan and Chris Brockett.

</span>
<span class="ltx_bibblock">Automatically constructing a corpus of sentential paraphrases.

</span>
<span class="ltx_bibblock">In <span id="bib.bib11.1.1" class="ltx_text ltx_font_italic">Third international workshop on paraphrasing (IWP2005)</span>, 2005.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[12]</span>
<span class="ltx_bibblock">
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.

</span>
<span class="ltx_bibblock">Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.

</span>
<span class="ltx_bibblock">In <span id="bib.bib12.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)</span>, pages 2368–2378, 2019.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[13]</span>
<span class="ltx_bibblock">
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto.

</span>
<span class="ltx_bibblock">Length-controlled alpacaeval: A simple way to debias automatic evaluators.

</span>
<span class="ltx_bibblock"><span id="bib.bib13.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2404.04475</span>, 2024.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[14]</span>
<span class="ltx_bibblock">
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.

</span>
<span class="ltx_bibblock">Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[15]</span>
<span class="ltx_bibblock">
Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.

</span>
<span class="ltx_bibblock">Gemini: a family of highly capable multimodal models.

</span>
<span class="ltx_bibblock"><span id="bib.bib15.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2312.11805</span>, 2023.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[16]</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding.

</span>
<span class="ltx_bibblock"><span id="bib.bib16.1.1" class="ltx_text ltx_font_italic">Proceedings of the International Conference on Learning Representations (ICLR)</span>, 2021.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[17]</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring mathematical problem solving with the math dataset.

</span>
<span class="ltx_bibblock"><span id="bib.bib17.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, 2021.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[18]</span>
<span class="ltx_bibblock">
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[19]</span>
<span class="ltx_bibblock">
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica.

</span>
<span class="ltx_bibblock">Livecodebench: Holistic and contamination free evaluation of large language models for code.

</span>
<span class="ltx_bibblock"><span id="bib.bib19.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2403.07974</span>, 2024.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[20]</span>
<span class="ltx_bibblock">
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica.

</span>
<span class="ltx_bibblock">R2e: Turning any github repository into a programming agent environment.

</span>
<span class="ltx_bibblock">In <span id="bib.bib20.1.1" class="ltx_text ltx_font_italic">ICML</span>, 2024.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[21]</span>
<span class="ltx_bibblock">
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.

</span>
<span class="ltx_bibblock">Mixtral of experts, 2024.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[22]</span>
<span class="ltx_bibblock">
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.

</span>
<span class="ltx_bibblock">Swe-bench: Can language models resolve real-world github issues?

</span>
<span class="ltx_bibblock">In <span id="bib.bib22.1.1" class="ltx_text ltx_font_italic">The Twelfth International Conference on Learning Representations</span>, 2024.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[23]</span>
<span class="ltx_bibblock">
Marzena Karpinska, Nader Akoury, and Mohit Iyyer.

</span>
<span class="ltx_bibblock">The perils of using Mechanical Turk to evaluate open-ended text generation.

</span>
<span class="ltx_bibblock">In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, <span id="bib.bib23.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing</span>, pages 1265–1285, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[24]</span>
<span class="ltx_bibblock">
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams.

</span>
<span class="ltx_bibblock">Dynabench: Rethinking benchmarking in nlp.

</span>
<span class="ltx_bibblock"><span id="bib.bib24.1.1" class="ltx_text ltx_font_italic">NAACL</span>, 2021.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[25]</span>
<span class="ltx_bibblock">
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.

</span>
<span class="ltx_bibblock">Alpacaeval: An automatic evaluator of instruction-following models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/tatsu-lab/alpaca_eval" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tatsu-lab/alpaca_eval</a>, 2023.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[26]</span>
<span class="ltx_bibblock">
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al.

</span>
<span class="ltx_bibblock">Holistic evaluation of language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib26.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2211.09110</span>, 2022.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[27]</span>
<span class="ltx_bibblock">
Rensis Likert.

</span>
<span class="ltx_bibblock">A technique for the measurement of attitudes.

</span>
<span class="ltx_bibblock"><span id="bib.bib27.1.1" class="ltx_text ltx_font_italic">Archives of psychology</span>, 1932.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[28]</span>
<span class="ltx_bibblock">
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang.

</span>
<span class="ltx_bibblock">Agentbench: Evaluating llms as agents, 2023.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[29]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Gpt-4 technical report.

</span>
<span class="ltx_bibblock"><span id="bib.bib29.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2303.08774</span>, 2023.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[30]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">New models and developer products announced at devday.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openai.com/blog/new-models-and-developer-products-announced-at-devday" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openai.com/blog/new-models-and-developer-products-announced-at-devday</a>, 2023.

</span>
<span class="ltx_bibblock">(Accessed on 06/05/2024).

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[31]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Hello gpt-4o.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openai.com/index/hello-gpt-4o/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openai.com/index/hello-gpt-4o/</a>, 2024.

</span>
<span class="ltx_bibblock">(Accessed on 06/05/2024).

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[32]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">New embedding models and api updates.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openai.com/index/new-embedding-models-and-api-updates/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openai.com/index/new-embedding-models-and-api-updates/</a>, 2024.

</span>
<span class="ltx_bibblock">(Accessed on 06/05/2024).

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[33]</span>
<span class="ltx_bibblock">
Karl Pearson.

</span>
<span class="ltx_bibblock">Note on regression and inheritance in the case of two parents.

</span>
<span class="ltx_bibblock"><span id="bib.bib33.1.1" class="ltx_text ltx_font_italic">Proceedings of the Royal Society of London</span>, 58:240–242, 1895.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[34]</span>
<span class="ltx_bibblock">
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.

</span>
<span class="ltx_bibblock">Squad: 100,000+ questions for machine comprehension of text.

</span>
<span class="ltx_bibblock"><span id="bib.bib34.1.1" class="ltx_text ltx_font_italic">EMNLP</span>, 2016.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[35]</span>
<span class="ltx_bibblock">
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman.

</span>
<span class="ltx_bibblock">Gpqa: A graduate-level google-proof q&amp;a benchmark, 2023.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[36]</span>
<span class="ltx_bibblock">
Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre.

</span>
<span class="ltx_bibblock">NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark.

</span>
<span class="ltx_bibblock">In Houda Bouamor, Juan Pino, and Kalika Bali, editors, <span id="bib.bib36.1.1" class="ltx_text ltx_font_italic">Findings of the Association for Computational Linguistics: EMNLP 2023</span>, pages 10776–10787, Singapore, December 2023. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[37]</span>
<span class="ltx_bibblock">
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.

</span>
<span class="ltx_bibblock">Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib37.1.1" class="ltx_text ltx_font_italic">Transactions on Machine Learning Research</span>, 2023.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[38]</span>
<span class="ltx_bibblock">
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.

</span>
<span class="ltx_bibblock">Stanford alpaca: An instruction-following llama model.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/tatsu-lab/stanford_alpaca" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tatsu-lab/stanford_alpaca</a>, 2023.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[39]</span>
<span class="ltx_bibblock">
Gemini Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry, Lepikhin, Timothy Lillicrap, Jean baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, Luke Vilnis, Oscar Chang, Nobuyuki Morioka, George Tucker, Ce Zheng, Oliver Woodman, Nithya Attaluri, Tomas Kocisky, Evgenii Eltyshev, Xi Chen, Timothy Chung, Vittorio Selo, Siddhartha Brahma, Petko
Georgiev, Ambrose Slone, Zhenkai Zhu, James Lottes, Siyuan Qiao, Ben Caine, Sebastian Riedel, Alex Tomala, Martin Chadwick, Juliette Love, Peter Choy, Sid Mittal, Neil Houlsby, Yunhao Tang, Matthew Lamm, Libin Bai, Qiao Zhang, Luheng He, Yong Cheng, Peter Humphreys, Yujia Li, Sergey Brin, Albin Cassirer, Yingjie Miao, Lukas Zilka, Taylor Tobin, Kelvin Xu, Lev Proleev, Daniel Sohn, Alberto Magni, Lisa Anne Hendricks, Isabel Gao, Santiago Ontanon, Oskar Bunyan, Nathan Byrd, Abhanshu Sharma, Biao Zhang, Mario Pinto, Rishika Sinha, Harsh Mehta, Dawei Jia, Sergi Caelles, Albert Webson, Alex Morris, Becca Roelofs, Yifan Ding, Robin Strudel, Xuehan Xiong, Marvin Ritter, Mostafa Dehghani, Rahma Chaabouni, Abhijit Karmarkar, Guangda Lai, Fabian Mentzer, Bibo Xu, YaGuang Li, Yujing Zhang, Tom Le Paine, Alex Goldin, Behnam Neyshabur, Kate Baumli, Anselm Levskaya, Michael Laskin, Wenhao Jia, Jack W. Rae, Kefan Xiao, Antoine He, Skye Giordano, Lakshman Yagati, Jean-Baptiste Lespiau, Paul Natsev, Sanjay Ganapathy, Fangyu
Liu, Danilo Martins, Nanxin Chen, Yunhan Xu, Megan Barnes, Rhys May, Arpi Vezer, Junhyuk Oh, Ken Franko, Sophie Bridgers, Ruizhe Zhao, Boxi Wu, Basil Mustafa, Sean Sechrist, Emilio Parisotto, Thanumalayan Sankaranarayana Pillai, Chris Larkin, Chenjie Gu, Christina Sorokin, Maxim Krikun, Alexey Guseynov, Jessica Landon, Romina Datta, Alexander Pritzel, Phoebe Thacker, Fan Yang, Kevin Hui, Anja Hauth, Chih-Kuan Yeh, David Barker, Justin Mao-Jones, Sophia Austin, Hannah Sheahan, Parker Schuh, James Svensson, Rohan Jain, Vinay Ramasesh, Anton Briukhov, Da-Woon Chung, Tamara von Glehn, Christina Butterfield, Priya Jhakra, Matthew Wiethoff, Justin Frye, Jordan Grimstad, Beer Changpinyo, Charline Le Lan, Anna Bortsova, Yonghui Wu, Paul Voigtlaender, Tara Sainath, Shane Gu, Charlotte Smith, Will Hawkins, Kris Cao, James Besley, Srivatsan Srinivasan, Mark Omernick, Colin Gaffney, Gabriela Surita, Ryan Burnell, Bogdan Damoc, Junwhan Ahn, Andrew Brock, Mantas Pajarskas, Anastasia Petrushkina, Seb Noury, Lorenzo
Blanco, Kevin Swersky, Arun Ahuja, Thi Avrahami, Vedant Misra, Raoul de Liedekerke, Mariko Iinuma, Alex Polozov, Sarah York, George van den Driessche, Paul Michel, Justin Chiu, Rory Blevins, Zach Gleicher, Adrià Recasens, Alban Rrustemi, Elena Gribovskaya, Aurko Roy, Wiktor Gworek, Sébastien M. R. Arnold, Lisa Lee, James Lee-Thorp, Marcello Maggioni, Enrique Piqueras, Kartikeya Badola, Sharad Vikram, Lucas Gonzalez, Anirudh Baddepudi, Evan Senter, Jacob Devlin, James Qin, Michael Azzam, Maja Trebacz, Martin Polacek, Kashyap Krishnakumar, Shuo yiin Chang, Matthew Tung, Ivo Penchev, Rishabh Joshi, Kate Olszewska, Carrie Muir, Mateo Wirth, Ale Jakse Hartman, Josh Newlan, Sheleem Kashem, Vijay Bolina, Elahe Dabir, Joost van Amersfoort, Zafarali Ahmed, James Cobon-Kerr, Aishwarya Kamath, Arnar Mar Hrafnkelsson, Le Hou, Ian Mackinnon, Alexandre Frechette, Eric Noland, Xiance Si, Emanuel Taropa, Dong Li, Phil Crone, Anmol Gulati, Sébastien Cevey, Jonas Adler, Ada Ma, David Silver, Simon Tokumine, Richard
Powell, Stephan Lee, Kiran Vodrahalli, Samer Hassan, Diana Mincu, Antoine Yang, Nir Levine, Jenny Brennan, Mingqiu Wang, Sarah Hodkinson, Jeffrey Zhao, Josh Lipschultz, Aedan Pope, Michael B. Chang, Cheng Li, Laurent El Shafey, Michela Paganini, Sholto Douglas, Bernd Bohnet, Fabio Pardo, Seth Odoom, Mihaela Rosca, Cicero Nogueira dos Santos, Kedar Soparkar, Arthur Guez, Tom Hudson, Steven Hansen, Chulayuth Asawaroengchai, Ravi Addanki, Tianhe Yu, Wojciech Stokowiec, Mina Khan, Justin Gilmer, Jaehoon Lee, Carrie Grimes Bostock, Keran Rong, Jonathan Caton, Pedram Pejman, Filip Pavetic, Geoff Brown, Vivek Sharma, Mario Lučić, Rajkumar Samuel, Josip Djolonga, Amol Mandhane, Lars Lowe Sjösund, Elena Buchatskaya, Elspeth White, Natalie Clay, Jiepu Jiang, Hyeontaek Lim, Ross Hemsley, Zeyncep Cankara, Jane Labanowski, Nicola De Cao, David Steiner, Sayed Hadi Hashemi, Jacob Austin, Anita Gergely, Tim Blyth, Joe Stanton, Kaushik Shivakumar, Aditya Siddhant, Anders Andreassen, Carlos Araya, Nikhil Sethi, Rakesh
Shivanna, Steven Hand, Ankur Bapna, Ali Khodaei, Antoine Miech, Garrett Tanzer, Andy Swing, Shantanu Thakoor, Lora Aroyo, Zhufeng Pan, Zachary Nado, Jakub Sygnowski, Stephanie Winkler, Dian Yu, Mohammad Saleh, Loren Maggiore, Yamini Bansal, Xavier Garcia, Mehran Kazemi, Piyush Patil, Ishita Dasgupta, Iain Barr, Minh Giang, Thais Kagohara, Ivo Danihelka, Amit Marathe, Vladimir Feinberg, Mohamed Elhawaty, Nimesh Ghelani, Dan Horgan, Helen Miller, Lexi Walker, Richard Tanburn, Mukarram Tariq, Disha Shrivastava, Fei Xia, Qingze Wang, Chung-Cheng Chiu, Zoe Ashwood, Khuslen Baatarsukh, Sina Samangooei, Raphaël Lopez Kaufman, Fred Alcober, Axel Stjerngren, Paul Komarek, Katerina Tsihlas, Anudhyan Boral, Ramona Comanescu, Jeremy Chen, Ruibo Liu, Chris Welty, Dawn Bloxwich, Charlie Chen, Yanhua Sun, Fangxiaoyu Feng, Matthew Mauger, Xerxes Dotiwalla, Vincent Hellendoorn, Michael Sharman, Ivy Zheng, Krishna Haridasan, Gabe Barth-Maron, Craig Swanson, Dominika Rogozińska, Alek Andreev, Paul Kishan Rubenstein, Ruoxin
Sang, Dan Hurt, Gamaleldin Elsayed, Renshen Wang, Dave Lacey, Anastasija Ilić, Yao Zhao, Adam Iwanicki, Alejandro Lince, Alexander Chen, Christina Lyu, Carl Lebsack, Jordan Griffith, Meenu Gaba, Paramjit Sandhu, Phil Chen, Anna Koop, Ravi Rajwar, Soheil Hassas Yeganeh, Solomon Chang, Rui Zhu, Soroush Radpour, Elnaz Davoodi, Ving Ian Lei, Yang Xu, Daniel Toyama, Constant Segal, Martin Wicke, Hanzhao Lin, Anna Bulanova, Adrià Puigdomènech Badia, Nemanja Rakićević, Pablo Sprechmann, Angelos Filos, Shaobo Hou, Víctor Campos, Nora Kassner, Devendra Sachan, Meire Fortunato, Chimezie Iwuanyanwu, Vitaly Nikolaev, Balaji Lakshminarayanan, Sadegh Jazayeri, Mani Varadarajan, Chetan Tekur, Doug Fritz, Misha Khalman, David Reitter, Kingshuk Dasgupta, Shourya Sarcar, Tina Ornduff, Javier Snaider, Fantine Huot, Johnson Jia, Rupert Kemp, Nejc Trdin, Anitha Vijayakumar, Lucy Kim, Christof Angermueller, Li Lao, Tianqi Liu, Haibin Zhang, David Engel, Somer Greene, Anaïs White, Jessica Austin, Lilly Taylor, Shereen
Ashraf, Dangyi Liu, Maria Georgaki, Irene Cai, Yana Kulizhskaya, Sonam Goenka, Brennan Saeta, Ying Xu, Christian Frank, Dario de Cesare, Brona Robenek, Harry Richardson, Mahmoud Alnahlawi, Christopher Yew, Priya Ponnapalli, Marco Tagliasacchi, Alex Korchemniy, Yelin Kim, Dinghua Li, Bill Rosgen, Kyle Levin, Jeremy Wiesner, Praseem Banzal, Praveen Srinivasan, Hongkun Yu, Çağlar Ünlü, David Reid, Zora Tung, Daniel Finchelstein, Ravin Kumar, Andre Elisseeff, Jin Huang, Ming Zhang, Ricardo Aguilar, Mai Giménez, Jiawei Xia, Olivier Dousse, Willi Gierke, Damion Yates, Komal Jalan, Lu Li, Eri Latorre-Chimoto, Duc Dung Nguyen, Ken Durden, Praveen Kallakuri, Yaxin Liu, Matthew Johnson, Tomy Tsai, Alice Talbert, Jasmine Liu, Alexander Neitz, Chen Elkind, Marco Selvi, Mimi Jasarevic, Livio Baldini Soares, Albert Cui, Pidong Wang, Alek Wenjiao Wang, Xinyu Ye, Krystal Kallarackal, Lucia Loher, Hoi Lam, Josef Broder, Dan Holtmann-Rice, Nina Martin, Bramandia Ramadhana, Mrinal Shukla, Sujoy Basu, Abhi Mohan, Nick
Fernando, Noah Fiedel, Kim Paterson, Hui Li, Ankush Garg, Jane Park, DongHyun Choi, Diane Wu, Sankalp Singh, Zhishuai Zhang, Amir Globerson, Lily Yu, John Carpenter, Félix de Chaumont Quitry, Carey Radebaugh, Chu-Cheng Lin, Alex Tudor, Prakash Shroff, Drew Garmon, Dayou Du, Neera Vats, Han Lu, Shariq Iqbal, Alex Yakubovich, Nilesh Tripuraneni, James Manyika, Haroon Qureshi, Nan Hua, Christel Ngani, Maria Abi Raad, Hannah Forbes, Jeff Stanway, Mukund Sundararajan, Victor Ungureanu, Colton Bishop, Yunjie Li, Balaji Venkatraman, Bo Li, Chloe Thornton, Salvatore Scellato, Nishesh Gupta, Yicheng Wang, Ian Tenney, Xihui Wu, Ashish Shenoy, Gabriel Carvajal, Diana Gage Wright, Ben Bariach, Zhuyun Xiao, Peter Hawkins, Sid Dalmia, Clement Farabet, Pedro Valenzuela, Quan Yuan, Ananth Agarwal, Mia Chen, Wooyeol Kim, Brice Hulse, Nandita Dukkipati, Adam Paszke, Andrew Bolt, Kiam Choo, Jennifer Beattie, Jennifer Prendki, Harsha Vashisht, Rebeca Santamaria-Fernandez, Luis C. Cobo, Jarek Wilkiewicz, David Madras, Ali
Elqursh, Grant Uy, Kevin Ramirez, Matt Harvey, Tyler Liechty, Heiga Zen, Jeff Seibert, Clara Huiyi Hu, Andrey Khorlin, Maigo Le, Asaf Aharoni, Megan Li, Lily Wang, Sandeep Kumar, Norman Casagrande, Jay Hoover, Dalia El Badawy, David Soergel, Denis Vnukov, Matt Miecnikowski, Jiri Simsa, Praveen Kumar, Thibault Sellam, Daniel Vlasic, Samira Daruki, Nir Shabat, John Zhang, Guolong Su, Jiageng Zhang, Jeremiah Liu, Yi Sun, Evan Palmer, Alireza Ghaffarkhah, Xi Xiong, Victor Cotruta, Michael Fink, Lucas Dixon, Ashwin Sreevatsa, Adrian Goedeckemeyer, Alek Dimitriev, Mohsen Jafari, Remi Crocker, Nicholas FitzGerald, Aviral Kumar, Sanjay Ghemawat, Ivan Philips, Frederick Liu, Yannie Liang, Rachel Sterneck, Alena Repina, Marcus Wu, Laura Knight, Marin Georgiev, Hyo Lee, Harry Askham, Abhishek Chakladar, Annie Louis, Carl Crous, Hardie Cate, Dessie Petrova, Michael Quinn, Denese Owusu-Afriyie, Achintya Singhal, Nan Wei, Solomon Kim, Damien Vincent, Milad Nasr, Christopher A. Choquette-Choo, Reiko Tojo, Shawn Lu, Diego
de Las Casas, Yuchung Cheng, Tolga Bolukbasi, Katherine Lee, Saaber Fatehi, Rajagopal Ananthanarayanan, Miteyan Patel, Charbel Kaed, Jing Li, Shreyas Rammohan Belle, Zhe Chen, Jaclyn Konzelmann, Siim Põder, Roopal Garg, Vinod Koverkathu, Adam Brown, Chris Dyer, Rosanne Liu, Azade Nova, Jun Xu, Alanna Walton, Alicia Parrish, Mark Epstein, Sara McCarthy, Slav Petrov, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, and Oriol Vinyals.

</span>
<span class="ltx_bibblock">Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[40]</span>
<span class="ltx_bibblock">
Mistral AI team.

</span>
<span class="ltx_bibblock">Au large.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://mistral.ai/news/mistral-large/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://mistral.ai/news/mistral-large/</a>, 2024.

</span>
<span class="ltx_bibblock">(Accessed on 06/05/2024).

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[41]</span>
<span class="ltx_bibblock">
Qwen Team.

</span>
<span class="ltx_bibblock">Introducing qwen1.5, February 2024.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[42]</span>
<span class="ltx_bibblock">
The Mosaic Research Team.

</span>
<span class="ltx_bibblock">Introducing dbrx: A new state-of-the-art open llm.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm/</a>, 2024.

</span>
<span class="ltx_bibblock">(Accessed on 06/05/2024).

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[43]</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><span id="bib.bib43.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2307.09288</span>, 2023.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[44]</span>
<span class="ltx_bibblock">
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis.

</span>
<span class="ltx_bibblock">Replacing judges with juries: Evaluating llm generations with a panel of diverse models.

</span>
<span class="ltx_bibblock"><span id="bib.bib44.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2404.18796</span>, 2024.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[45]</span>
<span class="ltx_bibblock">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.

</span>
<span class="ltx_bibblock">GLUE: A multi-task benchmark and analysis platform for natural language understanding.

</span>
<span class="ltx_bibblock">2019.

</span>
<span class="ltx_bibblock">In the Proceedings of ICLR.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[46]</span>
<span class="ltx_bibblock">
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Self-instruct: Aligning language models with self-generated instructions.

</span>
<span class="ltx_bibblock">In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, <span id="bib.bib46.1.1" class="ltx_text ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</span>, pages 13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[47]</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.

</span>
<span class="ltx_bibblock">Chain-of-thought prompting elicits reasoning in large language models, 2023.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[48]</span>
<span class="ltx_bibblock">
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica.

</span>
<span class="ltx_bibblock">Rethinking benchmark and contamination for language models with rephrased samples, 2023.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[49]</span>
<span class="ltx_bibblock">
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.

</span>
<span class="ltx_bibblock">Hellaswag: Can a machine really finish your sentence?

</span>
<span class="ltx_bibblock">In <span id="bib.bib49.1.1" class="ltx_text ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics</span>, pages 4791–4800, 2019.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[50]</span>
<span class="ltx_bibblock">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.

</span>
<span class="ltx_bibblock">Judging LLM-as-a-judge with MT-bench and chatbot arena.

</span>
<span class="ltx_bibblock">In <span id="bib.bib50.1.1" class="ltx_text ltx_font_italic">Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track</span>, 2023.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[51]</span>
<span class="ltx_bibblock">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.

</span>
<span class="ltx_bibblock">Judging llm-as-a-judge with mt-bench and chatbot arena.

</span>
<span class="ltx_bibblock"><span id="bib.bib51.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, 2023.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[52]</span>
<span class="ltx_bibblock">
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan.

</span>
<span class="ltx_bibblock">Agieval: A human-centric benchmark for evaluating foundation models.

</span>
<span class="ltx_bibblock"><span id="bib.bib52.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2304.06364</span>, 2023.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[53]</span>
<span class="ltx_bibblock">
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou.

</span>
<span class="ltx_bibblock">Instruction-following evaluation for large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib53.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2311.07911</span>, 2023.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[54]</span>
<span class="ltx_bibblock">
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao.

</span>
<span class="ltx_bibblock">Starling-7b: Improving llm helpfulness &amp; harmlessness with rlaif, November 2023.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="S10" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">10 </span>Appendix</h2>

<figure id="S10.T6" class="ltx_table">
<table id="S10.T6.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S10.T6.1.1.1" class="ltx_tr">
<th id="S10.T6.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S10.T6.1.1.1.1.1" class="ltx_text ltx_font_bold">Judge</span></th>
<th id="S10.T6.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S10.T6.1.1.1.2.1" class="ltx_text ltx_font_bold">Model Name</span></th>
<th id="S10.T6.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S10.T6.1.1.1.3.1" class="ltx_text ltx_font_bold">Win Rate</span></th>
<th id="S10.T6.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S10.T6.1.1.1.4.1" class="ltx_text ltx_font_bold">Average Token #</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S10.T6.1.2.1" class="ltx_tr">
<td id="S10.T6.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t">gpt-4-turbo-2024-04-09*</td>
<td id="S10.T6.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">82.6</td>
<td id="S10.T6.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">-1.8/+1.6</td>
<td id="S10.T6.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">662</td>
</tr>
<tr id="S10.T6.1.3.2" class="ltx_tr">
<td id="S10.T6.1.3.2.1" class="ltx_td ltx_align_left">gpt-4-0125-preview*</td>
<td id="S10.T6.1.3.2.2" class="ltx_td ltx_align_center">78.0</td>
<td id="S10.T6.1.3.2.3" class="ltx_td ltx_align_center">-2.2/+2.4</td>
<td id="S10.T6.1.3.2.4" class="ltx_td ltx_align_center">619</td>
</tr>
<tr id="S10.T6.1.4.3" class="ltx_tr">
<td id="S10.T6.1.4.3.1" class="ltx_td ltx_align_left">claude-3-opus-20240229</td>
<td id="S10.T6.1.4.3.2" class="ltx_td ltx_align_center">60.4</td>
<td id="S10.T6.1.4.3.3" class="ltx_td ltx_align_center">-3.3/+2.4</td>
<td id="S10.T6.1.4.3.4" class="ltx_td ltx_align_center">541</td>
</tr>
<tr id="S10.T6.1.5.4" class="ltx_tr">
<td id="S10.T6.1.5.4.1" class="ltx_td ltx_align_left">gpt-4-0314</td>
<td id="S10.T6.1.5.4.2" class="ltx_td ltx_align_center">50.0</td>
<td id="S10.T6.1.5.4.3" class="ltx_td ltx_align_center">-0.0/+0.0</td>
<td id="S10.T6.1.5.4.4" class="ltx_td ltx_align_center">423</td>
</tr>
<tr id="S10.T6.1.6.5" class="ltx_tr">
<td id="S10.T6.1.6.5.1" class="ltx_td ltx_align_left">claude-3-sonnet-20240229</td>
<td id="S10.T6.1.6.5.2" class="ltx_td ltx_align_center">46.8</td>
<td id="S10.T6.1.6.5.3" class="ltx_td ltx_align_center">-2.1/+2.2</td>
<td id="S10.T6.1.6.5.4" class="ltx_td ltx_align_center">552</td>
</tr>
<tr id="S10.T6.1.7.6" class="ltx_tr">
<td id="S10.T6.1.7.6.1" class="ltx_td ltx_align_left">claude-3-haiku-20240307</td>
<td id="S10.T6.1.7.6.2" class="ltx_td ltx_align_center">41.5</td>
<td id="S10.T6.1.7.6.3" class="ltx_td ltx_align_center">-2.8/+2.5</td>
<td id="S10.T6.1.7.6.4" class="ltx_td ltx_align_center">505</td>
</tr>
<tr id="S10.T6.1.8.7" class="ltx_tr">
<td id="S10.T6.1.8.7.1" class="ltx_td ltx_align_left">llama-3-70b-instruct</td>
<td id="S10.T6.1.8.7.2" class="ltx_td ltx_align_center">41.1</td>
<td id="S10.T6.1.8.7.3" class="ltx_td ltx_align_center">-2.5/+2.4</td>
<td id="S10.T6.1.8.7.4" class="ltx_td ltx_align_center">583</td>
</tr>
<tr id="S10.T6.1.9.8" class="ltx_tr">
<td id="S10.T6.1.9.8.1" class="ltx_td ltx_align_left">gpt-4-0613</td>
<td id="S10.T6.1.9.8.2" class="ltx_td ltx_align_center">37.9</td>
<td id="S10.T6.1.9.8.3" class="ltx_td ltx_align_center">-2.2/+2.0</td>
<td id="S10.T6.1.9.8.4" class="ltx_td ltx_align_center">354</td>
</tr>
<tr id="S10.T6.1.10.9" class="ltx_tr">
<td id="S10.T6.1.10.9.1" class="ltx_td ltx_align_left">mistral-large-2402</td>
<td id="S10.T6.1.10.9.2" class="ltx_td ltx_align_center">37.7</td>
<td id="S10.T6.1.10.9.3" class="ltx_td ltx_align_center">-1.9/+2.6</td>
<td id="S10.T6.1.10.9.4" class="ltx_td ltx_align_center">400</td>
</tr>
<tr id="S10.T6.1.11.10" class="ltx_tr">
<td id="S10.T6.1.11.10.1" class="ltx_td ltx_align_left">mixtral-8x22b-instruct-v0.1</td>
<td id="S10.T6.1.11.10.2" class="ltx_td ltx_align_center">36.4</td>
<td id="S10.T6.1.11.10.3" class="ltx_td ltx_align_center">-2.7/+2.9</td>
<td id="S10.T6.1.11.10.4" class="ltx_td ltx_align_center">430</td>
</tr>
<tr id="S10.T6.1.12.11" class="ltx_tr">
<td id="S10.T6.1.12.11.1" class="ltx_td ltx_align_left">Qwen1.5-72B-Chat</td>
<td id="S10.T6.1.12.11.2" class="ltx_td ltx_align_center">36.1</td>
<td id="S10.T6.1.12.11.3" class="ltx_td ltx_align_center">-2.5/+2.2</td>
<td id="S10.T6.1.12.11.4" class="ltx_td ltx_align_center">474</td>
</tr>
<tr id="S10.T6.1.13.12" class="ltx_tr">
<td id="S10.T6.1.13.12.1" class="ltx_td ltx_align_left">command-r-plus</td>
<td id="S10.T6.1.13.12.2" class="ltx_td ltx_align_center">33.1</td>
<td id="S10.T6.1.13.12.3" class="ltx_td ltx_align_center">-2.1/+2.2</td>
<td id="S10.T6.1.13.12.4" class="ltx_td ltx_align_center">541</td>
</tr>
<tr id="S10.T6.1.14.13" class="ltx_tr">
<td id="S10.T6.1.14.13.1" class="ltx_td ltx_align_left">mistral-medium</td>
<td id="S10.T6.1.14.13.2" class="ltx_td ltx_align_center">31.9</td>
<td id="S10.T6.1.14.13.3" class="ltx_td ltx_align_center">-2.3/+2.4</td>
<td id="S10.T6.1.14.13.4" class="ltx_td ltx_align_center">485</td>
</tr>
<tr id="S10.T6.1.15.14" class="ltx_tr">
<td id="S10.T6.1.15.14.1" class="ltx_td ltx_align_left">mistral-next</td>
<td id="S10.T6.1.15.14.2" class="ltx_td ltx_align_center">27.4</td>
<td id="S10.T6.1.15.14.3" class="ltx_td ltx_align_center">-2.1/+1.7</td>
<td id="S10.T6.1.15.14.4" class="ltx_td ltx_align_center">297</td>
</tr>
<tr id="S10.T6.1.16.15" class="ltx_tr">
<td id="S10.T6.1.16.15.1" class="ltx_td ltx_align_left">gpt-3.5-turbo-0613</td>
<td id="S10.T6.1.16.15.2" class="ltx_td ltx_align_center">24.8</td>
<td id="S10.T6.1.16.15.3" class="ltx_td ltx_align_center">-1.6/+2.0</td>
<td id="S10.T6.1.16.15.4" class="ltx_td ltx_align_center">401</td>
</tr>
<tr id="S10.T6.1.17.16" class="ltx_tr">
<td id="S10.T6.1.17.16.1" class="ltx_td ltx_align_left">claude-2.0</td>
<td id="S10.T6.1.17.16.2" class="ltx_td ltx_align_center">24.0</td>
<td id="S10.T6.1.17.16.3" class="ltx_td ltx_align_center">-2.5/+2.5</td>
<td id="S10.T6.1.17.16.4" class="ltx_td ltx_align_center">295</td>
</tr>
<tr id="S10.T6.1.18.17" class="ltx_tr">
<td id="S10.T6.1.18.17.1" class="ltx_td ltx_align_left">dbrx-instruct</td>
<td id="S10.T6.1.18.17.2" class="ltx_td ltx_align_center">23.9</td>
<td id="S10.T6.1.18.17.3" class="ltx_td ltx_align_center">-1.4/+1.5</td>
<td id="S10.T6.1.18.17.4" class="ltx_td ltx_align_center">415</td>
</tr>
<tr id="S10.T6.1.19.18" class="ltx_tr">
<td id="S10.T6.1.19.18.1" class="ltx_td ltx_align_left">Mixtral-8x7B-Instruct-v0.1</td>
<td id="S10.T6.1.19.18.2" class="ltx_td ltx_align_center">23.4</td>
<td id="S10.T6.1.19.18.3" class="ltx_td ltx_align_center">-2.3/+1.7</td>
<td id="S10.T6.1.19.18.4" class="ltx_td ltx_align_center">457</td>
</tr>
<tr id="S10.T6.1.20.19" class="ltx_tr">
<td id="S10.T6.1.20.19.1" class="ltx_td ltx_align_left">gpt-3.5-turbo-0125</td>
<td id="S10.T6.1.20.19.2" class="ltx_td ltx_align_center">23.3</td>
<td id="S10.T6.1.20.19.3" class="ltx_td ltx_align_center">-2.2/+2.3</td>
<td id="S10.T6.1.20.19.4" class="ltx_td ltx_align_center">329</td>
</tr>
<tr id="S10.T6.1.21.20" class="ltx_tr">
<td id="S10.T6.1.21.20.1" class="ltx_td ltx_align_left">Yi-34B-Chat</td>
<td id="S10.T6.1.21.20.2" class="ltx_td ltx_align_center">23.1</td>
<td id="S10.T6.1.21.20.3" class="ltx_td ltx_align_center">-1.8/+2.0</td>
<td id="S10.T6.1.21.20.4" class="ltx_td ltx_align_center">611</td>
</tr>
<tr id="S10.T6.1.22.21" class="ltx_tr">
<td id="S10.T6.1.22.21.1" class="ltx_td ltx_align_left">Starling-LM-7B-beta</td>
<td id="S10.T6.1.22.21.2" class="ltx_td ltx_align_center">23.0</td>
<td id="S10.T6.1.22.21.3" class="ltx_td ltx_align_center">-1.9/+2.2</td>
<td id="S10.T6.1.22.21.4" class="ltx_td ltx_align_center">530</td>
</tr>
<tr id="S10.T6.1.23.22" class="ltx_tr">
<td id="S10.T6.1.23.22.1" class="ltx_td ltx_align_left">claude-2.1</td>
<td id="S10.T6.1.23.22.2" class="ltx_td ltx_align_center">22.8</td>
<td id="S10.T6.1.23.22.3" class="ltx_td ltx_align_center">-1.6/+2.1</td>
<td id="S10.T6.1.23.22.4" class="ltx_td ltx_align_center">290</td>
</tr>
<tr id="S10.T6.1.24.23" class="ltx_tr">
<td id="S10.T6.1.24.23.1" class="ltx_td ltx_align_left">Snorkel-Mistral-PairRM-DPO</td>
<td id="S10.T6.1.24.23.2" class="ltx_td ltx_align_center">20.7</td>
<td id="S10.T6.1.24.23.3" class="ltx_td ltx_align_center">-2.2/+1.5</td>
<td id="S10.T6.1.24.23.4" class="ltx_td ltx_align_center">564</td>
</tr>
<tr id="S10.T6.1.25.24" class="ltx_tr">
<td id="S10.T6.1.25.24.1" class="ltx_td ltx_align_left">llama-3-8b-instruct</td>
<td id="S10.T6.1.25.24.2" class="ltx_td ltx_align_center">20.6</td>
<td id="S10.T6.1.25.24.3" class="ltx_td ltx_align_center">-2.5/+1.8</td>
<td id="S10.T6.1.25.24.4" class="ltx_td ltx_align_center">585</td>
</tr>
<tr id="S10.T6.1.26.25" class="ltx_tr">
<td id="S10.T6.1.26.25.1" class="ltx_td ltx_align_left">gpt-3.5-turbo-1106</td>
<td id="S10.T6.1.26.25.2" class="ltx_td ltx_align_center">18.9</td>
<td id="S10.T6.1.26.25.3" class="ltx_td ltx_align_center">-1.6/+2.1</td>
<td id="S10.T6.1.26.25.4" class="ltx_td ltx_align_center">285</td>
</tr>
<tr id="S10.T6.1.27.26" class="ltx_tr">
<td id="S10.T6.1.27.26.1" class="ltx_td ltx_align_left">gpt-3.5-turbo-0301</td>
<td id="S10.T6.1.27.26.2" class="ltx_td ltx_align_center">18.1</td>
<td id="S10.T6.1.27.26.3" class="ltx_td ltx_align_center">-1.7/+1.2</td>
<td id="S10.T6.1.27.26.4" class="ltx_td ltx_align_center">334</td>
</tr>
<tr id="S10.T6.1.28.27" class="ltx_tr">
<td id="S10.T6.1.28.27.1" class="ltx_td ltx_align_left">gemini-1.0-pro</td>
<td id="S10.T6.1.28.27.2" class="ltx_td ltx_align_center">17.8</td>
<td id="S10.T6.1.28.27.3" class="ltx_td ltx_align_center">-1.7/+1.7</td>
<td id="S10.T6.1.28.27.4" class="ltx_td ltx_align_center">322</td>
</tr>
<tr id="S10.T6.1.29.28" class="ltx_tr">
<td id="S10.T6.1.29.28.1" class="ltx_td ltx_align_left">command-r</td>
<td id="S10.T6.1.29.28.2" class="ltx_td ltx_align_center">17.0</td>
<td id="S10.T6.1.29.28.3" class="ltx_td ltx_align_center">-1.9/+1.7</td>
<td id="S10.T6.1.29.28.4" class="ltx_td ltx_align_center">432</td>
</tr>
<tr id="S10.T6.1.30.29" class="ltx_tr">
<td id="S10.T6.1.30.29.1" class="ltx_td ltx_align_left">tulu-2-dpo-70b</td>
<td id="S10.T6.1.30.29.2" class="ltx_td ltx_align_center">15.0</td>
<td id="S10.T6.1.30.29.3" class="ltx_td ltx_align_center">-1.4/+1.2</td>
<td id="S10.T6.1.30.29.4" class="ltx_td ltx_align_center">550</td>
</tr>
<tr id="S10.T6.1.31.30" class="ltx_tr">
<td id="S10.T6.1.31.30.1" class="ltx_td ltx_align_left">Starling-LM-7B-alpha</td>
<td id="S10.T6.1.31.30.2" class="ltx_td ltx_align_center">12.8</td>
<td id="S10.T6.1.31.30.3" class="ltx_td ltx_align_center">-1.4/+1.4</td>
<td id="S10.T6.1.31.30.4" class="ltx_td ltx_align_center">483</td>
</tr>
<tr id="S10.T6.1.32.31" class="ltx_tr">
<td id="S10.T6.1.32.31.1" class="ltx_td ltx_align_left">mistral-7b-instruct-v0.2</td>
<td id="S10.T6.1.32.31.2" class="ltx_td ltx_align_center">12.6</td>
<td id="S10.T6.1.32.31.3" class="ltx_td ltx_align_center">-1.6/+1.3</td>
<td id="S10.T6.1.32.31.4" class="ltx_td ltx_align_center">541</td>
</tr>
<tr id="S10.T6.1.33.32" class="ltx_tr">
<td id="S10.T6.1.33.32.1" class="ltx_td ltx_align_left">Llama-2-70b-chat-hf</td>
<td id="S10.T6.1.33.32.2" class="ltx_td ltx_align_center">11.6</td>
<td id="S10.T6.1.33.32.3" class="ltx_td ltx_align_center">-1.6/+1.4</td>
<td id="S10.T6.1.33.32.4" class="ltx_td ltx_align_center">595</td>
</tr>
<tr id="S10.T6.1.34.33" class="ltx_tr">
<td id="S10.T6.1.34.33.1" class="ltx_td ltx_align_left">vicuna-33b-v1.3</td>
<td id="S10.T6.1.34.33.2" class="ltx_td ltx_align_center">8.6</td>
<td id="S10.T6.1.34.33.3" class="ltx_td ltx_align_center">-1.3/+1.0</td>
<td id="S10.T6.1.34.33.4" class="ltx_td ltx_align_center">451</td>
</tr>
<tr id="S10.T6.1.35.34" class="ltx_tr">
<td id="S10.T6.1.35.34.1" class="ltx_td ltx_align_left">gemma-7b-it</td>
<td id="S10.T6.1.35.34.2" class="ltx_td ltx_align_center">7.5</td>
<td id="S10.T6.1.35.34.3" class="ltx_td ltx_align_center">-1.1/+1.2</td>
<td id="S10.T6.1.35.34.4" class="ltx_td ltx_align_center">378</td>
</tr>
<tr id="S10.T6.1.36.35" class="ltx_tr">
<td id="S10.T6.1.36.35.1" class="ltx_td ltx_align_left">Llama-2-7b-chat-hf</td>
<td id="S10.T6.1.36.35.2" class="ltx_td ltx_align_center">4.6</td>
<td id="S10.T6.1.36.35.3" class="ltx_td ltx_align_center">-0.8/+0.8</td>
<td id="S10.T6.1.36.35.4" class="ltx_td ltx_align_center">561</td>
</tr>
<tr id="S10.T6.1.37.36" class="ltx_tr">
<td id="S10.T6.1.37.36.1" class="ltx_td ltx_align_left">gemma-2b-it</td>
<td id="S10.T6.1.37.36.2" class="ltx_td ltx_align_center">3.0</td>
<td id="S10.T6.1.37.36.3" class="ltx_td ltx_align_center">-0.6/+0.7</td>
<td id="S10.T6.1.37.36.4" class="ltx_td ltx_align_center">369</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Arena-Hard-Auto v0.1 Leaderboard (baseline: GPT-4-0314). *GPT-4-Turbo’s high score can be due to the GPT-4 judge favoring GPT-4 outputs.</figcaption>
</figure>
<figure id="S10.T7" class="ltx_table">
<table id="S10.T7.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S10.T7.1.1.1" class="ltx_tr">
<th id="S10.T7.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S10.T7.1.1.1.1.1" class="ltx_text ltx_font_bold">Judge</span></th>
<th id="S10.T7.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S10.T7.1.1.1.2.1" class="ltx_text ltx_font_bold">Model Name</span></th>
<th id="S10.T7.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S10.T7.1.1.1.3.1" class="ltx_text ltx_font_bold">Win Rate</span></th>
<th id="S10.T7.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S10.T7.1.1.1.4.1" class="ltx_text ltx_font_bold">Average Token #</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S10.T7.1.2.1" class="ltx_tr">
<td id="S10.T7.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S10.T7.1.2.1.1.1" class="ltx_text ltx_font_bold">GPT-4-1106-Preview</span></td>
<td id="S10.T7.1.2.1.2" class="ltx_td ltx_align_left ltx_border_t">gpt-3.5-turbo-0125-detailed</td>
<td id="S10.T7.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">29.86</td>
<td id="S10.T7.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">421</td>
</tr>
<tr id="S10.T7.1.3.2" class="ltx_tr">
<td id="S10.T7.1.3.2.1" class="ltx_td"></td>
<td id="S10.T7.1.3.2.2" class="ltx_td ltx_align_left">gpt-3.5-turbo-0125-chatty</td>
<td id="S10.T7.1.3.2.3" class="ltx_td ltx_align_center">23.89</td>
<td id="S10.T7.1.3.2.4" class="ltx_td ltx_align_center">361</td>
</tr>
<tr id="S10.T7.1.4.3" class="ltx_tr">
<td id="S10.T7.1.4.3.1" class="ltx_td"></td>
<td id="S10.T7.1.4.3.2" class="ltx_td ltx_align_left">gpt-3.5-turbo-0125</td>
<td id="S10.T7.1.4.3.3" class="ltx_td ltx_align_center">23.2</td>
<td id="S10.T7.1.4.3.4" class="ltx_td ltx_align_center">328</td>
</tr>
<tr id="S10.T7.1.5.4" class="ltx_tr">
<td id="S10.T7.1.5.4.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S10.T7.1.5.4.1.1" class="ltx_text ltx_font_bold">Claude-3-Opus</span></td>
<td id="S10.T7.1.5.4.2" class="ltx_td ltx_align_left ltx_border_t">gpt-3.5-turbo-0125-detailed</td>
<td id="S10.T7.1.5.4.3" class="ltx_td ltx_align_center ltx_border_t">40.78</td>
<td id="S10.T7.1.5.4.4" class="ltx_td ltx_align_center ltx_border_t">421</td>
</tr>
<tr id="S10.T7.1.6.5" class="ltx_tr">
<td id="S10.T7.1.6.5.1" class="ltx_td"></td>
<td id="S10.T7.1.6.5.2" class="ltx_td ltx_align_left">gpt-3.5-turbo-0125-chatty</td>
<td id="S10.T7.1.6.5.3" class="ltx_td ltx_align_center">28.49</td>
<td id="S10.T7.1.6.5.4" class="ltx_td ltx_align_center">375</td>
</tr>
<tr id="S10.T7.1.7.6" class="ltx_tr">
<td id="S10.T7.1.7.6.1" class="ltx_td ltx_border_bb"></td>
<td id="S10.T7.1.7.6.2" class="ltx_td ltx_align_left ltx_border_bb">gpt-3.5-turbo-0125</td>
<td id="S10.T7.1.7.6.3" class="ltx_td ltx_align_center ltx_border_bb">27.97</td>
<td id="S10.T7.1.7.6.4" class="ltx_td ltx_align_center ltx_border_bb">328</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>Length Bias Comparison Between GPT and Claude as Judge. Prompt for detailed:“You are a helpful assistant who thoroughly explains things with as much detail as possible.”, prompt for chatty: ‘You are a helpful assistant who is chatty.”, and default prompt: "You are a helpful assistant."</figcaption>
</figure>
<figure id="S10.T8" class="ltx_table">
<table id="S10.T8.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S10.T8.1.1.1" class="ltx_tr">
<th id="S10.T8.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S10.T8.1.1.1.1.1" class="ltx_text ltx_font_bold">Model Name</span></th>
<th id="S10.T8.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S10.T8.1.1.1.2.1" class="ltx_text ltx_font_bold">Win Rate</span></th>
<th id="S10.T8.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S10.T8.1.1.1.3.1" class="ltx_text ltx_font_bold">Average Token #</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S10.T8.1.2.1" class="ltx_tr">
<td id="S10.T8.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t">gpt-3.5-turbo-0125-1</td>
<td id="S10.T8.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">23.05</td>
<td id="S10.T8.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">328</td>
</tr>
<tr id="S10.T8.1.3.2" class="ltx_tr">
<td id="S10.T8.1.3.2.1" class="ltx_td ltx_align_left">gpt-3.5-turbo-0125-2</td>
<td id="S10.T8.1.3.2.2" class="ltx_td ltx_align_center">22.93</td>
<td id="S10.T8.1.3.2.3" class="ltx_td ltx_align_center">328</td>
</tr>
<tr id="S10.T8.1.4.3" class="ltx_tr">
<td id="S10.T8.1.4.3.1" class="ltx_td ltx_align_left ltx_border_bb">gpt-3.5-turbo-0125-3</td>
<td id="S10.T8.1.4.3.2" class="ltx_td ltx_align_center ltx_border_bb">22.75</td>
<td id="S10.T8.1.4.3.3" class="ltx_td ltx_align_center ltx_border_bb">328</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 8: </span>Variances between 3 separate runs of Arena-Hard-Auto v0.1 on gpt-3.5-turbo’s performance using GPT-4-1106-Preview as judge.</figcaption>
</figure>
<figure id="S10.F5" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S10.F5.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.11939/assets/x5.png" id="S10.F5.sf1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="346" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">(a) </span>Chatbot Arena (20K Votes)</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S10.F5.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.11939/assets/x6.png" id="S10.F5.sf2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="346" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">(b) </span>AlpacaEval 2.0 LC</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S10.F5.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.11939/assets/x7.png" id="S10.F5.sf3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="346" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">(c) </span>Arena-Hard-Auto v0.1</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S10.F5.sf4" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.11939/assets/x8.png" id="S10.F5.sf4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="346" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">(d) </span>MT Bench</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Above is the predicted model predicted probability against the bootstrapped arena “ground truth” probability (jittered to show clusters). While both Alpaca eval and Arena-Hard-Auto v0.1 have large clusters around (0,0) and (1,1) signifying good forecasting, Arena-Hard-Auto v0.1 has lighter clusters on (0,1) and (1,0), if any, revealing less overconfidence. MT Bench has heavy tails along the top and bottom, revealing underconfidence. However, none of these benchmarks show an “ideal” y=x curve (with dense ends) expected with a perfectly calibrated forecast, signifying room for future research.</figcaption>
</figure>
<figure id="S10.F6" class="ltx_figure"><img src="/html/2406.11939/assets/x9.png" id="S10.F6.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="461" height="692" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>A more complete selection of mean scores of various topic clusters in descending order.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="S11" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">11 </span>Prompts</h2>

<div id="S11.p1" class="ltx_para ltx_noindent">
<svg id="S11.p1.pic1" class="ltx_picture" height="457.71" overflow="visible" version="1.1" width="603.54"><g transform="translate(0,457.71) matrix(1 0 0 -1 0 0) translate(0,3.54)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g stroke-opacity="0.5" fill="#F2F2F2" fill-opacity="0.5" stroke="#F2F2F2" color="#F2F2F2"><path d="M 3.54 4.13 L 3.54 442.95 C 3.54 447.19 6.98 450.63 11.22 450.63 L 595.87 450.63 C 600.11 450.63 603.54 447.19 603.54 442.95 L 603.54 4.13 C 603.54 -0.11 600.11 -3.54 595.87 -3.54 L 11.22 -3.54 C 6.98 -3.54 3.54 -0.11 3.54 4.13 Z" style="stroke:none"></path></g><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 448.26 C 0 451.53 2.64 454.17 5.91 454.17 L 594.09 454.17 C 597.36 454.17 600 451.53 600 448.26 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 437.93 L 598.03 437.93 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 17.72 439.9)"><foreignObject width="564.57" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S11.p1.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:408.0pt;">
<span id="S11.p1.pic1.1.1.1.1.1.1" class="ltx_p">Prompt Quality Systems Instruction:</span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 17.72 13.78)"><foreignObject width="564.57" height="412.34" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S11.p1.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:408.0pt;">
<span id="S11.p1.pic1.2.2.2.1.1.1" class="ltx_p">Your task is to evaluate how well the following input prompts can assess the capabilities of advanced AI assistants. For the input prompt, please analyze it based on the following 7 criteria. For each criteria, make sure to explain before determine whether the input satisfy it.</span>
<span id="S11.p1.pic1.2.2.2.1.1.2" class="ltx_p">1. Specificity: Does the prompt ask for a specific, well-defined output without leaving any ambiguity? This allows the AI to demonstrate its ability to follow instructions and generate a precise, targeted response.</span>
<span id="S11.p1.pic1.2.2.2.1.1.3" class="ltx_p">2. Domain Knowledge: Does the prompt test the AI’s knowledge and understanding in a specific domain or set of domains? The prompt must demand the AI to have a strong prior knowledge or mastery of domain-specific concepts, theories, or principles.</span>
<span id="S11.p1.pic1.2.2.2.1.1.4" class="ltx_p">3. Complexity: Does the prompt have multiple components, variables, or levels of depth and nuance? This assesses the AI’s capability to handle complex, multi-faceted problems beyond simple queries.</span>
<span id="S11.p1.pic1.2.2.2.1.1.5" class="ltx_p">4. Problem-Solving: Does the prompt require active problem-solving: analyzing and clearly defining the problem and systematically devising and implementing a solution? Note active problem-solving is not simply reciting facts or following a fixed set of instructions.</span>
<span id="S11.p1.pic1.2.2.2.1.1.6" class="ltx_p">5. Creativity: Does the prompt require a creative approach or solution? This tests the AI’s ability to generate novel ideas tailored to the specific needs of the request or problem at hand.</span>
<span id="S11.p1.pic1.2.2.2.1.1.7" class="ltx_p">6. Technical Accuracy: Does the prompt require an answer with a high degree of technical accuracy, correctness and precision? This assesses the reliability and truthfulness of the AI’s outputs.</span>
<span id="S11.p1.pic1.2.2.2.1.1.8" class="ltx_p">7. Real-World Application: Does the prompt relate to real-world applications? This tests the AI’s ability to provide practical and actionable information that could be implemented in real-life scenarios.</span>
<span id="S11.p1.pic1.2.2.2.1.1.9" class="ltx_p">After analyzing the input prompt based on these criteria, you must list the criteria numbers that the prompt satisfies in the format of a Python array. For example, "Criteria Satisfied: [1, 2, 4, 6, 7]".</span>
</span></foreignObject></g></g></svg>
</div>
<div id="S11.p2" class="ltx_para ltx_noindent">
<svg id="S11.p2.pic1" class="ltx_picture" height="356.7" overflow="visible" version="1.1" width="603.54"><g transform="translate(0,356.7) matrix(1 0 0 -1 0 0) translate(0,3.54)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g stroke-opacity="0.5" fill="#F2F2F2" fill-opacity="0.5" stroke="#F2F2F2" color="#F2F2F2"><path d="M 3.54 4.13 L 3.54 341.94 C 3.54 346.18 6.98 349.62 11.22 349.62 L 595.87 349.62 C 600.11 349.62 603.54 346.18 603.54 341.94 L 603.54 4.13 C 603.54 -0.11 600.11 -3.54 595.87 -3.54 L 11.22 -3.54 C 6.98 -3.54 3.54 -0.11 3.54 4.13 Z" style="stroke:none"></path></g><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 347.25 C 0 350.52 2.64 353.16 5.91 353.16 L 594.09 353.16 C 597.36 353.16 600 350.52 600 347.25 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 336.92 L 598.03 336.92 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 17.72 338.89)"><foreignObject width="564.57" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S11.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:408.0pt;">
<span id="S11.p2.pic1.1.1.1.1.1.1" class="ltx_p">LLM-Judge System Instruction</span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 17.72 13.78)"><foreignObject width="564.57" height="311.33" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S11.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:408.0pt;">
<span id="S11.p2.pic1.2.2.2.1.1.1" class="ltx_p">Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user prompt displayed below. You will be given assistant A’s answer and assistant B’s answer. Your job is to evaluate which assistant’s answer is better.</span>
<span id="S11.p2.pic1.2.2.2.1.1.2" class="ltx_p">Begin your evaluation by generating your own answer to the prompt. You must provide your answers before judging any answers.</span>
<span id="S11.p2.pic1.2.2.2.1.1.3" class="ltx_p">When evaluating the assistants’ answers, compare both assistants’ answers with your answer. You must identify and correct any mistakes or inaccurate information.</span>
<span id="S11.p2.pic1.2.2.2.1.1.4" class="ltx_p">Then consider if the assistant’s answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, it is more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based on assumptions. Relevant means all parts of the response closely connect or are appropriate to what is being asked. Concise means the response is clear and not verbose or excessive.</span>
<span id="S11.p2.pic1.2.2.2.1.1.5" class="ltx_p">Then consider the creativity and novelty of the assistant’s answers when needed. Finally, identify any missing important information in the assistants’ answers that would be beneficial to include when responding to the user prompt.</span>
<span id="S11.p2.pic1.2.2.2.1.1.6" class="ltx_p">After providing your explanation, you must output only one of the following choices as your final verdict with a label:</span>
<span id="S11.I1" class="ltx_enumerate">
<span id="S11.I1.i1" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">1.</span> 
<span id="S11.I1.i1.p1" class="ltx_para">
<span id="S11.I1.i1.p1.1" class="ltx_p">Assistant A is significantly better: [[A&gt;&gt;B]]</span>
</span></span>
<span id="S11.I1.i2" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">2.</span> 
<span id="S11.I1.i2.p1" class="ltx_para">
<span id="S11.I1.i2.p1.1" class="ltx_p">Assistant A is slightly better: [[A&gt;B]]</span>
</span></span>
<span id="S11.I1.i3" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">3.</span> 
<span id="S11.I1.i3.p1" class="ltx_para">
<span id="S11.I1.i3.p1.1" class="ltx_p">Tie, relatively the same: [[A=B]]</span>
</span></span>
<span id="S11.I1.i4" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">4.</span> 
<span id="S11.I1.i4.p1" class="ltx_para">
<span id="S11.I1.i4.p1.1" class="ltx_p">Assistant B is slightly better: [[B&gt;A]]</span>
</span></span>
<span id="S11.I1.i5" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">5.</span> 
<span id="S11.I1.i5.p1" class="ltx_para ltx_noindent">
<span id="S11.I1.i5.p1.1" class="ltx_p">Assistant B is significantly better: [[B&gt;&gt;A]</span>
</span></span>
</span>
<span id="S11.p2.pic1.2.2.2.1.1.7" class="ltx_p">Example output: "My final verdict is tie: [[A=B]]".</span>
</span></foreignObject></g></g></svg>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="S12" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">12 </span>Additional Examples</h2>

<figure id="S12.1" class="ltx_float ltx_float_tcbfloat">
<p id="S12.1.1" class="ltx_p">[htb] <svg id="S12.1.1.pic1" class="ltx_picture" height="642.7" overflow="visible" version="1.1" width="603.54"><g transform="translate(0,642.7) matrix(1 0 0 -1 0 0) translate(0,3.54)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g stroke-opacity="0.5" fill="#F2F2F2" fill-opacity="0.5" stroke="#F2F2F2" color="#F2F2F2"><path d="M 3.54 -3.54 L 3.54 635.61 L 603.54 635.61 L 603.54 -3.54 Z" style="stroke:none"></path></g><g fill="#FFFF99" fill-opacity="1.0"><path d="M 0 0 L 0 639.15 L 600 639.15 L 600 0 Z" style="stroke:none"></path></g><g fill="#FFFFE6" fill-opacity="1.0"><path d="M 1.97 1.97 L 1.97 637.19 L 598.03 637.19 L 598.03 1.97 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.84 13.78)"><foreignObject width="580.32" height="611.6" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S12.1.1.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:419.4pt;">
<span id="S12.1.1.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S12.1.1.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">User:</span> Use ABC notation to write a melody in the style of a folk tune.</span>
<span id="S12.1.1.pic1.1.1.1.1.1.2" class="ltx_p"><span id="S12.1.1.pic1.1.1.1.1.1.2.1" class="ltx_text ltx_font_bold">GPT-4:</span></span><pre id="S12.1.1.pic1.1.1.1.1.1.3" class="ltx_verbatim ltx_font_typewriter">
X:1 T:Untitled Folk Tune M:4/4 L:1/8
K:G|:G2A2|B2A2|G2E2|D4|E2F2|G2F2|E2C2|B,4|A2B2|c2B2|A2F2|E4|D2E2|F2E2|D2B,2|C4:|
</pre>
<span id="S12.1.1.pic1.1.1.1.1.1.4" class="ltx_p"><span id="S12.1.1.pic1.1.1.1.1.1.4.1" class="ltx_text ltx_font_bold">GPT-3.5-Turbo:</span>
Sure! Here’s a simple melody in the style of a folk tune written in ABC notation:</span><pre id="S12.1.1.pic1.1.1.1.1.1.5" class="ltx_verbatim ltx_font_typewriter">
X:1 T:Folk Tune M:4/4 L:1/4 K:C
CDE|FGA|BAG|FED| CDE|FGA|BAG|c2z| CDE|FGA|BAG|FED| CDE|FGA|BAG|c2z||
</pre>
<span id="S12.1.1.pic1.1.1.1.1.1.6" class="ltx_p">This melody is in the key of C major and consists of four measures. Feel free to modify it or add more sections to create your own folk tune!</span>
<span id="S12.1.1.pic1.1.1.1.1.1.7" class="ltx_p"><span id="S12.1.1.pic1.1.1.1.1.1.7.1" class="ltx_text ltx_font_bold">Judgement:</span> My own answer to the prompt would be:</span><pre id="S12.1.1.pic1.1.1.1.1.1.8" class="ltx_verbatim ltx_font_typewriter">
    X:1 T:My Folk Melody M:3/4 L:1/8
    K:D A2|:d2f2a2|b2a2f2|e2d2B2|A4F2| G2A2B2|A2G2F2|E2D2F2|1D6:|2D4A2|
    |:d2f2a2|b2a2f2|e2c2A2|B4A2| G2A2B2|A2G2F2|E2D2F2|1D6:|2D4||
</pre>
<span id="S12.1.1.pic1.1.1.1.1.1.9" class="ltx_p">This melody is in the key of D major, has a 3/4 time signature, and is structured in a typical AAB form common in folk music. It uses a range of an octave, which is typical for folk tunes that need to be easily singable.</span>
<span id="S12.1.1.pic1.1.1.1.1.1.10" class="ltx_p">Now, let’s evaluate the responses from Assistant A and Assistant B.</span>
<span id="S12.1.1.pic1.1.1.1.1.1.11" class="ltx_p">Assistant A’s Answer: Assistant A provides a melody in ABC notation that is coherent and follows the structure of a folk tune. The melody is in 4/4 time, which is common for folk music, and it is in the key of G major. The melody has a two-part form (indicated by the repeat signs |: and :|), which is typical in folk music. The range of the melody is appropriate for a folk tune, and the rhythm is simple and repetitive, which is characteristic of the genre.</span>
<span id="S12.1.1.pic1.1.1.1.1.1.12" class="ltx_p">Assistant B’s Answer: Assistant B also provides a melody in ABC notation that is coherent and could be considered in the style of a folk tune. The melody is in 4/4 time and the key of C major. The melody consists of four measures repeated twice, which is a simple structure that could be found in folk music. However, Assistant B’s melody is less traditional in its form compared to Assistant A’s, as it does not have a distinct A and B section, which is more common in folk tunes. Additionally, the use of “z” for a half-measure rest is less typical in folk tunes, which usually have a more continuous melodic line.</span>
<span id="S12.1.1.pic1.1.1.1.1.1.13" class="ltx_p">Comparison: Both assistants provided melodies that could be considered folk tunes. Assistant A’s melody has a more traditional structure with a clear A and B section, while Assistant B’s melody is simpler and does not follow the traditional AAB or AB form. Assistant A’s melody is also more detailed in terms of the ABC notation, providing a full eight-measure section that is repeated, which gives a more complete sense of the tune. Assistant B’s melody is correct in its notation but is more basic and less representative of a traditional folk tune structure.</span>
<span id="S12.1.1.pic1.1.1.1.1.1.14" class="ltx_p">In terms of helpfulness, relevance, and conciseness, both assistants provided relevant and concise answers. However, Assistant A’s answer is slightly more helpful due to the more traditional structure of the provided melody.</span>
<span id="S12.1.1.pic1.1.1.1.1.1.15" class="ltx_p">Creativity and novelty are not particularly applicable in this context, as the task is to create a melody in a traditional style rather than to innovate. Both assistants showed an understanding of the folk tune style, but Assistant A’s answer aligns more closely with traditional folk structures.</span>
<span id="S12.1.1.pic1.1.1.1.1.1.16" class="ltx_p">There are no significant mistakes or inaccurate information in either assistant’s answers. However, Assistant B’s use of the rest “z” in a folk tune is less conventional.</span>
<span id="S12.1.1.pic1.1.1.1.1.1.17" class="ltx_p">My final verdict is: Assistant A is slightly better: [[A&gt;B]].</span>
</span></foreignObject></g></g></svg></p>
</figure>
<figure id="S12.2" class="ltx_float ltx_float_tcbfloat">
<p id="S12.2.1" class="ltx_p">[htb] <svg id="S12.2.1.pic1" class="ltx_picture" height="304.92" overflow="visible" version="1.1" width="603.54"><g transform="translate(0,304.92) matrix(1 0 0 -1 0 0) translate(0,3.54)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g stroke-opacity="0.5" fill="#F2F2F2" fill-opacity="0.5" stroke="#F2F2F2" color="#F2F2F2"><path d="M 3.54 -3.54 L 3.54 297.83 L 603.54 297.83 L 603.54 -3.54 Z" style="stroke:none"></path></g><g fill="#FFFF99" fill-opacity="1.0"><path d="M 0 0 L 0 301.38 L 600 301.38 L 600 0 Z" style="stroke:none"></path></g><g fill="#FFFFE6" fill-opacity="1.0"><path d="M 1.97 1.97 L 1.97 299.41 L 598.03 299.41 L 598.03 1.97 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.84 13.78)"><foreignObject width="580.32" height="273.82" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000"><span id="S12.2.1.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.2" class="ltx_inline-logical-block ltx_minipage ltx_align_bottom" style="width:419.4pt;">
<span id="S12.2.1.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.p2" class="ltx_para">
<span id="S12.2.1.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.p2.1" class="ltx_p"><span id="S12.2.1.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.p2.1.1" class="ltx_text ltx_font_bold">User:</span> Can you come up with a 12 bar chord progression in C that works in the lydian mode?</span>
</span>
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1" class="ltx_para ltx_noindent">
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.1" class="ltx_block ltx_minipage ltx_align_top" style="width:208.1pt;">
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.1.1" class="ltx_p"><span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.1.1.1" class="ltx_text ltx_font_bold">GPT-4:</span>
Sure! In the Lydian mode, the chords are built on the scale degrees of the Lydian scale. Here’s a 12-bar chord progression in C Lydian:</span>
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.1.2" class="ltx_p">| Cmaj7 | Dmaj7 | Em7 | F#m7b5 |</span>
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.1.3" class="ltx_p">| Cmaj7 | Dmaj7 | Em7 | F#m7b5 |</span>
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.1.4" class="ltx_p">| Cmaj7 | Dmaj7 | Em7 | Cmaj7 |</span>
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.1.5" class="ltx_p">This progression uses the I, II, iii, and iv° chords of the C Lydian mode.</span>
</span>
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.2" class="ltx_block ltx_minipage ltx_align_top" style="width:208.1pt;">
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.2.1" class="ltx_p"><span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.2.1.1" class="ltx_text ltx_font_bold">GPT-3.5-Turbo:</span>
Certainly! Here’s a 12-bar chord progression in C that works in the Lydian mode:</span>
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.2.2" class="ltx_p">Cmaj7 | Cmaj7 | Cmaj7 | Cmaj7 | Fmaj7 | Fmaj7 | Cmaj7 | Cmaj7 | G7 | Fmaj7 | Cmaj7 | G7 |</span>
<span id="S12.2.1.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.p1.2.3" class="ltx_p">In this progression, the Cmaj7 chords establish the tonality in C Lydian, while the Fmaj7 and G7 chords add some tension and movement. Feel free to experiment with different voicings and rhythms to make it your own!</span>
</span>
</span>
<span id="S12.2.1.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.2.p3" class="ltx_para">
<span id="S12.2.1.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.2.p3.1" class="ltx_p"><span id="S12.2.1.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.2.p3.1.1" class="ltx_text ltx_font_bold">Judgement:</span> Assistant B is significantly better: [[B&gt;&gt;A]].</span>
</span></span></foreignObject></g></g></svg></p>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2406.11938" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2406.11939" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2406.11939">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2406.11939" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2406.11940" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Sat Jul  6 00:31:56 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2406.11939
The rapid evolution of language models has necessitated the development of more challenging benchmarks. Current static benchmarks often struggle to consistently distinguish between the capabilities of different models and fail to align with real-world user preferences. On the other hand, live crowd-sourced platforms like the Chatbot Arena collect a wide range of natural prompts and user feedback. However, these prompts vary in sophistication and the feedback cannot be applied offline to new models. In order to ensure that benchmarks keep up with the pace of LLM development, we address how one can evaluate benchmarks on their ability to confidently separate models and their alignment with human preference. Under these principles, we developed BenchBuilder, a living benchmark that filters high-quality prompts from live data sources to enable offline evaluation on fresh, challenging prompts. BenchBuilder identifies seven indicators of a high-quality prompt, such as the requirement for domain knowledge, and utilizes an LLM annotator to select a high-quality subset of prompts from various topic clusters. The LLM evaluation process employs an LLM judge to ensure a fully automated, high-quality, and constantly updating benchmark. We apply BenchBuilder on prompts from the Chatbot Arena to create Arena-Hard-Auto v0.1: 500 challenging user prompts from a wide range of tasks. Arena-Hard-Auto v0.1 offers 3x tighter confidence intervals than MT-Bench and achieves a state-of-the-art 89.1% agreement with human preference rankings, all at a cost of only $25 and without human labelers. The BenchBuilder pipeline enhances evaluation benchmarks and provides a valuable tool for developers, enabling them to extract high-quality benchmarks from extensive data with minimal effort.
abstract
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
Abstract
2406.11939
1IntroductionThe proliferation of Large Language Models (LLMs)  has spurred advancements, leading to models trained on vast datasets with significantly enhanced capabilities.
However, this innovation also presents challenges in designing effective evaluation benchmarks.
Many existing benchmarksare unable to keep up with the rapidly advancing capabilities of these models, becoming saturated and unable to differentiate between state-of-the-art models.
Additionally, many benchmarks are designed for automatic evaluation with restrictive prompts, such as close-ended or multiple-choice questions, which do not accurately reflect a model’s performance in real-world interactions.
Recent developments in benchmark evaluationhave made it feasible to evaluate more open-ended prompts, yet generating high-quality prompts at scale remains challenging. Furthermore, static benchmarks are prone to over-fitting and potential test leakage over time, reducing their effectiveness in ranking models.To address these issues, an effective benchmark must confidently differentiate model capabilities, reflect real-world human interactions, and update frequently to stay relevant and avoid overfitting.
Live crowdsourced evaluation platforms, like the Chatbot Arena, address some of these concerns by collecting real-world user prompts with preference labels. However, without quality control, these platforms can include low-quality prompts that do not adequately differentiate top models.In this work, we introduce BenchBuilder, a data pipeline designed to automatically build high-quality benchmarks from live crowdsourced data source such as the Chatbot Arena.
BenchBuilder identifies diverse benchmark prompts that can confidently separate model performance. It uses seven key indicators of high-quality prompts, such as specificity and domain knowledge, and employs an LLM to annotate prompts for these indicators.
The resulting benchmark, Arena-Hard-Auto v0.1, uses an LLM judgeto estimate human preference against a baseline model, making BenchBuilder and Arena-Hard-Auto v0.1 a fully automated system.
We further explore how to measure the quality of benchmarks by introducing metrics to assess a benchmark’s ability to confidently separate models and its agreement with human preferences.
Comparing Arena-Hard-Auto v0.1 to current leading benchmarks such as AlpacaEvaland MT-Bench, we demonstrate that Arena-Hard-Auto v0.1 offers significantly stronger separability with tighter confidence intervals. Additionally, Arena-Hard-Auto v0.1 achieves 89.1% agreement with Chatbot Arena rankings, making it an excellent predictor of downstream performance.
Arena-Hard-Auto v0.1 provides a cost-effective and informative evaluation benchmark, and the BenchBuilder pipeline serves as a valuable tool for developers seeking to automatically generate high-quality benchmarks from vast data sources.To summarize, our work makes the following contributions:1.We propose a novel data curation pipeline, BenchBuilder, to automatically construct high-quality benchmarks from crowdsourced data.2.We propose metrics to capture desired properties in a benchmark, and validate Arena-Hard-Auto v0.1 achieves significantly better separability and agreement than existing benchmarks.3.We open-source both the Arena-Hard-Auto v0.1 benchmark and BenchBuilder pipeline111Our code is available at:https://github.com/lm-sys/arena-hard-auto.
introduction
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
1Introduction
2406.11939
2Related worksLLM benchmarks.We briefly review widely used LLM benchmarks. Most existing benchmarks are static and ground-truth-based (e.g., multi-choice question answering) . They cover a wide range of domains, including math, science, coding, and reasoning. Common ones include MMLU, MATH, GSM-8K,
HumanEval,
DROP,
BigBench, HellaSwag, AGIEval, GPQA, as well as comprehensive collection like HELM.
Many have considered task-based evaluation such as IFEval, SWE-Bench, or AgentBench. As LLMs become widely adopted in open-ended scenarios involving interaction with humans (e.g., chatbot) , many have considered human evaluation using domain experts or crowd raters such as Amazon Mechanical Turkto examine models’ response quality.
As an alternative to human labeling, previous work has shown that LLM-as-a-judge can be effective human preference proxies (e.g., AlpacaFarm, MT-bench, AlpacaEval) .Benchmark leakage.A fundamental limitation of static benchmarks is the potential risk of test set leakage (i.e., contamination) . Existing workshave suggested a growing risk of contamination, which undermines the reliability of benchmarks over time, motivating the need for benchmarks that are more frequently updated.Live benchmarks.DynaBenchidentifies these
challenges and recommends creating living and continuously evolving benchmarks. Recent works LiveCodeBench, R2E, as well as the community based live evaluation, Chatbot Arena. However, none of these focus on developing a pipeline for automatic benchmark curation.
body
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
2Related works
2406.11939
3How do you measure benchmarks?Given a benchmark consists of a datasetof prompts and scoring functionto measure the performance of modelon prompt, we outline two key properties that an LLM chatbot benchmark approximating human preference should possess to provide a meaningful measurement of capabilities between models:1.Separability:the benchmark should separate models with high confidence.2.Alignment with Human Preference:should agree with human preference.While previous works have focused on alignment, separability is also a crucial consideration when comparing models of similar quality (e.g., different checkpoints from the same training run) . However, achieving high-confidence separability is challenging due to limitations in prompt design and inherent variances in LLM evaluations. Overly simplistic prompts fail to distinguish between models, while the randomness in human and LLM judgments leads to inconsistent predictions. As a result, it is often difficult to confidently determine if a model’s apparent performance reflects a genuine difference in capability or merely noisy observations, highlighting a need for methods to verify whether a benchmark can reliably separate similar models.Statistical measures like Pearsonand Spearman Correlations, commonly used in benchmarks such as AlpacaEvalto measure correlation to human preference ranking, may fail to adequately address model separability and ranking instability. In addition, these measures only provide a coarse signal of ranking correlation without quantifying the magnitude of performance differences between model pairs.To address these issues, we develop three novel metrics:Separability with Confidence,Agreement with Confidence, andPair Rank Brier Score.Separability with Confidencequantifies the benchmark’s confidence by measuring its consistency in predicting the winner of a model pair across random seeds through bootstrapping. This is done by calculating the percentage of model pairs that have non-overlapping confidence intervals of their benchmark scores. A higher percentage indicates that the benchmark is more confident in distinguishing between the performance of different models, as the confidence intervals of their scores do not overlap.Agreement with Confidence Intervalmeasures how well benchmarks A and B confidently distinguish between two models with the same ordering. Given two modelsto evaluated on A and B, we assign the following scores:1.If both A and B can confidently separate, we assign a score of 1 if the rankings match across A and B, otherwise -1.2.If either A or B cannot separatewith confidence, we assign a score of 0.The final agreement score across models is the average score across all unique model pairs. An agreement score of 1 implies that benchmark A confidently agrees with benchmark B on the preference of every unique model pair. Conversely, a score of -1 indicates that A confidently disagrees with B on every unique model pair.Pair Rank Brier Scorefurther assesses an LLM benchmark’s capability to predict the ranking of a pair of competing models by rewarding confidence in correct predictions while penalizing confidence when incorrect. Consider two modelswith disparate quality. Although two benchmarks A and B predict the same ranking, they predictasand, respectively (undetectable by Spearman correlation) . These benchmarks would result in very different Brier scores, reflecting their ability to quantify the magnitude of performance difference between the models. If both benchmarks give the wrong prediction of the winner, we prefer the benchmark with a less confident prediction. Brier score weighs a benchmark’s accuracy and its ability to quantify the appropriate level of uncertainty in its predictions.For each model pair, the benchmark’s forecast probabilitycan be estimated aswithand, whereandare the bootstrapped mean and variance by evaluating multiple times. The Brier Score Loss is then calculated over the benchmark’s prediction for each model pair with respect to the ground truth probability, where Brier Score is defined as, whereis the number of model pairs andis the ground truth outcome.Separability with Confidence, Agreement with Confidence Interval, and Pair Rank Brier Score each provide useful perspectives for evaluating the effectiveness of LLM benchmarks. However, no single metric is intended to be individually sufficient. Ultimately, these metrics are complementary and should be considered together holistically. We claim that together, these metrics offer a robust framework for assessing benchmark performance, balancing the need for clear differentiation with alignment to human preferences.
body
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
3How do you measure benchmarks?
2406.11939
4The BenchBuilder Pipeline and Arena-Hard-Auto v0.1 Dataset4.1BenchBuilder: A pipeline for converting messy data into high-quality benchmarksFigure 1:BenchBuilder Pipeline.Starting with a live data source of crowdsourced user prompts, we first cluster their embeddings to form topic clusters. An LLM annotator then assigns quality scores based on the required skills. Clusters with low quality scores are filtered out, and we sample from the remaining high-quality clusters to create a diverse and challenging dataset of benchmark prompts.BenchBuilder automatically extracts diverse, high quality prompts from a dataset of 200,000 user queries collected via Chatbot Arena. This is done by assigning a quality score to each prompt and by sampling prompts with a high quality score evenly across a wide range of topics. An outline of our data creation pipeline is shown in Figure1.Key Prompt Qualities•Specificity:Does the prompt ask for a specific, well-defined output without leaving any ambiguity?•Domain Knowledge:Does the prompt test the AI’s knowledge and understanding in a specific domain or set of domains?•Complexity:Does the prompt have multiple components, variables, or levels of depth and nuance?•Problem-Solving:Does the prompt require active problem-solving: analyzing and clearly defining the problem and systematically devising and implementing a solution?•Creativity:Does the prompt require a creative approach or solution?•Technical Accuracy:Does the prompt require an answer with a high degree of technical accuracy, correctness and precision?•Real-world Application:Does the prompt relate to real-world applications?To discover high quality prompts, we define seven qualities of a prompt which signify the skills required to successfully answer, including specificity, domain knowledge, complexity, and more shown in the figure above.
These qualities are used by an LLM annotator to assign a “quality score” to a given prompt, which indicates how many of the above qualities are seen in. Full instruction prompt is in Section11.To ensure that the filtered prompts represent a wide range of tasks, we adopt a topic modeling pipeline in BERTopic by first converting each prompt with OpenAI’s embedding model (text-embedding-3-small) , reducing dimension with UMAP, and using a hierarchical-based clustering algorithm (HDBSCAN)  to identify clusters which are then summarized using GPT-4-Turbo.As some topic clusters contain mostly trivial or ill defined prompt like “hi”, we select the topic clusters with mean quality score >=6 out of 7 and sample prompts equally across each of these clusters. The resulting dataset consists of mostly well-defined, technical problem-solving queries as required in the above key criteria. Dataset statistics and further description of the filtering process used for Arena-Hard-Auto v0.1 along are in the following section.4.2Arena-Hard-Auto v0.1: A set of challenging, high-quality prompts from Chatbot ArenaWe utilize the BenchBuilder pipeline to create Arena-Hard-Auto v0.1, a set of 500 high-quality prompts taken from 200,000 prompts from Chatbot Arena, an open source platform for LLM evaluation. We first remove any duplicate prompts, multi-turn conversations, or non-English conversations, and obtain over 4,000 topic clusters and 75,000 queries covering a wide range of domains.Then, we use the GPT-3.5-Turbo as judge to assign a “quality score” to each prompt and remove prompts which either have a score < 5 or which belong to a topic cluster with a mean score < 3. We present the statistics in Table1. Notably, around 18% out of 75,000 queries are higher quality according to GPT-3.5-Turbo. We then use GPT-4-Turbo as judge to assign a quality score and ensuring the quality of the remaining prompts, resulting with over 500 high quality clusters. We randomly sample 500 prompts from these clusters, creating Arena-Hard-Auto v0.1. Additionally, we confirm our final dataset does not contain any personally identifiable information or offensive content.In Figure3, we show examples of topic clusters ranging from low to high mean scores provided by GPT-3.5-Turbo. We can observe clusters with higher scores often correlate to challenging topics or tasks for LLMs like game development or mathematical proofs. On the other hand, clusters with lower scores point to trivial or ambiguous questions like "Flirty Texting Strategies". We present several example prompts and their respective topic cluster below.[htb]Cluster 1: Greetings and Well-Being Inquiry(Mean Score: 2.7) ⬇Yo,whatupmybrother(Qualities:None) Cluster 2: US Presidents Query(Mean Score: 3.2) ⬇WhowasthepresidentoftheUSin1975(Qualities:Specificity,Domain-Knowledge,TechnicalAccuracy,Real-World) Cluster 3: Physics Problem Solving(Mean Score: 5.0) ⬇A50,000kgairplaneinitiallyflyingataspeedof60.0m/sacceleratesat5.0m/s2for600meters.Whatisitsvelocityafterthisacceleration?Whatisthenetforcethatcausedthisacceleration?(Qualities:Specificity,Domain-Knowledge,Complexity,Problem-Solving,TechnicalAccuracy,Real-World) Cluster 4: OpenCV Image Processing Technique(Mean Score: 5.5) ⬇youaregivenatasktodetectnumberoffacesineachframeofanyvideousingpytorchanddisplaythenumberinthefinaleditedvideo.(Qualities:All) To see whether “quality score” assigned during BenchBuilder’s pipeline correlates with separability and agreement, we sample 50 prompts per score and compare the responses from GPT-4 and Llama-2-70b-Chat, with GPT-4-Turbo as judge. In Figure4(Left) , we observe a strong correlation between high potential score and the win-rate of GPT-4-Turbo over Llama-2-70b-Chat. A similar trend is also observed in other model pairs such as Claude Sonnet vs Haiku and Mistral-Largevs Mixtral.Since qualities assigned by GPT-4-Turbo are critical to ensure the final prompt selection is high-quality, we verified its accuracy as annotator. We construct “ground truth” labels for 200 randomly sampled queries by collecting majority votes from GPT-4o, Claude-3-Opus, and Gemini-1.5-Pro. GPT-4-Turbo has 85.6% accuracy against “ground truth” labels, validating its ability as annotator.Figure 2:Mean score of various topic clusters in descending order. Higher-scoring clusters correlate to challenging topics, while lower scores indicate trivial or ambiguous questions. A more complete topic cluster plot is inFigure 6.Figure 3:Comparison between MT-bench and Arena-Hard-Auto v0.1. The latter offers significantly better separability between models and tighter confidence intervals. GPT-4-0314 has no variance in Arena-Hard-Auto v0.1 because it is the anchor model.Figure 4:Win-rate of three model pairs
(GPT-4-0613 vs Llama-2-70b-chat, Claude-3-Sonnet-20240229 vs Claude-3-Haiku-20240307, and Mistral-Large vs Mixtral-8x7b-Instruct-v0.1) 
over “quality score”. We randomly sample 50 queries for each quality score 0-7 and bootstrap a win-rate and confidence interval between model pairs on each score interval of 2. We observe a similar trend of win-rate between model pairs becomes increasingly separable as the quality score increases.
body
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
4The BenchBuilder Pipeline and Arena-Hard-Auto v0.1 Dataset
2406.11939
5Evaluation with LLM-as-a-JudgeEvaluating models on challenging queries such as Arena-Hard-Auto v0.1 is a non-trivial task. Most queries involve deep domain knowledge and problem solving skills, requiring expert-level judgment to evaluate the answer quality. Unfortunately, it is prohibitively expensive and time consuming to use human judges. Following LLM-as-a-Judgeand AlpacaFarm, we employ LLM-as-a-judge framework to approximate human preference.To obtain a score for modelon prompt, we consider the pairwise comparison setup against a strong baseline model(GPT-4-0314)  and employ a strong judge model (e.g., GPT-4-Turbo or Gemini-1.5-Pro)  to measure preference. We obtain a score for each outputby asking the judge to rank its preference oforon a Likert scale(prefersmuch less than,prefersmuch more than) . This way, a model will be penalized more in big losses than small losses, which we find to be effective in separating models. We also employ chain-of-thought (CoT) to prompt the LLM judge to generate its own answer first before giving judgments. The full judge prompt can be found in Section11.To avoid potential position bias, we adopt a two-game setup – per query we swap the models on the first and second position. This results in 1000 judgments per model evaluation. Following Chatbot Arena, we adopt the Bradley-Terrymodel to produce model’s the final model scores. We aggregate all pairwise comparisons to the baseline model for all models and bootstrapping the comparisons to retrieve a bootstrapped confidence interval of all models’ win-rate against the baseline, producing a ordered ranking of all models by their win-rates.
body
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
5Evaluation with LLM-as-a-Judge
2406.11939
6Experimental ResultsQuality Score1+2+3+4+5+6+7+% of queries95.483.561.948.733.817.90.2QualitiesSpecificityDomain-knowledgeComplexityProblem-solvingCreativityTech. AccuracyReal-world% of queries57.363.435.034.926.139.087.9Table 1:First row is the percentage of queries with quality scores of the column or more in 75K Chatbot Arena data assigned by GPT-3.5-Turbo. Second row is the percentage of queries in 75K Chatbot Arena labeled by GPT-3.5-Turbo with each of the 7 qualities.Chatbot ArenaMT BenchAlpacaEval 2.0 LCArena-Hard-Auto v0.1Confiderence AgreementN/A26.1%81.2%89.1%Separability85.8%22.6%83.2%87.4%Spearman CorrelationN/A91.3%90.8%94.1%Brier ScoreN/A0.090.110.069Real-worldYesMixedMixedYesFreshnessLiveStaticStaticFrequent UpdatesEval cost per modelvery high$10$10$25Prompts per model10000+1608001000Table 2:We use a set of top-20 models1on Chatbot Arena (2024/04/13)  that are also present on the AlpacaEval leaderboard to calculate separability and agreement per benchmark. We consider the human preference ranking by Chatbot Arena (English only)  as the reference to calculate agreement.ModelGPT4-TClaude3-OpusGemini1.5-ProLlama3-70BEnsemble-as-JudgesConfiderence Agreement89.1%66.7%84.8%65.6%91.5%Separability87.4%83.68%82.11%81.6%89.5%Spearman Correlation94.1%77.0%95.2%70.5%96.5%Brier Score0.0690.1700.0640.1960.065Table 3:Statistics of Arena-Hard-Auto v0.1 with four LLM different judges: GPT4-T (gpt-4-1106-preview) , Claude-3-Opus, Gemini1.5-Pro (gemini-1.5-pro-0514) , Llama3-70B (llama-3-70b-instruct) . We compare rankings produced by these judges against Chatbot Arena (English)  ranking (as of 2024/04/13) . We observe GPT-4T and Gemini1.5-Pro have higher agreement than Claude-3-Opus and Llama-3-70B. Furthermore, the ensemble of GPT4-T and Gemini1.5-Pro shows even higher agreement.ModelArena-Hard-Auto v0.1Random Sample 1Random Sample 2Confiderence Agreement84.2%57.5%66.1%Separability80.5%74.7%76.3%Spearman Correlation94.7%64.7%72.5%Brier Score0.0690.2150.162Table 4:We compare Arena-Hard-Auto v0.1 with two sets of 500 prompts randomly sampled from 75K Chatbot Arena user queries. We evaluate the set of top-20 models and compare various statistics across. Each prompt is judged only once by positioning the baseline answer first.Arena-Hard-Auto v0.1Confiderence Agreement92.0%Spearman Correlation96.4%Brier Score0.055Table 5:We compare Arena-Hard-Auto v0.1 (gpt-4-1106-preview as judge)  to Chatbot Arena Category Hard Prompt (English)  on the same set of top-20 models*. By comparing Arena-Hard-Auto v0.1 to a similar distribution of queries from Chatbot Arena, we obtain even higher alignment to human preferences.6.1Setup and BaselinesTo compare Arena-Hard-Auto v0.1’sseparabilityandalignment with humansagainst other widely used benchmarks, MT-Benchand AlpacaEval 2.0 Length Controlled, we obtain 95% confidence interval of model performances via applying 100 rounds of bootstrapping on judgment results for each benchmark. For AlpacaEval, we use pre-existing results from their repository. We obtain MT-Bench judgment with no modification to their recommended evaluation setup. For Arena-Hard-Auto v0.1, we employ the system proposed insection 5by choosing gpt-4-0314 as baseline model for pairwise comparison.In order to compare these benchmarks fairly, we use a set of top-20 models††footnotemark:on Chatbot Arena(2024/04/13)  that are also presented on AlpacaEval leaderboard (2024/04/13)  as ground truth for human preferences on the model ranking orders. Insubsection 6.3, we individually compare Arena-Hard-Auto v0.1 to Chatbot Arena within a more similar distribution of user queries. We compute separability using 95% confidence intervals and consider the human preference ranking by Chatbot Arena as the reference to calculate agreement, Spearman correlation, and Pair Rank Brier Score across AlpacaEval, MT-Bench, and Arena-Hard-Auto v0.1.6.2Separability and Agreement Across BenchmarksIn Table2, Arena-Hard-Auto v0.1 shows the highest separability (87.4%)  against widely adopted LLM benchmarks and offers highest agreement (89.1%)  to Chatbot Arena at $25 cost. In Figure3, we show Arena-Hard-Auto v0.1 offers significantly stronger separability against MT Bench with tighter confidence intervals. With only 500 prompts, Arena-Hard-Auto v0.1 achieve impressive alignment to and even higher separability than Chatbot Arena Ranking, which constitutes for approximately 1 million real-world human preferences.Notably, we observe a significant gap between MT-bench’s Spearman Correlation (91.3%)  and confidence agreement (22.6%)  to Chatbot Arena, an example where Spearman Correlation fails to account for variance of the rankings, and hence cannot adequately measure important ranking granularity of top LLMs. We present a visual comparison between Arena-Hard-Auto v0.1 and MT-Bench inFigure 3, highlighting Arena-Hard-Auto v0.1’s improved separability.6.3Comparing to a similar distribution of human preferenceWe set Chatbot Arena Category Hard Prompts (English)  as ground truth reference. Chatbot Arena Category Hard Prompt is a human preference ranking leaderboard based on votes from a harder subset of 200,000+ prompts extracted from over 1 million Chatbot Arena queries using a similar method as BenchBuilder. InTable 5, we show Arena-Hard-Auto v0.1 achieving an overall higher alignment (92.0% Confidence Agreement)  to human preferences.6.4Comparing to Random Subsets of ArenaTo further verify BenchBuilder pipeline’s ability to select high-quality prompts, we compare Arena-Hard-Auto v0.1 to two sets of 500 randomly selected prompts from the 75,000 user queries prior to pipeline extraction. We compare their ranking preferences of the set of top-20 models produced by GPT-4-1106-Preview as judge. In Table4, we observe Arena-Hard-Auto v0.1 is significantly better than both random baselines, verifying BenchBuilder’s effectiveness at selecting superior prompts for a LLM benchmark.6.5Ensemble-as-JudgesWe study four popular LLMs222GPT-4-1106-Preview, Claude-3-Opus-20240229, Gemini-1.5-Pro-0514, and Llama-3-70b-instructas judge on Arena-Hard-Auto v0.1. We statistically compare their ranking preference of the set of top-20 models111gpt-4-turbo-2024-04-09, claude-3-opus-20240229, claude-3-sonnet-20240229, gpt-4-0314, gpt-4-0613, mistral-large-2402, qwen1.5-72b-chat, mistral-medium, claude-2.0, gpt-3.5-turbo-0613, claude-2.1, gemini-pro, mixtral-8x7b-instruct-v0.1, gpt-3.5-turbo-0314, yi-34b-chat, tulu-2-dpo-70b, dbrx-instruct-preview, vicuna-33b, starling-lm-7b-alpha, llama-2-70b-chatto the Chatbot Arena Leaderboard in Table3. Out of the four judges, GPT-4-1106-Preview has the highest separability and agreement to human ranking preference. We also observe GPT-4-1106-Preview and Gemini-1.5-Pro have much higher agreement than Claude-3-Opus and Llama-3-70b-instruct, with more than 20% improvement.We also experiment combining GPT-4-1106-Preview and Gemini-1.5-Pro as an ensemble judge by combining their judgment results. This achieves an even higher separability and agreement to human ranking preference, with agreement reaching 91.5%, demonstrating effectiveness of judging LLMs with Ensemble-as-Judges.
body
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
6Experimental Results
2406.11939
7LimitationsAlthough our data source comes from diverse distributions, we acknowledge possible biases within our pipeline. For example, the 7 qualities we defined might not be all-encompassing and could potentially bias towards a limited distribution of prompts, such as technical fields. Further, prompt selection process could be biased by LLM annotators. Additionally, for model evaluation, we are employing LLMs as judges, who are known to suffer from verbosity bias. In Table7, we present how verbosity can effect LLM as Judge on Arena-Hard-Auto v0.1. We also acknowledge potential self-bias in LLM judges (e.g., GPT-4 Judge prefers GPT-4 answers)  in Table3. Arena-Hard-Auto v0.1 also does not evaluate multi-turn or non-English interactions.In the future, we plan to further the exploration of a more generalized and diverse set of prompt qualities, establishing a pipeline to systematically produce benchmark across a more comprehensive and inclusive distribution of prompts while maintaining high separability and alignment to humans. Additionally, we believe our approach of utilizing LLM-as-a-judge can be improved, such as developing a system of Ensemble-as-Judges, similar to Judges with Jury, to reduce biases and improve correlation.
body
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
7Limitations
2406.11939
8ConclusionsWe introduced BenchBuilder, a data curation pipeline that transforms crowdsourced data into high-quality benchmarks by seven key qualities. This pipeline enables building challenging and evolving benchmarks which is crucial for evaluating today’s advanced language models.
Our evaluation metrics, including separability and agreement with confidence, provide a comprehensive assessment of benchmarks.
We show the resulting benchmark, Arena-Hard-Auto v0.1, significantly improves separability and alignment with human preferences over existing benchmarks, achieving 89.1% agreement with Chatbot Arena rankings at only $25 per evaluation.
We expect Arena-Hard-Auto v0.1 to be useful for LLM developers to evaluate their models with confidence and BenchBuilder to be a valuable tool for developers seeking to extract high-quality benchmark from vast amounts of data with minimal human effort.
body
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
8Conclusions
2406.11939
9AcknowledgmentWe thank Matei Zaharia, Yann Dubois, Anastasios Angelopoulos, Lianmin Zheng, Lewis Tunstall, Nathan Lambert, Xuechen Li, Naman Jain, Ying Sheng, Maarten Grootendorst for their valuable feedback. We thank Siyuan Zhuang and Dacheng Li for the valuable review and debug of the code. We thank Microsoft AFMR for Azure OpenAI credits support. We also thank Together.ai & Anyscale for open model endpoint support.
body
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
9Acknowledgment
2406.11939
10AppendixJudgeModel NameWin RateAverage Token #gpt-4-turbo-2024-04-09*82.6-1.8/+1.6662gpt-4-0125-preview*78.0-2.2/+2.4619claude-3-opus-2024022960.4-3.3/+2.4541gpt-4-031450.0-0.0/+0.0423claude-3-sonnet-2024022946.8-2.1/+2.2552claude-3-haiku-2024030741.5-2.8/+2.5505llama-3-70b-instruct41.1-2.5/+2.4583gpt-4-061337.9-2.2/+2.0354mistral-large-240237.7-1.9/+2.6400mixtral-8x22b-instruct-v0.136.4-2.7/+2.9430Qwen1.5-72B-Chat36.1-2.5/+2.2474command-r-plus33.1-2.1/+2.2541mistral-medium31.9-2.3/+2.4485mistral-next27.4-2.1/+1.7297gpt-3.5-turbo-061324.8-1.6/+2.0401claude-2.024.0-2.5/+2.5295dbrx-instruct23.9-1.4/+1.5415Mixtral-8x7B-Instruct-v0.123.4-2.3/+1.7457gpt-3.5-turbo-012523.3-2.2/+2.3329Yi-34B-Chat23.1-1.8/+2.0611Starling-LM-7B-beta23.0-1.9/+2.2530claude-2.122.8-1.6/+2.1290Snorkel-Mistral-PairRM-DPO20.7-2.2/+1.5564llama-3-8b-instruct20.6-2.5/+1.8585gpt-3.5-turbo-110618.9-1.6/+2.1285gpt-3.5-turbo-030118.1-1.7/+1.2334gemini-1.0-pro17.8-1.7/+1.7322command-r17.0-1.9/+1.7432tulu-2-dpo-70b15.0-1.4/+1.2550Starling-LM-7B-alpha12.8-1.4/+1.4483mistral-7b-instruct-v0.212.6-1.6/+1.3541Llama-2-70b-chat-hf11.6-1.6/+1.4595vicuna-33b-v1.38.6-1.3/+1.0451gemma-7b-it7.5-1.1/+1.2378Llama-2-7b-chat-hf4.6-0.8/+0.8561gemma-2b-it3.0-0.6/+0.7369Table 6:Arena-Hard-Auto v0.1 Leaderboard (baseline: GPT-4-0314) . *GPT-4-Turbo’s high score can be due to the GPT-4 judge favoring GPT-4 outputs.JudgeModel NameWin RateAverage Token #GPT-4-1106-Previewgpt-3.5-turbo-0125-detailed29.86421gpt-3.5-turbo-0125-chatty23.89361gpt-3.5-turbo-012523.2328Claude-3-Opusgpt-3.5-turbo-0125-detailed40.78421gpt-3.5-turbo-0125-chatty28.49375gpt-3.5-turbo-012527.97328Table 7:Length Bias Comparison Between GPT and Claude as Judge. Prompt for detailed:“You are a helpful assistant who thoroughly explains things with as much detail as possible.”, prompt for chatty: ‘You are a helpful assistant who is chatty.”, and default prompt: "You are a helpful assistant."Model NameWin RateAverage Token #gpt-3.5-turbo-0125-123.05328gpt-3.5-turbo-0125-222.93328gpt-3.5-turbo-0125-322.75328Table 8:Variances between 3 separate runs of Arena-Hard-Auto v0.1 on gpt-3.5-turbo’s performance using GPT-4-1106-Preview as judge.(a) Chatbot Arena (20K Votes) (b) AlpacaEval 2.0 LC(c) Arena-Hard-Auto v0.1(d) MT BenchFigure 5:Above is the predicted model predicted probability against the bootstrapped arena “ground truth” probability (jittered to show clusters) . While both Alpaca eval and Arena-Hard-Auto v0.1 have large clusters around (0,0)  and (1,1)  signifying good forecasting, Arena-Hard-Auto v0.1 has lighter clusters on (0,1)  and (1,0) , if any, revealing less overconfidence. MT Bench has heavy tails along the top and bottom, revealing underconfidence. However, none of these benchmarks show an “ideal” y=x curve (with dense ends)  expected with a perfectly calibrated forecast, signifying room for future research.Figure 6:A more complete selection of mean scores of various topic clusters in descending order.
body
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
10Appendix
2406.11939
11PromptsPrompt Quality Systems Instruction:Your task is to evaluate how well the following input prompts can assess the capabilities of advanced AI assistants. For the input prompt, please analyze it based on the following 7 criteria. For each criteria, make sure to explain before determine whether the input satisfy it.1. Specificity: Does the prompt ask for a specific, well-defined output without leaving any ambiguity? This allows the AI to demonstrate its ability to follow instructions and generate a precise, targeted response.2. Domain Knowledge: Does the prompt test the AI’s knowledge and understanding in a specific domain or set of domains? The prompt must demand the AI to have a strong prior knowledge or mastery of domain-specific concepts, theories, or principles.3. Complexity: Does the prompt have multiple components, variables, or levels of depth and nuance? This assesses the AI’s capability to handle complex, multi-faceted problems beyond simple queries.4. Problem-Solving: Does the prompt require active problem-solving: analyzing and clearly defining the problem and systematically devising and implementing a solution? Note active problem-solving is not simply reciting facts or following a fixed set of instructions.5. Creativity: Does the prompt require a creative approach or solution? This tests the AI’s ability to generate novel ideas tailored to the specific needs of the request or problem at hand.6. Technical Accuracy: Does the prompt require an answer with a high degree of technical accuracy, correctness and precision? This assesses the reliability and truthfulness of the AI’s outputs.7. Real-World Application: Does the prompt relate to real-world applications? This tests the AI’s ability to provide practical and actionable information that could be implemented in real-life scenarios.After analyzing the input prompt based on these criteria, you must list the criteria numbers that the prompt satisfies in the format of a Python array. For example, "Criteria Satisfied: [1, 2, 4, 6, 7]".LLM-Judge System InstructionPlease act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user prompt displayed below. You will be given assistant A’s answer and assistant B’s answer. Your job is to evaluate which assistant’s answer is better.Begin your evaluation by generating your own answer to the prompt. You must provide your answers before judging any answers.When evaluating the assistants’ answers, compare both assistants’ answers with your answer. You must identify and correct any mistakes or inaccurate information.Then consider if the assistant’s answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, it is more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based on assumptions. Relevant means all parts of the response closely connect or are appropriate to what is being asked. Concise means the response is clear and not verbose or excessive.Then consider the creativity and novelty of the assistant’s answers when needed. Finally, identify any missing important information in the assistants’ answers that would be beneficial to include when responding to the user prompt.After providing your explanation, you must output only one of the following choices as your final verdict with a label:1.Assistant A is significantly better: [[A>>B]]2.Assistant A is slightly better: [[A>B]]3.Tie, relatively the same: [[A=B]]4.Assistant B is slightly better: [[B>A]]5.Assistant B is significantly better: [[B>>A]Example output: "My final verdict is tie: [[A=B]]".
body
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
11Prompts
2406.11939
12Additional Examples[htb]User:Use ABC notation to write a melody in the style of a folk tune.GPT-4:X:1 T:Untitled Folk Tune M:4/4 L:1/8
K:G|:G2A2|B2A2|G2E2|D4|E2F2|G2F2|E2C2|B,4|A2B2|c2B2|A2F2|E4|D2E2|F2E2|D2B,2|C4:|GPT-3.5-Turbo:Sure! Here’s a simple melody in the style of a folk tune written in ABC notation:X:1 T:Folk Tune M:4/4 L:1/4 K:C
CDE|FGA|BAG|FED| CDE|FGA|BAG|c2z| CDE|FGA|BAG|FED| CDE|FGA|BAG|c2z||This melody is in the key of C major and consists of four measures. Feel free to modify it or add more sections to create your own folk tune!Judgement:My own answer to the prompt would be:X:1 T:My Folk Melody M:3/4 L:1/8
    K:D A2|:d2f2a2|b2a2f2|e2d2B2|A4F2| G2A2B2|A2G2F2|E2D2F2|1D6:|2D4A2|
    |:d2f2a2|b2a2f2|e2c2A2|B4A2| G2A2B2|A2G2F2|E2D2F2|1D6:|2D4||This melody is in the key of D major, has a 3/4 time signature, and is structured in a typical AAB form common in folk music. It uses a range of an octave, which is typical for folk tunes that need to be easily singable.Now, let’s evaluate the responses from Assistant A and Assistant B.Assistant A’s Answer: Assistant A provides a melody in ABC notation that is coherent and follows the structure of a folk tune. The melody is in 4/4 time, which is common for folk music, and it is in the key of G major. The melody has a two-part form (indicated by the repeat signs |: and :|) , which is typical in folk music. The range of the melody is appropriate for a folk tune, and the rhythm is simple and repetitive, which is characteristic of the genre.Assistant B’s Answer: Assistant B also provides a melody in ABC notation that is coherent and could be considered in the style of a folk tune. The melody is in 4/4 time and the key of C major. The melody consists of four measures repeated twice, which is a simple structure that could be found in folk music. However, Assistant B’s melody is less traditional in its form compared to Assistant A’s, as it does not have a distinct A and B section, which is more common in folk tunes. Additionally, the use of “z” for a half-measure rest is less typical in folk tunes, which usually have a more continuous melodic line.Comparison: Both assistants provided melodies that could be considered folk tunes. Assistant A’s melody has a more traditional structure with a clear A and B section, while Assistant B’s melody is simpler and does not follow the traditional AAB or AB form. Assistant A’s melody is also more detailed in terms of the ABC notation, providing a full eight-measure section that is repeated, which gives a more complete sense of the tune. Assistant B’s melody is correct in its notation but is more basic and less representative of a traditional folk tune structure.In terms of helpfulness, relevance, and conciseness, both assistants provided relevant and concise answers. However, Assistant A’s answer is slightly more helpful due to the more traditional structure of the provided melody.Creativity and novelty are not particularly applicable in this context, as the task is to create a melody in a traditional style rather than to innovate. Both assistants showed an understanding of the folk tune style, but Assistant A’s answer aligns more closely with traditional folk structures.There are no significant mistakes or inaccurate information in either assistant’s answers. However, Assistant B’s use of the rest “z” in a folk tune is less conventional.My final verdict is: Assistant A is slightly better: [[A>B]].[htb]User:Can you come up with a 12 bar chord progression in C that works in the lydian mode?GPT-4:Sure! In the Lydian mode, the chords are built on the scale degrees of the Lydian scale. Here’s a 12-bar chord progression in C Lydian:| Cmaj7 | Dmaj7 | Em7 | F#m7b5 || Cmaj7 | Dmaj7 | Em7 | F#m7b5 || Cmaj7 | Dmaj7 | Em7 | Cmaj7 |This progression uses the I, II, iii, and iv° chords of the C Lydian mode.GPT-3.5-Turbo:Certainly! Here’s a 12-bar chord progression in C that works in the Lydian mode:Cmaj7 | Cmaj7 | Cmaj7 | Cmaj7 | Fmaj7 | Fmaj7 | Cmaj7 | Cmaj7 | G7 | Fmaj7 | Cmaj7 | G7 |In this progression, the Cmaj7 chords establish the tonality in C Lydian, while the Fmaj7 and G7 chords add some tension and movement. Feel free to experiment with different voicings and rhythms to make it your own!Judgement:Assistant B is significantly better: [[B>>A]].
conclusion
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
12Additional Examples