<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">

<head>  <title>[2408.02666] Self-Taught Evaluators</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
  <link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
  <link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
  <meta name="msapplication-TileColor" content="#da532c">
  <meta name="theme-color" content="#ffffff">
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20240822" />
  <link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
  <script language="javascript" src="/static/browse/0.3.4/js/accordion.js" /></script>
  
    <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/slider.css?v=1.14" />
    <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript" src="/static/browse/0.3.4/js/donate.js?v=1.11"></script><link rel="canonical" href="/abs/2408.02666"/><meta property="og:type" content="website" />
<meta property="og:site_name" content="arXiv.org" />
<meta property="og:title" content="Self-Taught Evaluators" />
<meta property="og:url" content="https://arxiv.org/abs/2408.02666v2" />
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="700" />
<meta property="og:image:alt" content="arXiv logo"/>
<meta property="og:description" content="Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve. In this work, we present an approach that aims to im-prove evaluators without human annotations, using synthetic training data only. Starting from unlabeled instructions, our iterative self-improvement scheme generates contrasting model outputs and trains an LLM-as-a-Judge to produce reasoning traces and final judgments, repeating this training at each new iteration using the improved predictions. Without any labeled preference data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct) from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms commonly used LLM judges such as GPT-4 and matches the performance of the top-performing reward models trained with labeled examples."/>
<meta name="twitter:site" content="@arxiv"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Self-Taught Evaluators"/>
<meta name="twitter:description" content="Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach..."/>
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
<meta name="twitter:image:alt" content="arXiv logo"/>
  <link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/>  <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script>  <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
  <script src="/static/browse/0.3.4/js/toggle-labs.js?20230622" type="text/javascript"></script>
  <script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="Self-Taught Evaluators" /><meta name="citation_author" content="Wang, Tianlu" /><meta name="citation_author" content="Kulikov, Ilia" /><meta name="citation_author" content="Golovneva, Olga" /><meta name="citation_author" content="Yu, Ping" /><meta name="citation_author" content="Yuan, Weizhe" /><meta name="citation_author" content="Dwivedi-Yu, Jane" /><meta name="citation_author" content="Pang, Richard Yuanzhe" /><meta name="citation_author" content="Fazel-Zarandi, Maryam" /><meta name="citation_author" content="Weston, Jason" /><meta name="citation_author" content="Li, Xian" /><meta name="citation_date" content="2024/08/05" /><meta name="citation_online_date" content="2024/08/08" /><meta name="citation_pdf_url" content="http://arxiv.org/pdf/2408.02666" /><meta name="citation_arxiv_id" content="2408.02666" /><meta name="citation_abstract" content="Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve. In this work, we present an approach that aims to im-prove evaluators without human annotations, using synthetic training data only. Starting from unlabeled instructions, our iterative self-improvement scheme generates contrasting model outputs and trains an LLM-as-a-Judge to produce reasoning traces and final judgments, repeating this training at each new iteration using the improved predictions. Without any labeled preference data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct) from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms commonly used LLM judges such as GPT-4 and matches the performance of the top-performing reward models trained with labeled examples." />
</head>

<body  class="with-cu-identity">
      <aside class="slider-wrapper bps-banner forum blue">
      <a class="close-slider bps-banner" href="#"><img src="/static/browse/0.3.4/images/icons/close-slider.png" alt="close this message"></a>
      <div class="columns">
        <img role="presentation" class="bps-banner-image" src="/static/browse/0.3.4/images/icons/smileybones-infinity-large.png" alt="arXiv Accessibility Forum 2024">
        <div class="copy-donation bps-banner">
          <h2>The Accessibility Forum is back!</h2>
          <p>Coming this September, the Forum is free, virtual, and open to all. <a target="_blank" href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y">Sign Up</a> and <a href="https://accessibility2024.arxiv.org/" target="_blank">Learn more</a>.</p>
        </div>
        <div class="amount-donation bps-banner">
          <div class="donate-cta"><a class="banner_link banner-btn-grad" target="_blank" href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y"><b>Sign Up</b></a></div>
        </div>
      </div>
    </aside>
  <div class="flex-wrap-footer">
    <header>
      <a href="#content" class="is-sr-only">Skip to main content</a>
      <!-- start desktop header -->
      <div class="columns is-vcentered is-hidden-mobile" id="cu-identity">
        <div class="column" id="cu-logo">
          <a href="https://www.cornell.edu/"><img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University" /></a>
        </div><div class="column banner-minimal forum">
            <!-- /from July 7 at 1:00 AM to Sept 2 at 1:00 AM --><p>arXiv's Accessibility Forum starts next month!</p>
            <a href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y" target="_blank">Sign Up</a>

            <!-- /from Sept 2 at 1:00 AM to Sept 13 at 11:30PM --></div><div class="column" id="support-ack">
          <span id="support-ack-url">We gratefully acknowledge support from the Simons Foundation, <a href="https://info.arxiv.org/about/ourmembers.html">member institutions</a>, and all contributors.</span>
          <a href="https://info.arxiv.org/about/donate.html" class="btn-header-donate">Donate</a>
        </div>
      </div>

      <div id="header" class="is-hidden-mobile">
<a aria-hidden="true" href="{url_path('ignore_me')}"></a>
  <div class="header-breadcrumbs is-hidden-mobile">
    <a href="/"><img src="/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg" alt="arxiv logo" style="height:40px;"/></a> <span>&gt;</span> <a href="/list/cs/recent">cs</a> <span>&gt;</span> arXiv:2408.02666
  </div>
          <div class="search-block level-right">
    <form class="level-item mini-search" method="GET" action="https://arxiv.org/search">
      <div class="field has-addons">
        <div class="control">
          <input class="input is-small" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
          <p class="help"><a href="https://info.arxiv.org/help">Help</a> | <a href="https://arxiv.org/search/advanced">Advanced Search</a></p>
        </div>
        <div class="control">
          <div class="select is-small">
            <select name="searchtype" aria-label="Field to search">
              <option value="all" selected="selected">All fields</option>
              <option value="title">Title</option>
              <option value="author">Author</option>
              <option value="abstract">Abstract</option>
              <option value="comments">Comments</option>
              <option value="journal_ref">Journal reference</option>
              <option value="acm_class">ACM classification</option>
              <option value="msc_class">MSC classification</option>
              <option value="report_num">Report number</option>
              <option value="paper_id">arXiv identifier</option>
              <option value="doi">DOI</option>
              <option value="orcid">ORCID</option>
              <option value="author_id">arXiv author ID</option>
              <option value="help">Help pages</option>
              <option value="full_text">Full text</option>
            </select>
          </div>
        </div>
        <input type="hidden" name="source" value="header">
        <button class="button is-small is-cul-darker">Search</button>
      </div>
    </form>
  </div>
     </div><!-- /end desktop header -->

      <div class="mobile-header">
        <div class="columns is-mobile">
          <div class="column logo-arxiv"><a href="https://arxiv.org/"><img src="/static/browse/0.3.4/images/arxiv-logomark-small-white.svg" alt="arXiv logo" style="height:60px;" /></a></div>
          <div class="column logo-cornell"><a href="https://www.cornell.edu/">
            <picture>
              <source media="(min-width: 501px)"
                srcset="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg  400w"
                sizes="400w" />
              <source srcset="/static/browse/0.3.4/images/icons/cu/cornell_seal_simple_black.svg 2x" />
              <img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University Logo" />
            </picture>
          </a></div>
          <div class="column nav" id="toggle-container" role="menubar">
            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-white"><title>open search</title><path d="M505 442.7L405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9 0 208 0S0 93.1 0 208s93.1 208 208 208c48.3 0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9 0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7 0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7 0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <form class="mobile-search-form" method="GET" action="https://arxiv.org/search">
                <div class="field has-addons">
                  <input class="input" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
                  <input type="hidden" name="source" value="header">
                  <input type="hidden" name="searchtype" value="all">
                  <button class="button">GO</button>
                </div>
              </form>
            </div>

            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-white" role="menu"><title>open navigation menu</title><path d="M16 132h416c8.837 0 16-7.163 16-16V76c0-8.837-7.163-16-16-16H16C7.163 60 0 67.163 0 76v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16z"/ ></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <nav class="mobile-menu" aria-labelledby="mobilemenulabel">
                <h2 id="mobilemenulabel">quick links</h2>
                <ul>
                    <li><a href="https://arxiv.org/login">Login</a></li>
                    <li><a href="https://info.arxiv.org/help">Help Pages</a></li>
                    <li><a href="https://info.arxiv.org/about">About</a></li>
                </ul>
              </nav>
            </div>
          </div>
        </div>
      </div><!-- /end mobile-header -->
    </header>

    <main>
      <div id="content">
<!--
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
    <rdf:Description
        rdf:about="/abs/2408.02666"
        dc:identifier="/abs/2408.02666"
        dc:title="Self-Taught Evaluators"
        trackback:ping="/trackback/2408.02666" />
    </rdf:RDF>
--><div id="abs-outer">

  <div class="leftcolumn">
    <div class="subheader">
      <h1>Computer Science > Computation and Language</h1>
    </div>

    <div class="header-breadcrumbs-mobile">
      <strong>arXiv:2408.02666</strong> (cs)
    </div>
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
<div id="content-inner">
  <div id="abs">
    <div class="dateline">
  [Submitted on 5 Aug 2024 (<a href="https://arxiv.org/abs/2408.02666v1">v1</a>), last revised 8 Aug 2024 (this version, v2)]</div>
    <h1 class="title mathjax"><span class="descriptor">Title:</span>Self-Taught Evaluators</h1>
    <div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Wang,+T">Tianlu Wang</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Kulikov,+I">Ilia Kulikov</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Golovneva,+O">Olga Golovneva</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Yu,+P">Ping Yu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Yuan,+W">Weizhe Yuan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Dwivedi-Yu,+J">Jane Dwivedi-Yu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Pang,+R+Y">Richard Yuanzhe Pang</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Fazel-Zarandi,+M">Maryam Fazel-Zarandi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Weston,+J">Jason Weston</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Li,+X">Xian Li</a></div>            <div id="download-button-info" hidden>View a PDF of the paper titled Self-Taught Evaluators, by Tianlu Wang and 9 other authors</div>
    <a class="mobile-submission-download" href="/pdf/2408.02666">View PDF</a>
    <a class="mobile-submission-download" href="https://arxiv.org/html/2408.02666v2">HTML (experimental)</a>



    <blockquote class="abstract mathjax">
            <span class="descriptor">Abstract:</span>Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve. In this work, we present an approach that aims to im-prove evaluators without human annotations, using synthetic training data only. Starting from unlabeled instructions, our iterative self-improvement scheme generates contrasting model outputs and trains an LLM-as-a-Judge to produce reasoning traces and final judgments, repeating this training at each new iteration using the improved predictions. Without any labeled preference data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct) from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms commonly used LLM judges such as GPT-4 and matches the performance of the top-performing reward models trained with labeled examples.
    </blockquote>

    <!--CONTEXT-->
    <div class="metatable">
      <table summary="Additional metadata"><tr>
          <td class="tablecell label">Subjects:</td>
          <td class="tablecell subjects">
            <span class="primary-subject">Computation and Language (cs.CL)</span>; Artificial Intelligence (cs.AI)</td>
        </tr><tr>
          <td class="tablecell label">Cite as:</td>
          <td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2408.02666">arXiv:2408.02666</a> [cs.CL]</span></td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxividv">(or <span class="arxivid">
              <a href="https://arxiv.org/abs/2408.02666v2">arXiv:2408.02666v2</a> [cs.CL]</span> for this version)
          </td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxivdoi">              <a href="https://doi.org/10.48550/arXiv.2408.02666"  id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2408.02666</a><div class="button-and-tooltip">
              <button class="more-info" aria-describedby="more-info-desc-1">
                <svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
                <span class="visually-hidden">Focus to learn more</span>
              </button>
              <!-- tooltip description -->
              <div role="tooltip" id="more-info-desc-1">
                <span class="left-corner"></span>                  arXiv-issued DOI via DataCite</div>
            </div>
          </td>
        </tr></table>
    </div>
  </div>
</div>
    <div class="submission-history">
      <h2>Submission history</h2> From: Jason Weston [<a href="/show-email/6e218214/2408.02666">view email</a>]      <br/>            <strong><a href="/abs/2408.02666v1">[v1]</a></strong>
        Mon, 5 Aug 2024 17:57:02 UTC (1,171 KB)<br/>
    <strong>[v2]</strong>
        Thu, 8 Aug 2024 17:09:58 UTC (1,173 KB)<br/>
</div>
  </div>
  <!--end leftcolumn-->
<div class="extra-services">    <div class="full-text">
      <a name="other"></a>
      <span class="descriptor">Full-text links:</span>
      <h2>Access Paper:</h2>
      <ul>
  <div id="download-button-info" hidden>
View a PDF of the paper titled Self-Taught Evaluators, by Tianlu Wang and 9 other authors</div><li><a href="/pdf/2408.02666" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2408.02666v2" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2408.02666" class="abs-button download-eprint">TeX Source</a></li><li><a href="/format/2408.02666" class="abs-button download-format">Other Formats</a></li></ul>
      <div class="abs-license"><a href="http://creativecommons.org/publicdomain/zero/1.0/" title="Rights to this article" class="has_license">
          <img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/zero-1.0.png"/>
          <span>view license</span>
        </a></div>
    </div>
    <!--end full-text-->    <div class="browse">
    Current browse context: <div class="current">cs.CL</div>

  <div class="prevnext">
<span class="arrow">
      <a class="abs-button prev-url" href="/prevnext?id=2408.02666&amp;function=prev&amp;context=cs.CL"
         accesskey="p" title="previous in cs.CL (accesskey p)">&lt;&nbsp;prev</a>
    </span>
    <span class="is-hidden-mobile">&nbsp; | &nbsp;</span>    <span class="arrow">
      <a class="abs-button next-url" href="/prevnext?id=2408.02666&amp;function=next&amp;context=cs.CL" accesskey="n"
         title="next in cs.CL (accesskey n)">next&nbsp;&gt;</a>
    </span><br/>
  </div><div class="list">
    <a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.CL/new">new</a>
    <span class="is-hidden-mobile"> | </span>
    <a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.CL/recent">recent</a>
    <span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.CL/2024-08">2024-08</a>
  </div><div class="abs-switch-cat">
    Change to browse by:
    <div class="switch context-change">
        <a href="/abs/2408.02666?context=cs">cs</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.02666?context=cs.AI">cs.AI</a><br class="is-hidden-mobile">
    </div>
  </div>

    </div>

    <div class="extra-ref-cite">
      <h3>References &amp; Citations</h3>
      <ul>
        <li><a  class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2408.02666">NASA ADS</a></li><li><a  class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2408.02666" target="_blank" rel="noopener">Google Scholar</a></li>
        <li><a  class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2408.02666" target="_blank" rel="noopener">Semantic Scholar</a></li>
      </ul>
      <div style="clear:both;"></div>
    </div>

<div class='extra-ref-cite'>
    <a id='bib-cite-css' hidden='true' href='/static/browse/0.3.4/css/cite.css'>a</a>

    <span id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</span>
    <span id='bib-cite-loading' hidden='true'>Loading...</span>
</div>

<div id='bib-cite-modal' class='bib-modal' hidden='true'>
    <div class='bib-modal-content'>
        <div class='bib-modal-title'>
            <h2>BibTeX formatted citation</h2>
            <span class='bib-modal-close' >&times;</span>
        </div>
        <div>
            <textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
        </div>
        <div>
            <span>Data provided by: </span>
            <a id='bib-cite-source-api'></a>
        </div>
    </div>
</div><div class="bookmarks">
  <div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2408.02666&amp;description=Self-Taught Evaluators"
     title="Bookmark on BibSonomy">
    <img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
         alt="BibSonomy logo"/>
  </a>
  <a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2408.02666&amp;title=Self-Taught Evaluators"
     title="Bookmark on Reddit">
    <img src="/static/browse/0.3.4/images/icons/social/reddit.png"
         alt="Reddit logo"/>
  </a>
</div>  </div>
  <!--end extra-services-->
<!-- LABS AREA -->
<div id="labstabs">
  <div class="labstabs"><input type="radio" name="tabs" id="tabone"checked="checked">
    <label for="tabone">Bibliographic Tools</label>
    <div class="tab labs-display-bib">
      <h1>Bibliographic and Citation Tools</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input id="bibex-toggle" type="checkbox" class="lab-toggle">
              <span class="slider"></span>
              <span class="is-sr-only">Bibliographic Explorer Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
          </div>
        </div>
<div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="litmaps-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
                aria-labelledby="label-for-litmaps">
              <span class="slider"></span>
              <span class="is-sr-only">Litmaps Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
          </div>
        </div>
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="scite-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
                aria-labelledby="label-for-scite">
              <span class="slider"></span>
              <span class="is-sr-only">scite.ai Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
          </div>
        </div>
      </div>
        <div class="labs-content-placeholder labs-display" style="display: none;"></div>
        <div style="min-height: 15px" id="litmaps-open-in"></div>
        <div style="min-height: 15px" id="scite-open-in"></div>
    </div>


    <input type="radio" name="tabs" id="tabtwo">
    <label for="tabtwo">Code, Data, Media</label>
    <div class="tab">
      <h1>Code, Data and Media Associated with this Article</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input        
                id="catalyzex-toggle"
                data-script-url="/static/browse/0.3.4/js/catalyzex.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="dagshub-toggle"
                data-script-url="/static/browse/0.3.4/js/dagshub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
              <span class="slider"></span>
              <span class="is-sr-only">DagsHub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
          </div>
        </div>
  
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="gotitpub-toggle"
                data-script-url="/static/browse/0.3.4/js/gotitpub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
              <span class="slider"></span>
              <span class="is-sr-only">GotitPub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="paperwithcode-toggle"
                data-script-url="/static/browse/0.3.4/js/paperswithcode.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-pwc">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-pwc">Papers with Code</span> <em>(<a href="https://paperswithcode.com/" target="_blank">What is Papers with Code?</a>)</em>
          </div>
        </div>


        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="sciencecast-toggle"
                data-script-url="/static/browse/0.3.4/js/sciencecast.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
              <span class="slider"></span>
              <span class="is-sr-only">ScienceCast Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
          </div>
        </div>
      </div>

      <div id="catalyzex-output" style="display:none"></div>
      <div id="dagshub-output" style="display:none"></div>
      <div id="gotitpub-output" style="display:none"></div>
      <div id="pwc-output" style="display:none"></div>
      <div id="pwc-data-output" style="display:none"></div>
      <div id="sciencecast-output" style="display:none"></div>
    </div>


      <input type="radio" name="tabs" id="labstabs-demos-input">
      <label for="labstabs-demos-input" id="labstabs-demos-label">Demos</label>
      <div class="tab">
        <h1>Demos</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="replicate-toggle"
                  data-script-url="/static/browse/0.3.4/js/replicate.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
                <span class="slider"></span>
                <span class="is-sr-only">Replicate Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="spaces-toggle"
                  data-script-url="/static/browse/0.3.4/js/spaces.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="txyz-toggle"
                  data-script-url="/static/browse/0.3.4/js/txyz.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
            </div>
          </div>
        </div>
        <div id="replicate-output"></div>
        <div id="spaces-output"></div>
        <div id="txyz-output"></div>
      </div>
      <input type="radio" name="tabs" id="tabfour">
      <label for="tabfour">Related Papers</label>
      <div class="tab">
        <h1>Recommenders and Search Tools</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="influenceflower-toggle"
                data-script-url="/static/browse/0.3.4/js/influenceflower.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
                <span class="slider"></span>
                <span class="is-sr-only">Link to Influence Flower</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="connectedpapers-toggle" type="checkbox" class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
                       aria-labelledby="label-for-connected-papers" >
                <span class="slider"></span>
                <span class="is-sr-only">Connected Papers Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
                <span class="slider"></span>
                <span class="is-sr-only">Core recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
            </div>
          </div></div>
        <div id="influenceflower-output"></div>
        <div id="influenceflower-output-graph" style="display:none">
          <ul class="flower-tabs">
            <li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
          </ul>
          <div class="flower-tab-content">
            <div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
            <div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
            <div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
            <div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
          </div>
        </div>
        <div style="min-height: 15px" id="connectedpapers-output"></div>
        <div id="coreRecommenderOutput"></div>
        <div id="iarxivOutput"></div>
      </div>

      <input type="radio" name="tabs" id="tabfive">
      <label for="tabfive">
        About arXivLabs
      </label>
      <div class="tab">
        <div class="columns">
          <div class="column">
            <h1>arXivLabs: experimental projects with community collaborators</h1>
            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
            <p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
          </div>
          <div class="column is-narrow is-full-mobile">
            <p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
          </div>
        </div>
      </div>

    </div>
</div>
<!-- END LABS AREA -->
  <div class="endorsers">
    <a href="/auth/show-endorsers/2408.02666" class="endorser-who">Which authors of this paper are endorsers?</a> |
    <a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
    <span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
  </div>
  <script type="text/javascript" language="javascript">mathjaxToggle();</script>
</div>
      </div>
    </main>

    <footer style="clear: both;">
      <div class="columns is-desktop" role="navigation" aria-label="Secondary" style="margin: -0.75em -0.75em 0.75em -0.75em">
        <!-- Macro-Column 1 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/about">About</a></li>
                <li><a href="https://info.arxiv.org/help">Help</a></li>
              </ul>
            </div>
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>contact arXiv</title><desc>Click here to contact arXiv</desc><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>
                  <a href="https://info.arxiv.org/help/contact.html"> Contact</a>
                </li>
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>subscribe to arXiv mailings</title><desc>Click here to subscribe</desc><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg>
                  <a href="https://info.arxiv.org/help/subscribe"> Subscribe</a>
                </li>
              </ul>
            </div>
          </div>
        </div>
        <!-- End Macro-Column 1 -->
        <!-- Macro-Column 2 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/license/index.html">Copyright</a></li>
                <li><a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy Policy</a></li>
              </ul>
            </div>
            <div class="column sorry-app-links">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/web_accessibility.html">Web Accessibility Assistance</a></li>
                <li>
                  <p class="help">
                    <a class="a11y-main-link" href="https://status.arxiv.org" target="_blank">arXiv Operational Status <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 512" class="icon filter-dark_grey" role="presentation"><path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"/></svg></a><br>
                    Get status notifications via
                    <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/email/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>email</a>
                    or <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/slack/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-black" role="presentation"><path d="M94.12 315.1c0 25.9-21.16 47.06-47.06 47.06S0 341 0 315.1c0-25.9 21.16-47.06 47.06-47.06h47.06v47.06zm23.72 0c0-25.9 21.16-47.06 47.06-47.06s47.06 21.16 47.06 47.06v117.84c0 25.9-21.16 47.06-47.06 47.06s-47.06-21.16-47.06-47.06V315.1zm47.06-188.98c-25.9 0-47.06-21.16-47.06-47.06S139 32 164.9 32s47.06 21.16 47.06 47.06v47.06H164.9zm0 23.72c25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06H47.06C21.16 243.96 0 222.8 0 196.9s21.16-47.06 47.06-47.06H164.9zm188.98 47.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06h-47.06V196.9zm-23.72 0c0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06V79.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06V196.9zM283.1 385.88c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06v-47.06h47.06zm0-23.72c-25.9 0-47.06-21.16-47.06-47.06 0-25.9 21.16-47.06 47.06-47.06h117.84c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06H283.1z"/></svg>slack</a>
                  </p>
                </li>
              </ul>
            </div>
          </div>
        </div> <!-- end MetaColumn 2 -->
        <!-- End Macro-Column 2 -->
      </div>
    </footer>
  </div>

  <script src="/static/base/1.0.1/js/member_acknowledgement.js"></script>

</body>

</html>
2408.02666
 Model-based evaluation is at the heart of successful model development - as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve. In this work, we present an approach that aims to improve evaluators *without human annotations*, using synthetic training data only. Starting from unlabeled instructions, our iterative selfimprovement scheme generates contrasting model outputs and trains an LLM-as-a-Judge to produce reasoning traces and final judgments, repeating this training at each new iteration using the improved predictions. Without any labeled preference data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70BInstruct) from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms commonly used LLM judges such as GPT-4 and matches the performance of the top-performing reward models trained with labeled examples. 
abstract
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Abstract
2408.02666
 Large language models (LLMs) rely on strong evaluators at every stage of the development lifecycle. They are used at training time as reward models to align with human preferences (Bai et al., 2022; Ouyang et al., 2022) or for iterative selfimprovement (Yuan et al., 2024), and at inference time as an alternative to human evaluation (Li et al., 2023; Chiang and Lee, 2023; Wang et al., 2023a; Liu et al., 2023). Improvements in evaluation capabilities will thus clearly benefit this entire workflow - including empowering the scientific research process itself as we aim to develop better overall techniques.  Building such strong evaluator models usually relies on large amounts of high-quality preference data from human annotation over model responses, which can be costly and time-consuming to collect, as it requires expert annotation for challenging tasks (e.g., coding and mathematics). This dependency on human-generated data poses significant challenges for scaling to new tasks or evaluation criteria. Furthermore, as new models inevitably improve over older ones, these existing annotations will typically become outdated, as the judgments are based on annotations of older, less performant, model responses.  In this work, we instead explore an iterative selftraining approach which uses *no human annotated* preferences in the training loop, relying purely on synthetically generated data. Given a seed model, our method first uses prompting to generate contrasting synthetic preference pairs for a given input, such that one response is designed to be inferior to the other. Next, using the model as an LLM-as-aJudge, we generate reasoning traces and judgments for these pairs, which we can label as correct or not given our synthetic preference pair design. After training on this labeled data we obtain a superior LLM-as-a-Judge, from which we can then iterate the whole process in order for it to self-improve.  In our experiments, starting from Llama-3-70BInstruct, the proposed method improves the accuracy on RewardBench (Lambert et al., 2024) from 75.4 to 88.7 (with majority vote, or 88.3 without).  This matches or outperforms the performance of reward models derived from the same Llama-370B-Instruct model that uses human annotations, for example using the HelpSteer2 dataset (Wang et al., 2024b) of 10k annotations achieves a performance of 85.6 using the same LLM-as-a-Judge setup. 
introduction
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Introduction
2408.02666
 LLM-based Evaluators While traditional evaluation benchmarks employ automated metrics that require a reference answer (Wang et al., 2019; Ra-  ![1_image_0.png](1_image_0.png)  jpurkar et al., 2016), these types of benchmarks can pose severe limitations when evaluating openended or complex instructions where multiple valid answers are possible (e.g., creative writing and coding). Because human evaluation per response can be costly, many recent works have proposed LLMs as effective evaluators. These come in several flavors: as classifiers that output scores directly (Zhu et al., 2023; Wang et al., 2024a) or via *LLM-as-aJudge* prompting that can first generate a chain-ofthought in natural language, which helps provide explanations for judgments (Zheng et al., 2023). Responses can also be scored alone (Kim et al., 2023) or pairwise relative to each other (Dubois et al., 2023; Li et al., 2023; Bai et al., 2023; Saha et al., 2024). LLM evaluation shows great promise as a scalable proxy for human raters, and in the case of LLM-as-a-Judge as an explainable proxy as well (Ye et al., 2024; Zheng et al., 2023). However, many of these "off-the-shelf" evaluators demonstrate high variance across many tasks (Bavaresco et al., 2024), indicating the need for improved methods.  Synthetic Data Synthetic data has emerged as a promising solution for efficiently acquiring training examples and can be particularly valuable in settings where real-world data can be hard to access (e.g., weather data covering all conditions (Lam et al., 2023)) or where correct annotations can be challenging to acquire (e.g., coding tasks (Liu et al., 2024)). Additionally, synthetic data has the benefit of being easily customizable to specific requirements, such as different evaluation criteria or safety constraints (Kim et al., 2023; El Emam et al., 2020; Howe et al., 2017). The use of synthetic data has been beneficial in model alignment (Lee et al., 2023), improving the original model's capabilities (Yuan et al., 2024; Li et al., 2024a; Yu et al., 2024; Li et al., 2024b), and teaching the model new skills (Schick et al., 2023; Lanchantin et al., 2023). In the context of evaluation, synthetic data has been used to measure tasks such as factuality (Wei et al., 2024; Feng et al., 2023), safety (Perez et al., 2023; Hubinger et al., 2024), coding (Gu et al., 2024), and general instruction following (Zeng et al., 2024), showing strong correlation with real human judgments. The West-of-n approach (Pace et al., 2024) has been used to improve reward models by constructing preference pairs using the best and worst scoring pairs from an initial model. For LLM-as-a-Judge models specifically, synthetic responses have been generated prompting the LLM to produce a given quality response (Kim et al., 2023). 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Related Work
2408.02666
 We consider the setting of pairwise evaluation using the LLM-as-a-Judge approach (Zheng et al., 2023) that takes: - an input (user instruction) x; and  - two possible assistant responses y (A)and y (B) to the user instruction x; and - the evaluation prompt containing the rubric and asking to evaluate and choose the winning answer, see e.g., Figure 8. The goal of the LLM-as-a-Judge model is to output a preference of which response y is better: A or B. In order to do this it is common to output, prior to the final judgment, a chain-of-thought (or "reasoning chain"), which is a set of steps generated in natural language that helps the model decide its final judgment.  Such models can be used as pairwise reward models to build training data for preference optimization, e.g., for training methods like DPO (Rafailov et al., 2023), Iterative DPO (Xu et al., 2023) and Self-Rewarding methods (Yuan et al., 2024). They can also be used for evaluation; e.g., many popular benchmark leaderboards are built by using a fixed LLM-as-a-Judge evaluation model (Li et al., 2023) such as GPT4 (Achiam et al., 2023).  We propose a novel recipe for training such an evaluator. Our overall method is an iterative training scheme that bootstraps improvements by annotating the current model's judgments using constructed synthetic data - so that the Self-Taught Evaluator is more performant on the next iteration.  Our overall pipeline is thus as follows: - **Initialization:** We assume access to a large set of human-written user instructions, e.g., of the type that is commonly collected in production systems, and an initial seed LLM.  - **Instruction Selection:** We next select a challenging, balanced distribution of user instructions from the uncurated set by categorizing them via LLM.  - **Response Pair Construction:** For each user instruction (example) we create a preference pair of two model responses (chosen & rejected), generating them via prompting such that the rejected response is likely of lower quality than the chosen response.  - **Iterative Training:** We then iterate the following two steps: (i) **Judgment Annotation:** For each example, we sample from the current model up to N times LLM-as-a-Judge generated reasoning traces and judgments. If we find a correct judgment we add that example to our training set, otherwise we discard it.  (ii) **Model Fine-tuning:** We fine-tune the model on the newly constructed training set which yields an updated model for the next iteration. Note that in each iteration of training the size of the training set depends on the quality of the current model. We expect that as the model improves, the size of the training set will increase as well, as the model will be able to find more correct judgments, giving the model a kind of automatic curriculum.  We next describe each of these steps in detail. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Method
2408.02666
 We assume we have access to a pool of user instructions {xi}. Each sample xi can either be one single text instruction or a multi-turn dialog history of turns between the user and the assistant, with the last turn being an instruction or question from the user. Instructions typically involve different skills such as general knowledge and reasoning, coding, safety, and mathematical reasoning. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Initialization
2408.02666
 Given a pool of human-written user instructions, there may be a large degree of noise, as well as an imbalance in terms of topic, variety, difficulty, and ability of the model to answer. We therefore aim to select a subset of instructions to generate highquality synthetic responses and judgments that can be further used for training.  We classify each input using an LLM into a given category, for example coding, reasoning, brainstorming, etc. The precise prompt we use is given in Figure 7. We are then free to select data from within those categories, and to discard certain categories not deemed to be useful for training. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Instruction Selection
2408.02666
 For each input xiin our curated training pool, we next generate preference data involving two responses y (w) iand y (l) i where w is expected to be preferable (winning) over l (losing). We achieve this by generating the data in a synthetic manner without using human annotation.  Given the instruction xi, we first prompt an instruction-following LLM to generate a baseline response y w ias usual. We then prompt the LLM to generate a "noisy" version of the original instruction x ′ i = ϕ(xi). We do this using the prompt template given in Figure 2, where we ask to "generate a modified instruction that is highly relevant but not semantically identical to the instruction above from the user." We then prompt the LLM for a high-quality response y l i to x ′ i , which would not be a good response for xi. This yields a synthetic preference y w i ≻ y l i for the original input xi.  This paired data is then used to construct training examples: (xi, y (A) i, y (B) i) where we randomize the order of whether the winner is w = A or w = B, which is important to deal with position bias for LLM-as-a-Judge inference.  ![3_image_0.png](3_image_0.png)  4 Figure 2: **Generating Synthetic Response Pairs.** We use the following prompt template which is used to generate a "worse response" y l. Given an instruction x and baseline response y w generated by an instruction-following LLM as usual, this prompt is used to first generate a "noisy" version x ′ of the original instruction x, and then a best-attempt y lat responding to x ′. y lis then treated as a poor response to x, giving a preference pair y w i ≻ y l i . 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Response Pair Construction
2408.02666
 Our LLM-as-a-Judge model is used to generate evaluation judgments (reasoning chains and verdicts) {ji} for each training example ei:= (xi, y (A) i, y (B) i) in the following manner: for a given input ei, we collect N diverse evaluations J := {j 1 i , . . . , jN i } by sampling from the model.  We then apply rejection sampling to filter J by removing j n i when the final verdict disagrees with the ground truth labeling, derived from Subsection 3.3.  We then select a single correct reasoning chain and verdict at random from the pool of correct solutions.  If no such judgment exists (J is empty) then we discard the example.  This now allows us to construct our final training examples of synthetic preferences for fine-tuning: 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Judgment Annotation
2408.02666
 Our Self-Taught Evaluator (LLM-as-a-Judge model) is first initialized with the seed LLM. The model is then trained in an iterative manner. At each iteration, we annotate the training examples with judgments as described in Subsection 3.4 using the current model, giving training examples {(xi, y (A) i, y (B) i, ji)}. These are used to train the next iteration's model by fine-tuning. Note that we initialize from the seed model at each iteration. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
((Xi, Y (A) I, Y (B) I), Ji). 3.5 Model Fine-Tuning (Iterative Training)
2408.02666
 Training. Our initial model M0 is initialized from Llama3-70B-Instruct. In each iteration i = 1*, . . . T*, we use model Mi−1 from the previous iteration to generate synthetic preferences followed by judgments on the training data, and then finetune Llama3-70B-Instruct again. We use fairseq2 library (Balioglu, 2023) for instruction finetuning and vLLM (Kwon et al., 2023) for inference. During training the negative log-likelihood loss is only applied to the evaluation part, i.e., ji of the training example. Training hyperparameters are provided in Table 7. Model selection is done using a combination of pairwise judgment accuracy and position bias computed over the held out set. Sampling parameters used for generations are provided in Table 8.  Instructions and Responses. We start with a large pool of human-written instructions {xi} from the WildChat dataset (Zhao et al., 2024). To perform prompt selection, we annotate the category of each instruction with the Mixtral 22Bx8 Instruct model, using the template in Figure 7 and select 20,582 examples in the reasoning category, as we expect these to be challenging inputs. For the selected inputs we generate synthetic responses y w i and y l i using Mixtral 22Bx8 Instruct following Subsection 3.3 and Figure 2.  Judge Annotation. For each training example, we sample N = 15 judgments from the model Mi−1 and retain one positive sample ji per example. Then over the entire dataset we sample the same amount of examples from different labels ("A is better", "B is better") to ensure balanced training.  Judgements for training M0 were sampled from Mixtral 22Bx8 Instruct, and from the Llama model being trained in all subsequent iterations.  The training data is constructed as (<system prompt>, {(xi, y (A) i, y (B) i, ji)}). We use the standard system prompt from MT-Bench and RewardBench as shown in Figure 8.  Majority Vote Inference. As LLM-as-a-Judge uses chain-of-though reasoning chains generated by the LLM followed by a verdict, it is known that majority vote inference can yield improvements in these cases (Wang et al., 2023b). At inference time when evaluating final performance we sample generations N times, and take the final judgment to be the most common verdict. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Experiments 4.1 Experimental Setup
2408.02666
 To understand the effectiveness of the proposed method, we generate synthetic judgments using the same approach but based on the following data sources:  - HelpSteer2 (Wang et al., 2024b). We generate evaluations conditioned on the scores of helpfulness, correctness, coherence, complexity and verbosity provided the dataset.  We use the aggregated score to derive the ground truth preference for each example using the recommended weighting [0.65, 0.8, 0.45, 0.55, −0.4]1.  - GSM8K (Cobbe et al., 2021). We sample from an instruction-following model multiple times to get y w when the final solution agrees with the ground truth and y l vise versa.  - Coding instructions from WildChat. Similar to the "reasoning" prompts we selected from WildChat used in the main experiment, we also experimented with prompts annotated with the "Coding" category.  - hh_rlhf (Bai et al., 2022). We generate evaluations on the prompts and responses provided in the "harmless_base" training split. Then we take human preferences provided by the dataset as ground truth to perform rejection sampling to construct judgments. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Other Data Sources
2408.02666
 We evaluate the accuracy of our Self-Taught Evaluator model on the following benchmarks:  - RewardBench (Lambert et al., 2024). We use the standard evaluation protocol provided by the leaderboard.  - MT-Bench (Zheng et al., 2023). We report agreement rate with human judgments when examples with ties are excluded.  - HelpSteer2 (Wang et al., 2024b). We evaluate on the validation split. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Evaluation
2408.02666
 Results on RewardBench are given in Table 1. We find that our Self-Taught Evaluator which is trained iteratively on synthetic data *without* any annotated preference labels significantly improves over the seed Llama3-70B-Instruct model, matching topperforming reward models trained *with* labeled data. Our approach improves its results across training iterations, and achieves an overall score of 88.3 on iteration 5, while the seed model it starts from obtains 75.4. Training an LLM-as-a-Judge in a similar manner starting from the same seed using the labeled HelpSteer2 data we only obtain 85.6, hence we obtain superior performance *without using human labeled data*. Compared to the seed model, we observe improvements using our approach in evaluating instructions in the Chat Hard, Safety and Reasoning categories, while being worse on the easier Chat category - perhaps because our unlabeled training data focused the model on harder examples. Improving results further with majority voting As also shown in Table 1, with 32-sample majority voting, our third iteration of Self-Taught Evaluator model reaches an overall performance of 88.7 on RewardBench, outperforming many other existing reward models. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Results 5.1 Rewardbench
2408.02666
 We report results on MT-Bench in Table 2. Unlike RewardBench, the MT-Bench dataset contains tie 5  | Model                                                                          | Overall   | Chat   | Chat Hard   | Safety   | Reasoning   | |--------------------------------------------------------------------------------|-----------|--------|-------------|----------|-------------| | Llama-3-70B-Instruct (seed)                                                    | 75.4      | 97.6   | 58.9        | 69.2     | 78.5        | | Self-Taught Evaluator, trained on synthetic data only Iteration 1              | 83.9      | 98.3   | 69.0        | 85.7     | 82.6        | | Iteration 2                                                                    | 86.0      | 97.5   | 75.4        | 89.5     | 81.7        | | Iteration 3                                                                    | 87.5      | 97.2   | 79.1        | 89.7     | 83.9        | | Iteration 4                                                                    | 87.7      | 98.0   | 80.3        | 90.5     | 82.2        | | Iteration 5                                                                    | 88.3      | 96.6   | 84.2        | 91.5     | 81.0        | | w/ majority voting using 32 samples                                            | 88.7      | 96.9   | 84.0        | 91.5     | 82.5        | | Baselines with Labeled Data Llama-3-70B-Instruct w/ HelpSteer2, LLM-as-a-Judge | 85.6      | 96.9   | 70.0        | 88.8     | 86.7        | | nvidia/Llama3 70B RM with HelpSteer2, classifier *                             | 88.8      | 91.3   | 80.3        | 92.8     | 90.7        | | Other SoTA LLM-as-a-Judge baseline models GPT4 0125 *                          | 84.3      | 95.3   | 74.3        | 87.2     | 86.9        | | Gemini 1.5 Pro 0514 *                                                          | 88.1      | 92.3   | 80.6        | 87.5     | 92.0        | | Llama3.1-405B-Instruct                                                         | 83.7      | 98.0   | 75.1        | 74.7     | 86.8        | | Llama3.1-70B-Instruct                                                          | 82.2      | 97.8   | 69.7        | 76.3     | 85.2        |  Table 1: **RewardBench Results**. Our Self-Taught Evaluator trained on synthetic data without any human annotated preference labels matches top-performing reward models trained with labeled data. Models marked with (*) are taken from the RewardBench leaderboard.  | Model                                                                                                                                                                                                                                        | Agreement with Human   | |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------| | Llama-3-70B-Instruct (seed)                                                                                                                                                                                                                  | 77.8                   | | Self-Taught Evaluator, trained on synthetic data only Iteration 1 79.0 Iteration 2 78.7 Iteration 3 78.9 Iteration 4 77.5 Iteration 5 78.9 w/ majority voting using 32 samples 79.5 Other SoTA LLM-as-a-Judge baseline models GPT4-0125 79.1 |                        |  Table 2: **MT-Bench Results**. Our Self-Taught Evaluator trained on synthetic data without any human annotated preference labels performs on par with GPT-4 judgments.  | Model                                                                       | 0-1 Acc   | 1-0 Acc   | Avg Acc   | Position-consistent Acc   | |-----------------------------------------------------------------------------|-----------|-----------|-----------|---------------------------| | Llama-3-70B-Instruct (seed)                                                 | 65.2      | 65.8      | 65.5      | 56.5                      | | Self-Taught Evaluator, trained on synthetic data only Iteration 1 68.1 68.7 | 68.4      | 59.4      |           |                           | | Iteration 2                                                                 | 69.6      | 69.4      | 69.5      | 58.8                      | | Iteration 3                                                                 | 70.3      | 71.2      | 70.8      | 61.1                      | | Iteration 4                                                                 | 71.0      | 71.7      | 71.4      | 61.9                      | | Iteration 5                                                                 | 71.6      | 70.3      | 71.0      | 60.6                      |  Table 3: **HelpSteer2 results**. Iterative training on synthetic preferences improves position-consistent accuracy compared to Llama3-70B-Instruct, measured on the HelpSteer2 (Wang et al., 2024b) validation split.  Source for  Model synthetic preferences Overall Chat Chat Hard Safety Reasoning  Llama-3-70B-Instruct 75.4 **97.6** 58.9 69.2 78.5  safety (hh_rlhf) 79.6 97.2 55.4 **87.0** 78.8 math (GSM8K) 79.3 96.1 58.8 79.4 **83.0** coding (WildChat) 79.4 96.6 55.9 85.3 79.7 reasoning (WildChat) 83.5 97.5 **70.6** 84.2 81.6  Table 4: Supervised fine-tuning with synthetic preferences from different sources improves Llama-3-70B-Instruct on various categories, as measured on RewardBench. Largest improvement in each category is highlighted in bold.  votes (A and B are considered equally good). Since our models are trained to give binary decisions, we only report the agreement on non-tie examples. For each pair of responses A and B, we test two orders: where response A appears first and response B appears first, and average the results. We find that our Self-Taught Evaluator again outperforms the Llama3-70B-Instruct seed model, and is on par or slightly outperforms GPT4-0125. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Mt-Bench
2408.02666
 Results on the HelpSteer2 validation set are given in Table 3 . We report the average accuracy of two orders and three seeds by swapping the response order in a similar manner, as well as reporting both orders separately (right answer first or second) to test for position bias. We further compute the position-consistent accuracy, treating a judgment as incorrect when a model has different predictions on the two orderings. We use the human labels from the Helpsteer2 dataset and treat the response with higher summed scores as the better response.  We find that our Self-Taught Evaluator method improves both average accuracy and position-consistent accuracy compared to the seed Llama-3-70B-Instruct model. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Helpsteer2
2408.02666
 In Table 4 , we compare Self-Taught Evaluator models trained on synthetic preferences constructed from different sources.  We found data sources focusing on different skills, such as coding, mathematical reasoning, etc. are all effective in turning a strong instruction-following LLM into a strong LLM-as-a-Judge.  Intuitively, we find that data sources generally improve the categories in RewardBench that are related to their distribution. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Ablations And Analysis 6 Synthetic Data From Other Sources 6.1
2408.02666
 In our experiments we generate synthetic data by first generating a similar instruction, and then a good response for the similar instruction - with the aim that this will be a bad response for the original instruction.  An alternative is to just prompt an LLM to generate a bad response to the original instruction directly.  We use the prompt template given in Figure 10 and otherwise conduct training as before on the same set of reasoning-based instructions. This approach obtains a RewardBench overall score of 80.7, which still works - but is  ![6_image_0.png](6_image_0.png)  ![6_image_1.png](6_image_1.png)  ![6_image_2.png](6_image_2.png)  ![6_image_3.png](6_image_3.png)  | Model                                                                                       | Overall   | Chat   | Chat Hard   | Safety   | Reasoning   | |---------------------------------------------------------------------------------------------|-----------|--------|-------------|----------|-------------| | Llama-3-70B-Instruct (seed)                                                                 | 75.4      | 97.6   | 58.9        | 69.2     | 78.5        | | Self-Taught Evaluator, trained on labeled HelpSteer2 preferences Iteration 1 85.6 96.9 70.0 | 88.8      | 86.7   |             |          |             | | Iteration 2                                                                                 | 86.3      | 96.1   | 72.4        | 91.1     | 85.7        | | Iteration 3                                                                                 | 87.0      | 95.0   | 74.2        | 91.2     | 87.8        | | Iteration 4                                                                                 | 87.0      | 94.1   | 77.2        | 91.6     | 85.1        |  Table 5: Iterative training with labeled data also shows improvement on RewardBench. However, it does not outperform iterative training with synthetic preferences .  | synthetic:HelpSteer2 ratio   | Overall   | Chat   | Chat Hard   | Safety   | Reasoning   | |------------------------------|-----------|--------|-------------|----------|-------------| | 1 : 0                        | 0.835     | 0.975  | 0.706       | 0.842    | 0.816       | | 0 : 1                        | 0.856     | 0.969  | 0.700       | 0.888    | 0.867       | | 1 : 1                        | 0.842     | 0.972  | 0.681       | 0.881    | 0.836       | | 1 : 2                        | 0.858     | 0.972  | 0.711       | 0.891    | 0.857       | | 1 : 5                        | 0.847     | 0.975  | 0.681       | 0.889    | 0.844       | | 2 : 1                        | 0.833     | 0.972  | 0.689       | 0.847    | 0.823       | | 5 : 1                        | 0.858     | 0.972  | 0.726       | 0.880    | 0.853       |  worse than using our proposed approach, which achieves 83.8. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Synthetic Bad Response Generation 6.2
2408.02666
 We conducted the same iterative training using labeled preference data from HelpSteer2 (Wang et al., 2024b), rather than synthetic data. On RewardBench, as is shown in Table 5, the improvement from each iteration is smaller and the final model did not outperform iterative training on synthetic preferences. We note that these experiments use data to train an LLM-as-a-Judge. Other results in the literature have used the HelpSteer2 to train classifier-based reward models with slightly better results on RewardBench, e.g., obtaining 88.8 using Llama-3-70B, see Table 1. 
references
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
**Combining Synthetic And Human Labeled** Preference Data 6.3 Comparison Of Synthetic Data With Human Annotated Data
2408.02666
 We further explore how to utilize labeled data in our pipeline. We first finetune a model on Helpsteer2 (Wang et al., 2024b) and use this model to generate judgements. In this way, we obtain synthetic data by utilizing a model finetuned on labeled data. We conducted iterative training and present results in Table 12. We observed good performance compared to the seed model (Llama3-70B-Instruct), however it does not clearly outperform conducting iterative training with unlabeled data alone.  We compare how combining synthetic preference data with human labelled preference data affects model performance. In particular, we combine synthetic preferences generated from reasoning WildChat prompts with the human labeled HelpSteer2 dataset (train split) and report performance in Table 6. We compare to first-iteration models trained on single data source, and select the best checkpoint for joint training using the validation split of HelpSteer2 and holdout set of synthetic preferences (in-distribution), as well as safety and code synthetic preferences (out-of-distribution). We then report evaluation results on RewardBench. The results show that overall the models retain strong performance across different data mixing weights, with slight improvements on overall accuracy. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Iterative Training By Initializing From Labeled Data
2408.02666
 We analyze the length distribution of the curated training set of selected instructions in Figure 3. The dataset has a long-tail distribution of input length, with most of the examples less than 500 tokens. In contrast, the full dataset (i.e., the full data before the instruction selection step of Subsection 3.2) has a cluster of very long instructions, containing content such as long-form coding instructions or transcripts.  We further instruct Llama-3-70B-Instruct to infer the complexity (using a score of 1–5) and category of each input instruction, as well as the length of the expected output, following the procedure in Yuan et al. (2024). From Figure 4 and Figure 6, we see that the curated dataset has more complex instructions involving logical reasoning/science whereas the full dataset has a greater proportion focused on relationships and entertainment. Finally, in Figure 5 we see that the anticipated length of the response is higher for the full dataset than the curated one, perhaps because of the greater frequency of lengthy, and sometimes repetitive instructions. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Instruction Complexity
2408.02666
 We present a scalable approach to build a strong generalist evaluator to perform model-based evaluation of LLM outputs. Our method constructs synthetic preferences over pairs of responses without using any human annotation. Our Self-Taught evaluator with iterative training over these synthetic preferences greatly boosts the accuracy of a strong seed LLM (Llama3-70B-Instruct) as an evaluator, from 75.4 to 88.7 on RewardBench, a new state-ofthe-art for generative LLM-as-a-Judge methods. 
conclusion
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Conclusion
2408.02666
 Generative LLM-as-a-Judge models usually have longer outputs and thus higher inference cost than reward models that simply output a score, as LLMas-a-Judge typically first generates a reasoning chain. Further, we have used relatively large LLMs in this work (70B parameters) and made no study of whether our approach works on smaller models. Since we use a seed model to generate first synthetic preferences during our iterative training scheme, one of the assumptions is that the model is capable of generating reasonable evaluations. Thus, our approach is limited by having a capable instruction fine-tuned model which is already reasonably aligned to human (or legal/policy) preferences. Furthermore, we only investigated and reported metrics involving evaluation accuracy improvements, rather than computational requirement concerns.  We also only investigated *pairwise evaluation*, i.e., comparing two responses, whereas it is also possible to use LLM-as-a-Judge models (or any other model) to evaluate the quality of *single responses*, e.g., giving them a score out of 5 or 10, rather than a pairwise A vs B judgment. We leave evaluating single responses to future work. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Limitations
2408.02666
 We thank Jing Xu and Janice Lan for discussions and support in the project overall, and Can Balioglu for his feedback and support in LLM training using the fairseq2 (Balioglu, 2023) library. We thank Nathan Lambert for the help with RewardBench, and Yuntian Deng and Yejin Choi for the help with WildChat. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
Acknowledgements
2408.02666
 Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. *arXiv preprint arXiv:2303.08774*.  Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.  Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. 2023. Benchmarking foundation models with language-model-as-an-examiner. In *Thirty-seventh* Conference on Neural Information Processing Systems Datasets and Benchmarks Track.  Can Balioglu. 2023. fairseq2.  Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. 2024. LLMs instead of human judges? A large scale empirical study across 20 NLP evaluation tasks. *arXiv preprint* arXiv:2406.18403.  Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In *Proceedings of the 61st Annual Meeting of* the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.  Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. *arXiv preprint arXiv:2110.14168*.  Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. AlpacaFarm: A simulation framework for methods that learn from human feedback. In *Thirty-seventh Conference on* Neural Information Processing Systems.  Khaled El Emam, Lucy Mosquera, and Richard Hoptroff. 2020. *Practical synthetic data generation: balancing privacy and the broad availability of data*.  O'Reilly Media. Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. 2023. FactKB: Generalizable factuality evaluation using language models enhanced with factual knowledge. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 933–952.  Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang.  2024. CRUXEval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065.  Bill Howe, Julia Stoyanovich, Haoyue Ping, Bernease Herman, and Matt Gee. 2017. Synthetic data for social good. *arXiv preprint arXiv:1710.08874*.  Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. 2024. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566.  Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al.  2023. Prometheus: Inducing fine-grained evaluation capability in language models. In *The Twelfth International Conference on Learning Representations*.  Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E.  Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In *Proceedings of the* ACM SIGOPS 29th Symposium on Operating Systems Principles.  Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. 2023. Learning skillful mediumrange global weather forecasting. *Science (New York,* NY), 382(6677):1416–1421.  Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. RewardBench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787.  Jack Lanchantin, Shubham Toshniwal, Jason E Weston, Arthur Szlam, and Sainbayar Sukhbaatar. 2023.  Learning to reason and memorize with self-notes.  In *Thirty-seventh Conference on Neural Information* Processing Systems.  Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023. RLAIF: Scaling reinforcement learning from human feedback with ai feedback. *arXiv preprint arXiv:2309.00267*.  Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024a. GSM-Plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. *arXiv preprint* arXiv:2402.19255. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2024b. Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations.  Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. AlpacaEval: An automatic evaluator of instruction-following models.  Changshu Liu, Shizhuo Dylan Zhang, and Reyhaneh Jabbarvand. 2024. CodeMind: A framework to challenge large language models for code reasoning.  arXiv preprint arXiv:2402.09664. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using GPT-4 with better human alignment. In *Proceedings of the 2023 Conference* on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics.  Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  2022. Training language models to follow instructions with human feedback. *Advances in Neural* Information Processing Systems, 35:27730–27744.  Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2024. West-of-N: Synthetic preference generation for improved reward modeling. In *ICLR 2024 Workshop on Navigating* and Addressing Data Problems for Foundation Models.  Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. 2023. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13387–13434.  Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.  Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.  Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2024. Branchsolve-merge improves large language model evaluation and generation. In *Proceedings of the 2024* Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8345–8363.  Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools. In *Thirty-seventh Conference on Neural* Information Processing Systems.  Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding. In *International Conference on Learning Representations*.  Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024a. Interpretable preferences via multi-objective reward modeling and mixture-ofexperts. *arXiv preprint arXiv:2406.12845*.  Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is ChatGPT a good NLG evaluator? a preliminary study. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 1–11, Singapore. Association for Computational Linguistics.  Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.  Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev.  2024b. HelpSteer2: Open-source dataset for training top-performing reward models. *arXiv preprint* arXiv:2406.08673.  Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, et al. 2024. Long-form factuality in large language models. *arXiv preprint* arXiv:2403.18802.  Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2023. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss. *arXiv preprint arXiv:2312.16682*.  Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2024. FLASK: Fine-grained language model evaluation based on alignment skill sets. In *The Twelfth International* Conference on Learning Representations. Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. In *The Twelfth International Conference on Learning Representations*.  Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  2024. Self-rewarding language models. *arXiv* preprint arXiv:2401.10020.  Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations.  Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. WildChat: 1M ChatGPT interaction logs in the wild. In *The Twelfth* International Conference on Learning Representations.  Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena.  In *Thirty-seventh Conference on Neural Information* Processing Systems Datasets and Benchmarks Track. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. 2023. Starling-7B: Increasing LLM Helpfulness & Harmlessness with RLAIF. 
references
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
References
2408.02666
 We provide the prompt templates used for annotating and selecting instructions (Figure 7), annotating judgments with synthetic preferences (Figure 8), and generating ablation synthetic preference data with bad responses (Figure 10). Figure 9 illustrates an training example based on synthetic preference data. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
A Appendix A.1 Prompt Templates
2408.02666
 We include training hyper-parameters in Table 7 and sampling parameters in Table 8. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
A.2 More Training And Evaluation Details
2408.02666
 We notice that when we evaluate generative models on RewardBench, the order of two responses in  ![11_image_0.png](11_image_0.png)  Figure 7: **Prompt template for Selecting Instructions.** We prompt an instruction following model to annotate the category of each instruction in order to curate our training data instructions.  Name Value max_seq_len 4096 max_num_tokens 8192  model llama3_70b_instruct  dtype bfloat16  data_parallelism fsdp  tensor_parallel_size 8 activation_checkpointing true  lr 1.0e-06  betas 0.9, 0.95 final_lr_ratio 0.2  weight_decay 0.1  num_lr_warmup_steps 100  gradient_accumulation 1  max_num_data_epochs 2 checkpoint_every_n_steps 100  seed 2  | Stage                                                                                                                  | Generation for    | Temperature   | Top p   | |------------------------------------------------------------------------------------------------------------------------|-------------------|---------------|---------| | Train                                                                                                                  | Judgment          | 0.7           | 0.9     | | Eval                                                                                                                   | MT-Bench          | 0.0           | 1.0     | | Eval                                                                                                                   | Reward Bench (RB) | 0.0           | 1.0     | | Eval                                                                                                                   | RB w/ maj voting  | 0.7           | 0.9     | | Eval                                                                                                                   | Helpsteer 2 valid | 0.7           | 0.9     | | Table 8: Sampling parameters (temperature and top p) used during generations at each stage of training and evaluation. |                   |               |         |  ![11_image_1.png](11_image_1.png)  each example is not fixed. More specifically, for each example, the winning response (y w) can be randomly placed before or after the losing response (y l). Generative models may output different judgements when the order of responses changes. Thus, we analyze how the performance varies when different seeds are used to decide response order. In Table 9, we test our model from the 5th iteration of training on RewardBench with the response order randomly shuffled, as well as two extreme cases where the winning answer always appear first or last. We recommend to report the average performance (88.3 for our 5th iteration) of "y w always first" and "y lalways first" as it fairly considers both orders.  ![12_image_0.png](12_image_0.png)  Figure 8: **Prompt template for Judgment Annotation.** This is the same prompt as used in MT-Bench and RewardBench.  | Seed             | Average Accuracy   | |------------------|--------------------| | 1                | 88.9               | | 11               | 88.4               | | 111              | 88.6               | | 1111             | 88.7               | | 11111            | 88.3               | | w always first   | 85.5               | | y l always first | 91.1               | | y                |                    |  Table 9: Average accuracy on RewardBench when order of responses changes. 
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
A.3 Position Order Evaluation On Rewardbench
2408.02666
 In Table 10 we present evaluation on RewardBench of models finetuned on different training data. Note in our Self-Taught Evaluator approach we can use different LLMs to generate responses and judgements. Specifically, we try using Mixtral 22Bx8 Instruct or Llama-3-70B-Instruct in various combinations. We then finetune the Llama-3-70B-Instruct model and test on RewardBench. As shown in Table 10, the model finetuned on data generated by using the Mixtral 22Bx8 Instruct model to judge Mixtral 22Bx8 Instruct model generated responses achieves the best performance.  | Model                                                                                      | Overall   | Chat   | Chat Hard   | Safety   | Reasoning   | |--------------------------------------------------------------------------------------------|-----------|--------|-------------|----------|-------------| | Llama-3-70B-Instruct (seed)                                                                 | 75.4      | 97.6   | 58.9        | 69.2     | 78.5        | | Self-Taught Evaluator, trained on synthetic data only Mixtral judge Mixtral 83.9 98.3 69.0 | 85.7      | 82.6   |             |          |             | | Llama3.0 judge Llama3.0                                                                    | 81.4      | 97.2   | 66.0        | 85.0     | 77.5        | | Llama3.0 judge Mixtral                                                                     | 80.0      | 97.5   | 70.0        | 72.8     | 79.4        |  Table 10: Performance on RewardBench of models finetuned on different training data.  | Model                                                                               | Overall   | writing   | stem   | coding   | math   | humanities   | reasoning   | roleplay   | extraction   | |-------------------------------------------------------------------------------------|-----------|-----------|--------|----------|--------|--------------|-------------|------------|--------------| | Llama-3-70B-Instruct (seed)                                                          | 77.8      | 70        | 76.9   | 73.8     | 80     | 79.85        | 78.8        | 78.8       | 85.1         | | Self-Taught Evaluator, trained on synthetic data only Iteration 1 78.95 71.15 78.95 | 76.6      | 81.65     | 80.25  | 82.3     | 80.7   | 80.25        |             |            |              | | Iteration 2                                                                         | 78.65     | 69.6      | 77.9   | 82.55    | 79.15  | 82.2         | 80.8        | 77.6       | 79.8         | | Iteration 3                                                                         | 78.9      | 70.4      | 78.6   | 79.35    | 79.55  | 82.9         | 82.3        | 77.9       | 80.7         | | Iteration 4                                                                         | 77.45     | 71.15     | 77.25  | 75.8     | 73.3   | 82.6         | 81.3        | 78.85      | 79.35        | | Iteration 5                                                                         | 78.9      | 68.45     | 78.2   | 81.75    | 82.5   | 82.25        | 81.35       | 75.15      | 83.75        | | w/ majority voting @ 32                                                             | 79.45     | 68.45     | 78.55  | 82.95    | 83.75  | 82.9         | 82.8        | 76.35      | 81.6         | | Other SoTA LLM-as-a-Judge baseline models GPT4-0125 79.15 70.4                      | 79.9      | 82.9      | 82.1   | 80.55    | 80.8   | 77           | 80.7        |            |              |  Table 11: **MT-Bench Per-category Results**. Our Self-Taught Evaluator trained on synthetic data without any human annotated preference labels performs on par with GPT-4 judgments.  | Model                                                                                                                   | Overall   | Chat   | Chat Hard   | Safety   | Reasoning   | |-------------------------------------------------------------------------------------------------------------------------|-----------|--------|-------------|----------|-------------| | Llama-3-70B-Instruct (seed)                                                                                              | 75.4      | 97.6   | 58.9        | 69.2     | 78.5        | | Self-Taught Evaluator, trained on synthetic data generated by a finetuned model (Helpsteer2)  Iteration 1 87.0 95.8 75.8 | 90.7      | 85.8   |             |          |             | | Iteration 2                                                                                                             | 86.6      | 92.2   | 77.4        | 91.2     | 85.8        |  Table 12: Iterative training on synthetic data generated by a model that is first fine-tuned on labeled data (Helpsteer2) .  Example Judgment Generation  Instruction:  ![14_image_0.png](14_image_0.png)   **Assistant A:** * Accuracy: 5/5 (correct calculation)  ![14_image_1.png](14_image_1.png)   Figure 9: An illustrative example of judgment generation given an instruction and two responses.  ![15_image_0.png](15_image_0.png)   Figure 10: **Generating a Bad Response for an Instruction.** This approach is an ablation compared to our proposed approach described in the main paper.
body
Self-Taught Evaluators Tianlu Wang Ilia Kulikov∗ Olga Golovneva∗ **Ping Yu**∗
A.4 **Using Different Models For Training Data** Generation