<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.21787] Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling</title><meta property="og:description" content="Scaling the amount of compute used to train language models has dramatically improved their capabilities. However, when it comes to inference, we often limit the amount of compute to only one attempt per problem. Here,…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.21787">

<!--Generated on Mon Aug  5 15:27:43 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<!--Document created on July 30, 2024.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Large Language Monkeys<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">0</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">0</sup><span class="ltx_note_type">footnotetext: </span>Title inspired by <a target="_blank" href="https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem</a>.</span></span></span>: Scaling Inference Compute 
<br class="ltx_break">with Repeated Sampling</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Bradley Brown<sup id="id4.2.id1" class="ltx_sup">∗</sup>
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Jordan Juravsky<sup id="id5.2.id1" class="ltx_sup">∗</sup>
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ryan Ehrlich<sup id="id6.2.id1" class="ltx_sup">∗</sup>
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ronald Clark
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Quoc V. Le
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Christopher Ré
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Azalia Mirhoseini
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
</div>
<div class="ltx_dates">(July 30, 2024)</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id7.id1" class="ltx_p">Scaling the amount of compute used to train language models has dramatically improved their capabilities. However, when it comes to inference, we often limit the amount of compute to only one attempt per problem. Here, we explore inference compute as another axis for scaling by increasing the number of generated samples. Across multiple tasks and models, we observe that coverage – the fraction of problems solved by any attempt – scales with the number of samples over four orders of magnitude. In domains like coding and formal proofs, where all answers can be automatically verified, these increases in coverage directly translate into improved performance.
When we apply repeated sampling to SWE-bench Lite, the fraction of issues solved with DeepSeek-V2-Coder-Instruct increases from 15.9% with one sample to 56% with 250 samples, outperforming the single-attempt state-of-the-art of 43% which uses more capable frontier models.
Moreover, using current API pricing, amplifying the cheaper DeepSeek model with five samples is more cost-effective and solves more issues than paying a premium for one sample from GPT-4o or Claude 3.5 Sonnet. Interestingly, the relationship between coverage and the number of samples is often log-linear and can be modelled with an exponentiated power law, suggesting the existence of inference-time scaling laws.
Finally, we find that identifying correct samples out of many generations remains an important direction for future research in domains without automatic verifiers.
When solving math word problems from GSM8K and MATH, coverage with Llama-3 models grows to over 95% with 10,000 samples.
However, common methods to pick correct solutions from a sample collection, such as majority voting or reward models, plateau beyond several hundred samples and fail to fully scale with the sample budget.</p>
<span id="footnotex2" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">0</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">0</sup><span class="ltx_note_type">footnotetext: </span><sup id="footnotex2.1" class="ltx_sup">∗</sup> Equal Contribution. Work done by BB as a visiting researcher at Stanford.</span></span></span>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">The ability of large language models (LLMs) to solve coding, mathematics, and other reasoning tasks has improved dramatically over the past several years <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>, <a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>. Scaling up model training has been a consistent driver of these gains. Investments in larger models, larger pre-training datasets, and more extensive post-training (e.g. through collecting human preference labels) has led to remarkably capable generalist systems <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>, <a href="#bib.bib3" title="" class="ltx_ref">3</a>, <a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">In contrast, a comparatively limited investment has been made in scaling the amount of computation used during inference. Larger models do require more inference compute than smaller ones, and prompting techniques like chain-of-thought <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib56" title="" class="ltx_ref">56</a>]</cite> can increase answer quality at the cost of longer (and therefore more computationally expensive) outputs. However, when interacting with LLMs, users and developers often restrict models to making only one attempt when solving a problem. In this work, we investigate repeated sampling (depicted in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) as an alternative axis for scaling inference compute to improve LLM reasoning performance.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.21787/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="103" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.2.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.3.2" class="ltx_text" style="font-size:90%;">The repeated sampling procedure that we follow in this paper. 1) We generate many candidate solutions for a given problem by sampling from an LLM with a positive temperature. 2) We use a domain-specific verifier (ex. unit tests for code) to select a final answer from the generated samples.</span></figcaption>
</figure>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">The effectiveness of repeated sampling is determined by two key properties:</p>
</div>
<div id="S1.p4" class="ltx_para">
<ol id="S1.I1" class="ltx_enumerate">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">Coverage: As the number of samples increases, what fraction of problems can we solve using any sample that was generated?</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">Precision: In settings where we must select a final answer from the collection of generated samples, can we identify correct samples?</p>
</div>
</li>
</ol>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">With unlimited attempts, any model that assigns a non-zero probability to every sequence will achieve perfect coverage. However, repeated sampling is only practical if we can improve coverage with a feasible budget.
Moreover, without the ability to decide between samples, the applications of repeated sampling are limited. Existing work provides encouraging evidence along both of these directions, showing examples of repeated sampling improving performance in math, coding, and puzzle-solving settings <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>, <a href="#bib.bib43" title="" class="ltx_ref">43</a>, <a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>. Notably, AlphaCode <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>, a state-of-the-art system for competitive programming, finds that performance continues to improve with a million samples per problem.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">Here, we show that repeated sampling is an effective methodology for improving coverage across a range of tasks, models, and sample budgets.
For example, when solving CodeContests <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite> programming problems using Gemma-2B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite>, we increase coverage by over 300x as we scale the number of samples, from 0.02% with one attempt to 7.1% with 10,000 attempts.
Interestingly, the relationship between <math id="S1.p6.1.m1.2" class="ltx_Math" alttext="\log(\text{coverage})" display="inline"><semantics id="S1.p6.1.m1.2a"><mrow id="S1.p6.1.m1.2.3.2" xref="S1.p6.1.m1.2.3.1.cmml"><mi id="S1.p6.1.m1.1.1" xref="S1.p6.1.m1.1.1.cmml">log</mi><mo id="S1.p6.1.m1.2.3.2a" xref="S1.p6.1.m1.2.3.1.cmml">⁡</mo><mrow id="S1.p6.1.m1.2.3.2.1" xref="S1.p6.1.m1.2.3.1.cmml"><mo stretchy="false" id="S1.p6.1.m1.2.3.2.1.1" xref="S1.p6.1.m1.2.3.1.cmml">(</mo><mtext id="S1.p6.1.m1.2.2" xref="S1.p6.1.m1.2.2a.cmml">coverage</mtext><mo stretchy="false" id="S1.p6.1.m1.2.3.2.1.2" xref="S1.p6.1.m1.2.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.p6.1.m1.2b"><apply id="S1.p6.1.m1.2.3.1.cmml" xref="S1.p6.1.m1.2.3.2"><log id="S1.p6.1.m1.1.1.cmml" xref="S1.p6.1.m1.1.1"></log><ci id="S1.p6.1.m1.2.2a.cmml" xref="S1.p6.1.m1.2.2"><mtext id="S1.p6.1.m1.2.2.cmml" xref="S1.p6.1.m1.2.2">coverage</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p6.1.m1.2c">\log(\text{coverage})</annotation></semantics></math> and the number of samples often follows an approximate power law. With Llama-3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref">3</a>]</cite> and Gemma models, we observe that coverage grows nearly log-linearly with the number of samples over several orders of magnitude.</p>
</div>
<div id="S1.p7" class="ltx_para">
<p id="S1.p7.1" class="ltx_p">In settings where all of a model’s solutions can be automatically verified, such as with proof checkers or unit tests, these increases in coverage translate directly into improved task performance. When applying repeated sampling to competitive programming and writing Lean proofs, models like Llama-3-8B-Instruct can exceed the single-attempt performance of much stronger ones like GPT-4o <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>. This ability to amplify weaker models extends to the challenging SWE-bench Lite dataset of real-life GitHub issues <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>]</cite>, where the current single-attempt state-of-the-art (SOTA), achieved by a mixture of GPT-4o and Claude 3.5 Sonnet models, is 43% <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite>. When given a single attempt, DeepSeek-Coder-V2-Instruct <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite> achieves only 15.9% on the benchmark. By simply increasing the number of attempts to 250, we increase the fraction of solved problems to 56%, exceeding the single-attempt state-of-the-art by 13%.</p>
</div>
<div id="S1.p8" class="ltx_para">
<p id="S1.p8.1" class="ltx_p">In addition to improving model quality, repeated sampling provides a new mechanism for minimizing LLM inference costs. When holding the total number of inference FLOPs constant, we find that on some datasets (e.g. MATH) coverage is maximized with a smaller model and more attempts, while on others (e.g CodeContests) it is better to use a larger model. We also compare API prices between DeepSeek-V2-Coder-Instruct, GPT-4o, and Claude Sonnet 3.5 in the context of solving SWE-bench Lite issues. When keeping the agent framework (Moatless Tools <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib62" title="" class="ltx_ref">62</a>]</cite>) constant, sampling five times from the weaker and cheaper DeepSeek model solves more issues than single attempts from Claude or GPT, while also being over 3x cheaper.</p>
</div>
<div id="S1.p9" class="ltx_para">
<p id="S1.p9.1" class="ltx_p">Finally, in math word problem settings, where answers cannot be automatically verified by existing tools, we identify a large gap between coverage and the performance of common methods for deciding on a final answer. When solving MATH <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite> problems with Llama-3-8B-Instruct, coverage increases from 79.8% with 100 samples to 95.3% with 10,000 samples. However, methods such as majority voting and using reward models plateau with a lower sample budget, scaling only from 38.7% to 39.8% over the same range. These results highlight that building robust verifiers remains an open problem.</p>
</div>
<div id="S1.p10" class="ltx_para">
<p id="S1.p10.1" class="ltx_p">In summary, our primary observations are:</p>
</div>
<div id="S1.p11" class="ltx_para">
<ol id="S1.I2" class="ltx_enumerate">
<li id="S1.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I2.i1.p1" class="ltx_para">
<p id="S1.I2.i1.p1.1" class="ltx_p">We demonstrate that scaling inference compute through repeated sampling leads to large improvements in coverage across a variety tasks, models, and sample budgets. This makes it possible, and sometimes cost-effective, to amplify weaker models with many samples and outperform single attempts from more capable models. Notably, we are able to solve 56% of issues from SWE-bench Lite by sampling 250 times from DeepSeek-V2-Coder-Instruct, exceeding the single-attempt SOTA of 43%.</p>
</div>
</li>
<li id="S1.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I2.i2.p1" class="ltx_para">
<p id="S1.I2.i2.p1.1" class="ltx_p">We show that the relationship between coverage and the number of samples can often be modelled using an exponentiated power law, suggesting a form of scaling laws for inference-time compute.</p>
</div>
</li>
<li id="S1.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S1.I2.i3.p1" class="ltx_para">
<p id="S1.I2.i3.p1.1" class="ltx_p">In domains without automatic verifiers, we show that common approaches to verification like majority voting and reward model scoring plateau beyond approximately 100 samples. This leads to a growing gap between the performance achieved with these methods and the coverage upper bound.</p>
</div>
</li>
</ol>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Scaling Repeated Sampling</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">We focus on pass-fail tasks where a candidate solution can be scored as right or wrong. The primary metric of interest for these tasks is the <span id="S2.p1.1.1" class="ltx_text ltx_font_italic">success rate:</span> the fraction of problems that we are able to solve. With repeated sampling, we consider a setup where a model can generate many candidate solutions while attempting to solve a problem. The success rate is therefore influenced both by the ability to generate correct samples for many problems (i.e. coverage), as well as the ability to identify these correct samples (i.e. precision).</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.1" class="ltx_p">The difficulty of the precision problem depends on the availability of tools for sample verification. When proving formal statements in Lean, proof checkers can quickly identify whether a candidate solution is correct. Similarly, unit tests can be used to verify candidate solutions to coding tasks. In these cases, precision is handled automatically, and improving coverage directly translates into higher success rates. In contrast, the tools available for verifying solutions to math word problems from GSM8K and MATH are limited, necessitating additional verification methods that decide on a single final answer from many (often conflicting) samples.</p>
</div>
<div id="S2.p3" class="ltx_para">
<p id="S2.p3.1" class="ltx_p">We consider the following five tasks:</p>
</div>
<div id="S2.p4" class="ltx_para">
<ol id="S2.I1" class="ltx_enumerate">
<li id="S2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S2.I1.i1.p1" class="ltx_para">
<p id="S2.I1.i1.p1.1" class="ltx_p"><span id="S2.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">GSM8K:</span> A dataset of grade-school level math word problems <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>. We evaluate on a random subset of 128 problems from the GSM8K test set.</p>
</div>
</li>
<li id="S2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S2.I1.i2.p1" class="ltx_para">
<p id="S2.I1.i2.p1.1" class="ltx_p"><span id="S2.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">MATH:</span> Another dataset of math word problems that are generally harder than those from GSM8K <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite>. Similarly, we evaluate on 128 random problems from this dataset’s test set.</p>
</div>
</li>
<li id="S2.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S2.I1.i3.p1" class="ltx_para">
<p id="S2.I1.i3.p1.1" class="ltx_p"><span id="S2.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">MiniF2F-MATH:</span> A dataset of mathematics problems that have been formalized into proof checking languages <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>. We use Lean4 as our language, and evaluate on the 130 test set problems that are formalized from the MATH dataset.</p>
</div>
</li>
<li id="S2.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="S2.I1.i4.p1" class="ltx_para">
<p id="S2.I1.i4.p1.1" class="ltx_p"><span id="S2.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">CodeContests:</span> A dataset of competitive programming problems <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>. Each problem has a text description, along with a set of input-output test cases (hidden from the model) that can be used to verify the correctness of a candidate solution. We enforce that models write their solutions using Python3.</p>
</div>
</li>
<li id="S2.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">5.</span> 
<div id="S2.I1.i5.p1" class="ltx_para">
<p id="S2.I1.i5.p1.1" class="ltx_p"><span id="S2.I1.i5.p1.1.1" class="ltx_text ltx_font_bold">SWE-bench Lite:</span> A dataset of real world Github issues, where each problem consists of a description and a snapshot of a code repository <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>]</cite>. To solve a problem, models must edit files in the codebase (in the Lite subset of SWE-bench that we use, only a single file needs to be changed). Candidate solutions can be automatically checked using the repository’s suite of unit tests.</p>
</div>
</li>
</ol>
</div>
<div id="S2.p5" class="ltx_para">
<p id="S2.p5.5" class="ltx_p">Among these tasks, MiniF2F-MATH, CodeContests, and SWE-bench Lite have automatic verifiers (in the form of the Lean4 proof checker, test cases, and unit test suites, respectively).
We begin by investigating how repeated sampling improves model coverage. Coverage improvements correspond directly with increased success rates for tasks with automatic verifiers and in the general case provide an upper bound on the success rate.
In coding settings, our definition of coverage is equivalent to the commonly-used pass@k metric <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>, where <math id="S2.p5.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.p5.1.m1.1a"><mi id="S2.p5.1.m1.1.1" xref="S2.p5.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.p5.1.m1.1b"><ci id="S2.p5.1.m1.1.1.cmml" xref="S2.p5.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p5.1.m1.1c">k</annotation></semantics></math> denotes the number of samples per problem. We use this metric directly when evaluating on CodeContests and SWE-bench Lite. For MiniF2F the metric is similar, with a “pass” defined according to the Lean4 proof checker. For GSM8K and MATH, coverage corresponds to using an oracle verifier that checks if any sample “passes” by outputting the correct final answer. To reduce the variance when calculating coverage, we adopt the unbiased estimation formula from <cite class="ltx_cite ltx_citemacro_citet">Chen et al. [<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>. In each experiment, we first generate <math id="S2.p5.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.p5.2.m2.1a"><mi id="S2.p5.2.m2.1.1" xref="S2.p5.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.p5.2.m2.1b"><ci id="S2.p5.2.m2.1.1.cmml" xref="S2.p5.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p5.2.m2.1c">N</annotation></semantics></math> samples for each problem index <math id="S2.p5.3.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.p5.3.m3.1a"><mi id="S2.p5.3.m3.1.1" xref="S2.p5.3.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.p5.3.m3.1b"><ci id="S2.p5.3.m3.1.1.cmml" xref="S2.p5.3.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p5.3.m3.1c">i</annotation></semantics></math> and calculate the number of correct samples <math id="S2.p5.4.m4.1" class="ltx_Math" alttext="C_{i}" display="inline"><semantics id="S2.p5.4.m4.1a"><msub id="S2.p5.4.m4.1.1" xref="S2.p5.4.m4.1.1.cmml"><mi id="S2.p5.4.m4.1.1.2" xref="S2.p5.4.m4.1.1.2.cmml">C</mi><mi id="S2.p5.4.m4.1.1.3" xref="S2.p5.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.p5.4.m4.1b"><apply id="S2.p5.4.m4.1.1.cmml" xref="S2.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S2.p5.4.m4.1.1.1.cmml" xref="S2.p5.4.m4.1.1">subscript</csymbol><ci id="S2.p5.4.m4.1.1.2.cmml" xref="S2.p5.4.m4.1.1.2">𝐶</ci><ci id="S2.p5.4.m4.1.1.3.cmml" xref="S2.p5.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p5.4.m4.1c">C_{i}</annotation></semantics></math>. We then calculate the pass@k scores at each <math id="S2.p5.5.m5.1" class="ltx_Math" alttext="k\leq N" display="inline"><semantics id="S2.p5.5.m5.1a"><mrow id="S2.p5.5.m5.1.1" xref="S2.p5.5.m5.1.1.cmml"><mi id="S2.p5.5.m5.1.1.2" xref="S2.p5.5.m5.1.1.2.cmml">k</mi><mo id="S2.p5.5.m5.1.1.1" xref="S2.p5.5.m5.1.1.1.cmml">≤</mo><mi id="S2.p5.5.m5.1.1.3" xref="S2.p5.5.m5.1.1.3.cmml">N</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.p5.5.m5.1b"><apply id="S2.p5.5.m5.1.1.cmml" xref="S2.p5.5.m5.1.1"><leq id="S2.p5.5.m5.1.1.1.cmml" xref="S2.p5.5.m5.1.1.1"></leq><ci id="S2.p5.5.m5.1.1.2.cmml" xref="S2.p5.5.m5.1.1.2">𝑘</ci><ci id="S2.p5.5.m5.1.1.3.cmml" xref="S2.p5.5.m5.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p5.5.m5.1c">k\leq N</annotation></semantics></math> of interest according to:</p>
</div>
<div id="S2.p6" class="ltx_para">
<table id="A5.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><span id="S2.E1.2.1.1.1" class="ltx_text ltx_markedasmath">pass@k</span></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E1.m2.5" class="ltx_Math" alttext="\displaystyle=\frac{1}{\text{\# of problems}}\sum_{i=1}^{\text{\# of problems}}\left(1-\frac{\binom{N-C_{i}}{k}}{\binom{N}{k}}\right)" display="inline"><semantics id="S2.E1.m2.5a"><mrow id="S2.E1.m2.5.5" xref="S2.E1.m2.5.5.cmml"><mi id="S2.E1.m2.5.5.3" xref="S2.E1.m2.5.5.3.cmml"></mi><mo id="S2.E1.m2.5.5.2" xref="S2.E1.m2.5.5.2.cmml">=</mo><mrow id="S2.E1.m2.5.5.1" xref="S2.E1.m2.5.5.1.cmml"><mstyle displaystyle="true" id="S2.E1.m2.5.5.1.3" xref="S2.E1.m2.5.5.1.3.cmml"><mfrac id="S2.E1.m2.5.5.1.3a" xref="S2.E1.m2.5.5.1.3.cmml"><mn id="S2.E1.m2.5.5.1.3.2" xref="S2.E1.m2.5.5.1.3.2.cmml">1</mn><mtext id="S2.E1.m2.5.5.1.3.3" xref="S2.E1.m2.5.5.1.3.3a.cmml"># of problems</mtext></mfrac></mstyle><mo lspace="0em" rspace="0em" id="S2.E1.m2.5.5.1.2" xref="S2.E1.m2.5.5.1.2.cmml">​</mo><mrow id="S2.E1.m2.5.5.1.1" xref="S2.E1.m2.5.5.1.1.cmml"><mstyle displaystyle="true" id="S2.E1.m2.5.5.1.1.2" xref="S2.E1.m2.5.5.1.1.2.cmml"><munderover id="S2.E1.m2.5.5.1.1.2a" xref="S2.E1.m2.5.5.1.1.2.cmml"><mo movablelimits="false" id="S2.E1.m2.5.5.1.1.2.2.2" xref="S2.E1.m2.5.5.1.1.2.2.2.cmml">∑</mo><mrow id="S2.E1.m2.5.5.1.1.2.2.3" xref="S2.E1.m2.5.5.1.1.2.2.3.cmml"><mi id="S2.E1.m2.5.5.1.1.2.2.3.2" xref="S2.E1.m2.5.5.1.1.2.2.3.2.cmml">i</mi><mo id="S2.E1.m2.5.5.1.1.2.2.3.1" xref="S2.E1.m2.5.5.1.1.2.2.3.1.cmml">=</mo><mn id="S2.E1.m2.5.5.1.1.2.2.3.3" xref="S2.E1.m2.5.5.1.1.2.2.3.3.cmml">1</mn></mrow><mtext id="S2.E1.m2.5.5.1.1.2.3" xref="S2.E1.m2.5.5.1.1.2.3a.cmml"># of problems</mtext></munderover></mstyle><mrow id="S2.E1.m2.5.5.1.1.1.1" xref="S2.E1.m2.5.5.1.1.1.1.1.cmml"><mo id="S2.E1.m2.5.5.1.1.1.1.2" xref="S2.E1.m2.5.5.1.1.1.1.1.cmml">(</mo><mrow id="S2.E1.m2.5.5.1.1.1.1.1" xref="S2.E1.m2.5.5.1.1.1.1.1.cmml"><mn id="S2.E1.m2.5.5.1.1.1.1.1.2" xref="S2.E1.m2.5.5.1.1.1.1.1.2.cmml">1</mn><mo id="S2.E1.m2.5.5.1.1.1.1.1.1" xref="S2.E1.m2.5.5.1.1.1.1.1.1.cmml">−</mo><mstyle displaystyle="true" id="S2.E1.m2.4.4a" xref="S2.E1.m2.4.4a.cmml"><mfrac id="S2.E1.m2.4.4aa" xref="S2.E1.m2.4.4a.cmml"><mrow id="S2.E1.m2.2.2.2a.4" xref="S2.E1.m2.2.2.2a.3.cmml"><mo id="S2.E1.m2.2.2.2a.4.1" xref="S2.E1.m2.2.2.2a.3.1.cmml">(</mo><mfrac linethickness="0pt" id="S2.E1.m2.2.2.2.2.2.2a" xref="S2.E1.m2.2.2.2a.3.cmml"><mrow id="S2.E1.m2.1.1.1.1.1.1.1.1" xref="S2.E1.m2.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E1.m2.1.1.1.1.1.1.1.1.3" xref="S2.E1.m2.1.1.1.1.1.1.1.1.3.cmml">N</mi><mo id="S2.E1.m2.1.1.1.1.1.1.1.1.2" xref="S2.E1.m2.1.1.1.1.1.1.1.1.2.cmml">−</mo><msub id="S2.E1.m2.1.1.1.1.1.1.1.1.4" xref="S2.E1.m2.1.1.1.1.1.1.1.1.4.cmml"><mi id="S2.E1.m2.1.1.1.1.1.1.1.1.4.2" xref="S2.E1.m2.1.1.1.1.1.1.1.1.4.2.cmml">C</mi><mi id="S2.E1.m2.1.1.1.1.1.1.1.1.4.3" xref="S2.E1.m2.1.1.1.1.1.1.1.1.4.3.cmml">i</mi></msub></mrow><mi id="S2.E1.m2.2.2.2.2.2.2.2.1" xref="S2.E1.m2.2.2.2.2.2.2.2.1.cmml">k</mi></mfrac><mo id="S2.E1.m2.2.2.2a.4.2" xref="S2.E1.m2.2.2.2a.3.1.cmml">)</mo></mrow><mrow id="S2.E1.m2.4.4.4a.4" xref="S2.E1.m2.4.4.4a.3.cmml"><mo id="S2.E1.m2.4.4.4a.4.1" xref="S2.E1.m2.4.4.4a.3.1.cmml">(</mo><mfrac linethickness="0pt" id="S2.E1.m2.4.4.4.2.2.2a" xref="S2.E1.m2.4.4.4a.3.cmml"><mi id="S2.E1.m2.3.3.3.1.1.1.1.1" xref="S2.E1.m2.3.3.3.1.1.1.1.1.cmml">N</mi><mi id="S2.E1.m2.4.4.4.2.2.2.2.1" xref="S2.E1.m2.4.4.4.2.2.2.2.1.cmml">k</mi></mfrac><mo id="S2.E1.m2.4.4.4a.4.2" xref="S2.E1.m2.4.4.4a.3.1.cmml">)</mo></mrow></mfrac></mstyle></mrow><mo id="S2.E1.m2.5.5.1.1.1.1.3" xref="S2.E1.m2.5.5.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m2.5b"><apply id="S2.E1.m2.5.5.cmml" xref="S2.E1.m2.5.5"><eq id="S2.E1.m2.5.5.2.cmml" xref="S2.E1.m2.5.5.2"></eq><csymbol cd="latexml" id="S2.E1.m2.5.5.3.cmml" xref="S2.E1.m2.5.5.3">absent</csymbol><apply id="S2.E1.m2.5.5.1.cmml" xref="S2.E1.m2.5.5.1"><times id="S2.E1.m2.5.5.1.2.cmml" xref="S2.E1.m2.5.5.1.2"></times><apply id="S2.E1.m2.5.5.1.3.cmml" xref="S2.E1.m2.5.5.1.3"><divide id="S2.E1.m2.5.5.1.3.1.cmml" xref="S2.E1.m2.5.5.1.3"></divide><cn type="integer" id="S2.E1.m2.5.5.1.3.2.cmml" xref="S2.E1.m2.5.5.1.3.2">1</cn><ci id="S2.E1.m2.5.5.1.3.3a.cmml" xref="S2.E1.m2.5.5.1.3.3"><mtext id="S2.E1.m2.5.5.1.3.3.cmml" xref="S2.E1.m2.5.5.1.3.3"># of problems</mtext></ci></apply><apply id="S2.E1.m2.5.5.1.1.cmml" xref="S2.E1.m2.5.5.1.1"><apply id="S2.E1.m2.5.5.1.1.2.cmml" xref="S2.E1.m2.5.5.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m2.5.5.1.1.2.1.cmml" xref="S2.E1.m2.5.5.1.1.2">superscript</csymbol><apply id="S2.E1.m2.5.5.1.1.2.2.cmml" xref="S2.E1.m2.5.5.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m2.5.5.1.1.2.2.1.cmml" xref="S2.E1.m2.5.5.1.1.2">subscript</csymbol><sum id="S2.E1.m2.5.5.1.1.2.2.2.cmml" xref="S2.E1.m2.5.5.1.1.2.2.2"></sum><apply id="S2.E1.m2.5.5.1.1.2.2.3.cmml" xref="S2.E1.m2.5.5.1.1.2.2.3"><eq id="S2.E1.m2.5.5.1.1.2.2.3.1.cmml" xref="S2.E1.m2.5.5.1.1.2.2.3.1"></eq><ci id="S2.E1.m2.5.5.1.1.2.2.3.2.cmml" xref="S2.E1.m2.5.5.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="S2.E1.m2.5.5.1.1.2.2.3.3.cmml" xref="S2.E1.m2.5.5.1.1.2.2.3.3">1</cn></apply></apply><ci id="S2.E1.m2.5.5.1.1.2.3a.cmml" xref="S2.E1.m2.5.5.1.1.2.3"><mtext mathsize="70%" id="S2.E1.m2.5.5.1.1.2.3.cmml" xref="S2.E1.m2.5.5.1.1.2.3"># of problems</mtext></ci></apply><apply id="S2.E1.m2.5.5.1.1.1.1.1.cmml" xref="S2.E1.m2.5.5.1.1.1.1"><minus id="S2.E1.m2.5.5.1.1.1.1.1.1.cmml" xref="S2.E1.m2.5.5.1.1.1.1.1.1"></minus><cn type="integer" id="S2.E1.m2.5.5.1.1.1.1.1.2.cmml" xref="S2.E1.m2.5.5.1.1.1.1.1.2">1</cn><apply id="S2.E1.m2.4.4a.cmml" xref="S2.E1.m2.4.4a"><divide id="S2.E1.m2.4.4a.5.cmml" xref="S2.E1.m2.4.4a"></divide><apply id="S2.E1.m2.2.2.2a.3.cmml" xref="S2.E1.m2.2.2.2a.4"><csymbol cd="latexml" id="S2.E1.m2.2.2.2a.3.1.cmml" xref="S2.E1.m2.2.2.2a.4.1">binomial</csymbol><apply id="S2.E1.m2.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m2.1.1.1.1.1.1.1.1"><minus id="S2.E1.m2.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E1.m2.1.1.1.1.1.1.1.1.2"></minus><ci id="S2.E1.m2.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E1.m2.1.1.1.1.1.1.1.1.3">𝑁</ci><apply id="S2.E1.m2.1.1.1.1.1.1.1.1.4.cmml" xref="S2.E1.m2.1.1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E1.m2.1.1.1.1.1.1.1.1.4.1.cmml" xref="S2.E1.m2.1.1.1.1.1.1.1.1.4">subscript</csymbol><ci id="S2.E1.m2.1.1.1.1.1.1.1.1.4.2.cmml" xref="S2.E1.m2.1.1.1.1.1.1.1.1.4.2">𝐶</ci><ci id="S2.E1.m2.1.1.1.1.1.1.1.1.4.3.cmml" xref="S2.E1.m2.1.1.1.1.1.1.1.1.4.3">𝑖</ci></apply></apply><ci id="S2.E1.m2.2.2.2.2.2.2.2.1.cmml" xref="S2.E1.m2.2.2.2.2.2.2.2.1">𝑘</ci></apply><apply id="S2.E1.m2.4.4.4a.3.cmml" xref="S2.E1.m2.4.4.4a.4"><csymbol cd="latexml" id="S2.E1.m2.4.4.4a.3.1.cmml" xref="S2.E1.m2.4.4.4a.4.1">binomial</csymbol><ci id="S2.E1.m2.3.3.3.1.1.1.1.1.cmml" xref="S2.E1.m2.3.3.3.1.1.1.1.1">𝑁</ci><ci id="S2.E1.m2.4.4.4.2.2.2.2.1.cmml" xref="S2.E1.m2.4.4.4.2.2.2.2.1">𝑘</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m2.5c">\displaystyle=\frac{1}{\text{\# of problems}}\sum_{i=1}^{\text{\# of problems}}\left(1-\frac{\binom{N-C_{i}}{k}}{\binom{N}{k}}\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
</div>
<div id="S2.p7" class="ltx_para">
<p id="S2.p7.1" class="ltx_p">We use the numerically stable implementation of the above formula suggested in <cite class="ltx_cite ltx_citemacro_citet">Chen et al. [<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>.
Results and code will be available at <a target="_blank" href="https://scalyresearch.stanford.edu/pubs/large_language_monkeys/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://scalyresearch.stanford.edu/pubs/large_language_monkeys/</a>.</p>
</div>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Repeated Sampling is Effective Across Tasks</h3>

<figure id="S2.F2" class="ltx_figure"><img src="/html/2407.21787/assets/x2.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="184" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F2.2.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S2.F2.3.2" class="ltx_text" style="font-size:90%;">Across five tasks, we find that coverage (the fraction of problems solved by at least one generated sample) increases as we scale the number of samples. Notably, using repeated sampling, we are able to increase the solve rate of an open-source method from 15.9% to 56% on SWE-bench Lite.</span></figcaption>
</figure>
<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">Here, we establish that repeated sampling improves coverage across multiple tasks and a range of sample budgets. We evaluate Llama-3-8B-Instruct and Llama-3-70B-Instruct on CodeContests, MiniF2F, GSM8K, and MATH, generating 10,000 independent samples per problem. For SWE-bench Lite, we use DeepSeek-V2-Coder-Instruct <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite>, as the required context length of this task exceeds the limits of the Llama-3 models. As is standard when solving SWE-bench issues, we equip our LLM with a software framework that provides the model with tools for navigating through and editing codebases. In our work, we use the open-source Moatless Tools library <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib62" title="" class="ltx_ref">62</a>]</cite>. Note that solving a SWE-bench issue involves a back-and-forth exchange between the LLM and Moatless Tools. One sample/attempt for this benchmark refers to one entire multi-turn trajectory. To minimize costs, we restrict the number of attempts per issue to 250, with all attempts made independently of one another.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.1" class="ltx_p">We report our results in Figure <a href="#S2.F2" title="Figure 2 ‣ 2.1 Repeated Sampling is Effective Across Tasks ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. We also include the single-attempt performance of GPT-4o on each task, as well the single-attempt state-of-the-art for SWE-bench Lite (CodeStory Aide <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite> which uses a combination of GPT-4o and Claude 3.5 Sonnet). Across all five tasks, we find that coverage smoothly improves as the sample budget increases. When all LLMs are given a single attempt, GPT-4o outperforms the Llama and DeepSeek models at every task. However, as the number of samples increases, all three of the weaker models exceed GPT-4o’s single-attempt performance. In the case of SWE-bench Lite, we solve 56% of problems, exceeding the single-attempt SOTA of 43%.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Repeated Sampling is Effective Across Model Sizes and Families</h3>

<figure id="S2.F3" class="ltx_figure"><img src="/html/2407.21787/assets/x3.png" id="S2.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="208" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F3.2.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S2.F3.3.2" class="ltx_text" style="font-size:90%;">Scaling inference time compute via repeated sampling leads to consistent coverage gains across a variety of model sizes (70M-70B), families (Llama, Gemma and Pythia) and levels of post-training (Base and Instruct models). </span></figcaption>
</figure>
<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">The results from Section <a href="#S2.SS1" title="2.1 Repeated Sampling is Effective Across Tasks ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a> indicate that repeated sampling improves coverage. However, we only show this trend for three recent, instruction-tuned models with 8B or more parameters. We now show that these trends hold across other model sizes, families, and levels of post-training. We expand our evaluation to include a broader set of models:</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<ul id="S2.I2" class="ltx_itemize">
<li id="S2.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I2.i1.p1" class="ltx_para">
<p id="S2.I2.i1.p1.1" class="ltx_p"><span id="S2.I2.i1.p1.1.1" class="ltx_text ltx_font_bold">Llama 3:</span> Llama-3-8B, Llama-3-8B-Instruct, Llama-3-70B-Instruct.</p>
</div>
</li>
<li id="S2.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I2.i2.p1" class="ltx_para">
<p id="S2.I2.i2.p1.1" class="ltx_p"><span id="S2.I2.i2.p1.1.1" class="ltx_text ltx_font_bold">Gemma:</span> Gemma-2B, Gemma-7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite>.</p>
</div>
</li>
<li id="S2.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I2.i3.p1" class="ltx_para">
<p id="S2.I2.i3.p1.1" class="ltx_p"><span id="S2.I2.i3.p1.1.1" class="ltx_text ltx_font_bold">Pythia:</span> Pythia-70M through Pythia-12B (eight models in total) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite>.</p>
</div>
</li>
</ul>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.1" class="ltx_p">We restrict evaluation to the MATH and CodeContests datasets to minimize inference costs, reporting results in Figure <a href="#S2.F3" title="Figure 3 ‣ 2.2 Repeated Sampling is Effective Across Model Sizes and Families ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Coverage increases across almost every model we test, with smaller models showing some of the sharpest increases in coverage when repeated sampling is applied. On CodeContests, the coverage of Gemma-2B increases by over 300x, from a pass@1 of 0.02% to a pass@10k of 7.1%.
Similarly, when solving MATH problems with Pythia-160M, coverage increases from a pass@1 of 0.27% to a pass@10k of 57%.</p>
</div>
<div id="S2.SS2.p4" class="ltx_para">
<p id="S2.SS2.p4.1" class="ltx_p">The exception to this pattern of increasing coverage across models is with the Pythia family evaluated on CodeContests. All Pythia models achieve zero coverage on this dataset, even with a budget of 10,000 samples. We speculate that this due to Pythia being trained on less coding-specific data than Llama and Gemma.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Repeated Sampling Can Help Balance Performance and Cost</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">One takeaway from the results in Sections <a href="#S2.SS1" title="2.1 Repeated Sampling is Effective Across Tasks ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a> and <a href="#S2.SS2" title="2.2 Repeated Sampling is Effective Across Model Sizes and Families ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a> is that repeated sampling makes it possible to amplify a weaker model’s capabilities and outperform single samples from stronger models.
Here, we demonstrate that this amplification can be more cost-effective than using a stronger, more expensive model, providing practitioners with a new degree of freedom when trying to jointly optimize performance and costs.</p>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<p id="S2.SS3.p2.1" class="ltx_p">We first consider FLOPs as a cost metric, examining the Llama-3 results from Section <a href="#S2.SS1" title="2.1 Repeated Sampling is Effective Across Tasks ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>. We re-plot our results from Figure <a href="#S2.F2" title="Figure 2 ‣ 2.1 Repeated Sampling is Effective Across Tasks ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, now visualizing coverage as a function of total inference FLOPs instead of the sample budget. Since Llama-3 models are dense transformers where the majority of parameters are used in matrix multiplications, we approximate inference FLOPs with the formula:</p>
</div>
<div id="S2.SS3.p3" class="ltx_para">
<table id="A5.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.Ex1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><span id="S2.Ex1.2.1.1.1" class="ltx_text ltx_markedasmath">FLOPs per token</span></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.Ex1.m2.1" class="ltx_Math" alttext="\displaystyle\approx 2*\left(\text{num parameters}+2*\text{num layers}*\text{token dim}*\text{context length}\right)" display="inline"><semantics id="S2.Ex1.m2.1a"><mrow id="S2.Ex1.m2.1.1" xref="S2.Ex1.m2.1.1.cmml"><mi id="S2.Ex1.m2.1.1.3" xref="S2.Ex1.m2.1.1.3.cmml"></mi><mo id="S2.Ex1.m2.1.1.2" xref="S2.Ex1.m2.1.1.2.cmml">≈</mo><mrow id="S2.Ex1.m2.1.1.1" xref="S2.Ex1.m2.1.1.1.cmml"><mn id="S2.Ex1.m2.1.1.1.3" xref="S2.Ex1.m2.1.1.1.3.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S2.Ex1.m2.1.1.1.2" xref="S2.Ex1.m2.1.1.1.2.cmml">∗</mo><mrow id="S2.Ex1.m2.1.1.1.1.1" xref="S2.Ex1.m2.1.1.1.1.1.1.cmml"><mo id="S2.Ex1.m2.1.1.1.1.1.2" xref="S2.Ex1.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.Ex1.m2.1.1.1.1.1.1" xref="S2.Ex1.m2.1.1.1.1.1.1.cmml"><mtext id="S2.Ex1.m2.1.1.1.1.1.1.2" xref="S2.Ex1.m2.1.1.1.1.1.1.2a.cmml">num parameters</mtext><mo id="S2.Ex1.m2.1.1.1.1.1.1.1" xref="S2.Ex1.m2.1.1.1.1.1.1.1.cmml">+</mo><mrow id="S2.Ex1.m2.1.1.1.1.1.1.3" xref="S2.Ex1.m2.1.1.1.1.1.1.3.cmml"><mn id="S2.Ex1.m2.1.1.1.1.1.1.3.2" xref="S2.Ex1.m2.1.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S2.Ex1.m2.1.1.1.1.1.1.3.1" xref="S2.Ex1.m2.1.1.1.1.1.1.3.1.cmml">∗</mo><mtext id="S2.Ex1.m2.1.1.1.1.1.1.3.3" xref="S2.Ex1.m2.1.1.1.1.1.1.3.3a.cmml">num layers</mtext><mo lspace="0.222em" rspace="0.222em" id="S2.Ex1.m2.1.1.1.1.1.1.3.1a" xref="S2.Ex1.m2.1.1.1.1.1.1.3.1.cmml">∗</mo><mtext id="S2.Ex1.m2.1.1.1.1.1.1.3.4" xref="S2.Ex1.m2.1.1.1.1.1.1.3.4a.cmml">token dim</mtext><mo lspace="0.222em" rspace="0.222em" id="S2.Ex1.m2.1.1.1.1.1.1.3.1b" xref="S2.Ex1.m2.1.1.1.1.1.1.3.1.cmml">∗</mo><mtext id="S2.Ex1.m2.1.1.1.1.1.1.3.5" xref="S2.Ex1.m2.1.1.1.1.1.1.3.5a.cmml">context length</mtext></mrow></mrow><mo id="S2.Ex1.m2.1.1.1.1.1.3" xref="S2.Ex1.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex1.m2.1b"><apply id="S2.Ex1.m2.1.1.cmml" xref="S2.Ex1.m2.1.1"><approx id="S2.Ex1.m2.1.1.2.cmml" xref="S2.Ex1.m2.1.1.2"></approx><csymbol cd="latexml" id="S2.Ex1.m2.1.1.3.cmml" xref="S2.Ex1.m2.1.1.3">absent</csymbol><apply id="S2.Ex1.m2.1.1.1.cmml" xref="S2.Ex1.m2.1.1.1"><times id="S2.Ex1.m2.1.1.1.2.cmml" xref="S2.Ex1.m2.1.1.1.2"></times><cn type="integer" id="S2.Ex1.m2.1.1.1.3.cmml" xref="S2.Ex1.m2.1.1.1.3">2</cn><apply id="S2.Ex1.m2.1.1.1.1.1.1.cmml" xref="S2.Ex1.m2.1.1.1.1.1"><plus id="S2.Ex1.m2.1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.1"></plus><ci id="S2.Ex1.m2.1.1.1.1.1.1.2a.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.2"><mtext id="S2.Ex1.m2.1.1.1.1.1.1.2.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.2">num parameters</mtext></ci><apply id="S2.Ex1.m2.1.1.1.1.1.1.3.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.3"><times id="S2.Ex1.m2.1.1.1.1.1.1.3.1.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.3.1"></times><cn type="integer" id="S2.Ex1.m2.1.1.1.1.1.1.3.2.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.3.2">2</cn><ci id="S2.Ex1.m2.1.1.1.1.1.1.3.3a.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.3.3"><mtext id="S2.Ex1.m2.1.1.1.1.1.1.3.3.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.3.3">num layers</mtext></ci><ci id="S2.Ex1.m2.1.1.1.1.1.1.3.4a.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.3.4"><mtext id="S2.Ex1.m2.1.1.1.1.1.1.3.4.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.3.4">token dim</mtext></ci><ci id="S2.Ex1.m2.1.1.1.1.1.1.3.5a.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.3.5"><mtext id="S2.Ex1.m2.1.1.1.1.1.1.3.5.cmml" xref="S2.Ex1.m2.1.1.1.1.1.1.3.5">context length</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex1.m2.1c">\displaystyle\approx 2*\left(\text{num parameters}+2*\text{num layers}*\text{token dim}*\text{context length}\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.Ex2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><span id="S2.Ex2.2.1.1.1" class="ltx_text ltx_markedasmath">total inference FLOPs</span></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.Ex2.m2.1" class="ltx_Math" alttext="\displaystyle\approx\text{num prompt tokens}*\text{FLOPs per token}" display="inline"><semantics id="S2.Ex2.m2.1a"><mrow id="S2.Ex2.m2.1.1" xref="S2.Ex2.m2.1.1.cmml"><mi id="S2.Ex2.m2.1.1.2" xref="S2.Ex2.m2.1.1.2.cmml"></mi><mo id="S2.Ex2.m2.1.1.1" xref="S2.Ex2.m2.1.1.1.cmml">≈</mo><mrow id="S2.Ex2.m2.1.1.3" xref="S2.Ex2.m2.1.1.3.cmml"><mtext id="S2.Ex2.m2.1.1.3.2" xref="S2.Ex2.m2.1.1.3.2a.cmml">num prompt tokens</mtext><mo lspace="0.222em" rspace="0.222em" id="S2.Ex2.m2.1.1.3.1" xref="S2.Ex2.m2.1.1.3.1.cmml">∗</mo><mtext id="S2.Ex2.m2.1.1.3.3" xref="S2.Ex2.m2.1.1.3.3a.cmml">FLOPs per token</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex2.m2.1b"><apply id="S2.Ex2.m2.1.1.cmml" xref="S2.Ex2.m2.1.1"><approx id="S2.Ex2.m2.1.1.1.cmml" xref="S2.Ex2.m2.1.1.1"></approx><csymbol cd="latexml" id="S2.Ex2.m2.1.1.2.cmml" xref="S2.Ex2.m2.1.1.2">absent</csymbol><apply id="S2.Ex2.m2.1.1.3.cmml" xref="S2.Ex2.m2.1.1.3"><times id="S2.Ex2.m2.1.1.3.1.cmml" xref="S2.Ex2.m2.1.1.3.1"></times><ci id="S2.Ex2.m2.1.1.3.2a.cmml" xref="S2.Ex2.m2.1.1.3.2"><mtext id="S2.Ex2.m2.1.1.3.2.cmml" xref="S2.Ex2.m2.1.1.3.2">num prompt tokens</mtext></ci><ci id="S2.Ex2.m2.1.1.3.3a.cmml" xref="S2.Ex2.m2.1.1.3.3"><mtext id="S2.Ex2.m2.1.1.3.3.cmml" xref="S2.Ex2.m2.1.1.3.3">FLOPs per token</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex2.m2.1c">\displaystyle\approx\text{num prompt tokens}*\text{FLOPs per token}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.Ex3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.Ex3.m1.1" class="ltx_Math" alttext="\displaystyle+\text{num decoded tokens}*\text{FLOPs per token}*\text{num completions}" display="inline"><semantics id="S2.Ex3.m1.1a"><mrow id="S2.Ex3.m1.1.1" xref="S2.Ex3.m1.1.1.cmml"><mo id="S2.Ex3.m1.1.1a" xref="S2.Ex3.m1.1.1.cmml">+</mo><mrow id="S2.Ex3.m1.1.1.2" xref="S2.Ex3.m1.1.1.2.cmml"><mtext id="S2.Ex3.m1.1.1.2.2" xref="S2.Ex3.m1.1.1.2.2a.cmml">num decoded tokens</mtext><mo lspace="0.222em" rspace="0.222em" id="S2.Ex3.m1.1.1.2.1" xref="S2.Ex3.m1.1.1.2.1.cmml">∗</mo><mtext id="S2.Ex3.m1.1.1.2.3" xref="S2.Ex3.m1.1.1.2.3a.cmml">FLOPs per token</mtext><mo lspace="0.222em" rspace="0.222em" id="S2.Ex3.m1.1.1.2.1a" xref="S2.Ex3.m1.1.1.2.1.cmml">∗</mo><mtext id="S2.Ex3.m1.1.1.2.4" xref="S2.Ex3.m1.1.1.2.4a.cmml">num completions</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex3.m1.1b"><apply id="S2.Ex3.m1.1.1.cmml" xref="S2.Ex3.m1.1.1"><plus id="S2.Ex3.m1.1.1.1.cmml" xref="S2.Ex3.m1.1.1"></plus><apply id="S2.Ex3.m1.1.1.2.cmml" xref="S2.Ex3.m1.1.1.2"><times id="S2.Ex3.m1.1.1.2.1.cmml" xref="S2.Ex3.m1.1.1.2.1"></times><ci id="S2.Ex3.m1.1.1.2.2a.cmml" xref="S2.Ex3.m1.1.1.2.2"><mtext id="S2.Ex3.m1.1.1.2.2.cmml" xref="S2.Ex3.m1.1.1.2.2">num decoded tokens</mtext></ci><ci id="S2.Ex3.m1.1.1.2.3a.cmml" xref="S2.Ex3.m1.1.1.2.3"><mtext id="S2.Ex3.m1.1.1.2.3.cmml" xref="S2.Ex3.m1.1.1.2.3">FLOPs per token</mtext></ci><ci id="S2.Ex3.m1.1.1.2.4a.cmml" xref="S2.Ex3.m1.1.1.2.4"><mtext id="S2.Ex3.m1.1.1.2.4.cmml" xref="S2.Ex3.m1.1.1.2.4">num completions</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex3.m1.1c">\displaystyle+\text{num decoded tokens}*\text{FLOPs per token}*\text{num completions}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS3.p4" class="ltx_para">
<p id="S2.SS3.p4.1" class="ltx_p">We present our re-scaled results for MiniF2F, CodeContests, MATH, and GSM8K in Figure <a href="#S2.F4" title="Figure 4 ‣ 2.3 Repeated Sampling Can Help Balance Performance and Cost ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. Interestingly, the model that maximizes coverage varies with the compute budget and task. On MiniF2F, GSM8K and MATH, Llama-3-8B-Instruct always obtains higher coverage than the larger (and more expensive) 70B model when the FLOP budget is fixed.
However for CodeContests, the 70B model is almost always more cost effective.
We note that examining FLOPs alone can be a crude cost metric that ignores other aspects of system efficiency <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite>.
In particular, repeated sampling can make use of high batch sizes and specialized optimizations that improve system throughput relative to single-attempt inference workloads <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>, <a href="#bib.bib6" title="" class="ltx_ref">6</a>, <a href="#bib.bib61" title="" class="ltx_ref">61</a>]</cite>.
We discuss this in more detail in Section <a href="#S5" title="5 Discussion and Limitations ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<figure id="S2.F4" class="ltx_figure"><img src="/html/2407.21787/assets/x4.png" id="S2.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="145" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F4.2.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S2.F4.3.2" class="ltx_text" style="font-size:90%;">Comparing cost, measured in number of inference FLOPs, and coverage for Llama-3-8B-Instruct and Llama-3-70B-Instruct. We see that the ideal model size depends on the task, compute budget, and coverage requirements. Note that Llama-3-70B-Instruct does not achieve 100% coverage on GSM8K due to an incorrectly labelled ground truth answer: see Appendix <a href="#A5" title="Appendix E GSM8K incorrect answer ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>.</span></figcaption>
</figure>
<div id="S2.SS3.p5" class="ltx_para">
<p id="S2.SS3.p5.1" class="ltx_p">We also examine the dollar costs of repeated sampling when solving SWE-bench Lite issues using current API pricing. Keeping the agent framework (Moatless Tools) constant, we consider making a single attempt per issue with Claude 3.5 Sonnet and GPT-4o, as well as repeated sampling using DeepSeek-V2-Coder-Instruct. We report the average cost per issue and issue resolution rate with each approach in Table <a href="#S2.T1" title="Table 1 ‣ 2.3 Repeated Sampling Can Help Balance Performance and Cost ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. While the DeepSeek model is weaker than the GPT and Claude models, it is also over 10x cheaper. In this case, repeated sampling provides a cheaper alternative to paying a premium for access to strong models while achieving a superior issue solve rate.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<div id="S2.T1.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:346.9pt;height:88.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-49.0pt,12.5pt) scale(0.779786843569064,0.779786843569064) ;">
<table id="S2.T1.2.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S2.T1.2.1.1.1" class="ltx_tr">
<th id="S2.T1.2.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.2.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S2.T1.2.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<table id="S2.T1.2.1.1.1.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S2.T1.2.1.1.1.2.1.1" class="ltx_tr">
<td id="S2.T1.2.1.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.2.1.1.1.1" class="ltx_text ltx_font_bold">Cost per</span></td>
</tr>
<tr id="S2.T1.2.1.1.1.2.1.2" class="ltx_tr">
<td id="S2.T1.2.1.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.2.1.2.1.1" class="ltx_text ltx_font_bold">attempt</span></td>
</tr>
<tr id="S2.T1.2.1.1.1.2.1.3" class="ltx_tr">
<td id="S2.T1.2.1.1.1.2.1.3.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.2.1.3.1.1" class="ltx_text ltx_font_bold">(USD)</span></td>
</tr>
</table>
</th>
<th id="S2.T1.2.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<table id="S2.T1.2.1.1.1.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S2.T1.2.1.1.1.3.1.1" class="ltx_tr">
<td id="S2.T1.2.1.1.1.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.3.1.1.1.1" class="ltx_text ltx_font_bold">Number of</span></td>
</tr>
<tr id="S2.T1.2.1.1.1.3.1.2" class="ltx_tr">
<td id="S2.T1.2.1.1.1.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.3.1.2.1.1" class="ltx_text ltx_font_bold">attempts</span></td>
</tr>
</table>
</th>
<th id="S2.T1.2.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<table id="S2.T1.2.1.1.1.4.1" class="ltx_tabular ltx_align_middle">
<tr id="S2.T1.2.1.1.1.4.1.1" class="ltx_tr">
<td id="S2.T1.2.1.1.1.4.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.4.1.1.1.1" class="ltx_text ltx_font_bold">Issues</span></td>
</tr>
<tr id="S2.T1.2.1.1.1.4.1.2" class="ltx_tr">
<td id="S2.T1.2.1.1.1.4.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.4.1.2.1.1" class="ltx_text ltx_font_bold">solved (%)</span></td>
</tr>
</table>
</th>
<th id="S2.T1.2.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<table id="S2.T1.2.1.1.1.5.1" class="ltx_tabular ltx_align_middle">
<tr id="S2.T1.2.1.1.1.5.1.1" class="ltx_tr">
<td id="S2.T1.2.1.1.1.5.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.5.1.1.1.1" class="ltx_text ltx_font_bold">Total Cost</span></td>
</tr>
<tr id="S2.T1.2.1.1.1.5.1.2" class="ltx_tr">
<td id="S2.T1.2.1.1.1.5.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.5.1.2.1.1" class="ltx_text ltx_font_bold">(USD)</span></td>
</tr>
</table>
</th>
<th id="S2.T1.2.1.1.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<table id="S2.T1.2.1.1.1.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S2.T1.2.1.1.1.6.1.1" class="ltx_tr">
<td id="S2.T1.2.1.1.1.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.6.1.1.1.1" class="ltx_text ltx_font_bold">Relative</span></td>
</tr>
<tr id="S2.T1.2.1.1.1.6.1.2" class="ltx_tr">
<td id="S2.T1.2.1.1.1.6.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S2.T1.2.1.1.1.6.1.2.1.1" class="ltx_text ltx_font_bold">Total Cost</span></td>
</tr>
</table>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S2.T1.2.1.2.1" class="ltx_tr">
<td id="S2.T1.2.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t">DeepSeek-V2-Coder-Instruct</td>
<td id="S2.T1.2.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">0.008</td>
<td id="S2.T1.2.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">5</td>
<td id="S2.T1.2.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">29.62</td>
<td id="S2.T1.2.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">12</td>
<td id="S2.T1.2.1.2.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">1x</td>
</tr>
<tr id="S2.T1.2.1.3.2" class="ltx_tr">
<td id="S2.T1.2.1.3.2.1" class="ltx_td ltx_align_left">GPT-4o</td>
<td id="S2.T1.2.1.3.2.2" class="ltx_td ltx_align_center">0.13</td>
<td id="S2.T1.2.1.3.2.3" class="ltx_td ltx_align_center">1</td>
<td id="S2.T1.2.1.3.2.4" class="ltx_td ltx_align_center">24.00</td>
<td id="S2.T1.2.1.3.2.5" class="ltx_td ltx_align_center">39</td>
<td id="S2.T1.2.1.3.2.6" class="ltx_td ltx_nopad_r ltx_align_center">3.25x</td>
</tr>
<tr id="S2.T1.2.1.4.3" class="ltx_tr">
<td id="S2.T1.2.1.4.3.1" class="ltx_td ltx_align_left ltx_border_bb">Claude 3.5 Sonnet</td>
<td id="S2.T1.2.1.4.3.2" class="ltx_td ltx_align_center ltx_border_bb">0.17</td>
<td id="S2.T1.2.1.4.3.3" class="ltx_td ltx_align_center ltx_border_bb">1</td>
<td id="S2.T1.2.1.4.3.4" class="ltx_td ltx_align_center ltx_border_bb">26.70</td>
<td id="S2.T1.2.1.4.3.5" class="ltx_td ltx_align_center ltx_border_bb">51</td>
<td id="S2.T1.2.1.4.3.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">4.25x</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S2.T1.3.1.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="S2.T1.4.2" class="ltx_text" style="font-size:90%;">Comparing API cost (in US dollars) and performance for various models on the SWE-bench Lite dataset using the Moatless Tools agent framework. When sampled more, the open-source DeepSeekCoder-V2 model can achieve the same issue solve-rate as closed-source frontier models for under a third of the price.</span></figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Characterizing the Benefits of Repeated Sampling</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">The relationship between an LLM’s loss and its training compute has been well-characterized with training scaling laws <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>, <a href="#bib.bib33" title="" class="ltx_ref">33</a>, <a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite>.
These laws have empirically held over many orders of magnitude and inspire confidence in model developers that large investments in training will pay off.
Inspired by training scaling laws, here we aim to better characterize the relationship between coverage and the sample budget (i.e. the amount of inference compute), presenting two interesting observations:</p>
</div>
<div id="S3.p2" class="ltx_para">
<ol id="S3.I1" class="ltx_enumerate">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p">The relationship between coverage and the number of samples can often be modelled with an exponentiated power law.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.1" class="ltx_p">For a given task, the coverage curves of different models from the same family resemble S-curves with similar slopes but distinct horizontal offsets.</p>
</div>
</li>
</ol>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Scaling Laws for Repeated Sampling</h3>

<figure id="S3.F5" class="ltx_figure"><img src="/html/2407.21787/assets/x5.png" id="S3.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="245" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F5.2.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="S3.F5.3.2" class="ltx_text" style="font-size:90%;">The relationship between coverage and the number of samples can be modelled with an exponentiated power law for most tasks and models. We highlight that some curves, such as Llama-3-8B-Instruct on MiniF2F-MATH, do not follow this trend closely.</span></figcaption>
</figure>
<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.2" class="ltx_p">Here, we develop an explicit model for the relationship between coverage and the number of samples. The GPT-4 technical report <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite> finds that the relationship between a model’s mean-log-pass-rate on coding problems and its training compute can be modelled well using a power law. We start by adopting the same function class, but now modelling the log of coverage <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">c</annotation></semantics></math> as a function of the number of samples <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mi id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><ci id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">k</annotation></semantics></math>:</p>
<table id="A5.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E2.m1.2" class="ltx_Math" alttext="\displaystyle\log(c)\approx ak^{-b}" display="inline"><semantics id="S3.E2.m1.2a"><mrow id="S3.E2.m1.2.3" xref="S3.E2.m1.2.3.cmml"><mrow id="S3.E2.m1.2.3.2.2" xref="S3.E2.m1.2.3.2.1.cmml"><mi id="S3.E2.m1.1.1" xref="S3.E2.m1.1.1.cmml">log</mi><mo id="S3.E2.m1.2.3.2.2a" xref="S3.E2.m1.2.3.2.1.cmml">⁡</mo><mrow id="S3.E2.m1.2.3.2.2.1" xref="S3.E2.m1.2.3.2.1.cmml"><mo stretchy="false" id="S3.E2.m1.2.3.2.2.1.1" xref="S3.E2.m1.2.3.2.1.cmml">(</mo><mi id="S3.E2.m1.2.2" xref="S3.E2.m1.2.2.cmml">c</mi><mo stretchy="false" id="S3.E2.m1.2.3.2.2.1.2" xref="S3.E2.m1.2.3.2.1.cmml">)</mo></mrow></mrow><mo id="S3.E2.m1.2.3.1" xref="S3.E2.m1.2.3.1.cmml">≈</mo><mrow id="S3.E2.m1.2.3.3" xref="S3.E2.m1.2.3.3.cmml"><mi id="S3.E2.m1.2.3.3.2" xref="S3.E2.m1.2.3.3.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.2.3.3.1" xref="S3.E2.m1.2.3.3.1.cmml">​</mo><msup id="S3.E2.m1.2.3.3.3" xref="S3.E2.m1.2.3.3.3.cmml"><mi id="S3.E2.m1.2.3.3.3.2" xref="S3.E2.m1.2.3.3.3.2.cmml">k</mi><mrow id="S3.E2.m1.2.3.3.3.3" xref="S3.E2.m1.2.3.3.3.3.cmml"><mo id="S3.E2.m1.2.3.3.3.3a" xref="S3.E2.m1.2.3.3.3.3.cmml">−</mo><mi id="S3.E2.m1.2.3.3.3.3.2" xref="S3.E2.m1.2.3.3.3.3.2.cmml">b</mi></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.2b"><apply id="S3.E2.m1.2.3.cmml" xref="S3.E2.m1.2.3"><approx id="S3.E2.m1.2.3.1.cmml" xref="S3.E2.m1.2.3.1"></approx><apply id="S3.E2.m1.2.3.2.1.cmml" xref="S3.E2.m1.2.3.2.2"><log id="S3.E2.m1.1.1.cmml" xref="S3.E2.m1.1.1"></log><ci id="S3.E2.m1.2.2.cmml" xref="S3.E2.m1.2.2">𝑐</ci></apply><apply id="S3.E2.m1.2.3.3.cmml" xref="S3.E2.m1.2.3.3"><times id="S3.E2.m1.2.3.3.1.cmml" xref="S3.E2.m1.2.3.3.1"></times><ci id="S3.E2.m1.2.3.3.2.cmml" xref="S3.E2.m1.2.3.3.2">𝑎</ci><apply id="S3.E2.m1.2.3.3.3.cmml" xref="S3.E2.m1.2.3.3.3"><csymbol cd="ambiguous" id="S3.E2.m1.2.3.3.3.1.cmml" xref="S3.E2.m1.2.3.3.3">superscript</csymbol><ci id="S3.E2.m1.2.3.3.3.2.cmml" xref="S3.E2.m1.2.3.3.3.2">𝑘</ci><apply id="S3.E2.m1.2.3.3.3.3.cmml" xref="S3.E2.m1.2.3.3.3.3"><minus id="S3.E2.m1.2.3.3.3.3.1.cmml" xref="S3.E2.m1.2.3.3.3.3"></minus><ci id="S3.E2.m1.2.3.3.3.3.2.cmml" xref="S3.E2.m1.2.3.3.3.3.2">𝑏</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.2c">\displaystyle\log(c)\approx ak^{-b}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p1.3" class="ltx_p">where <math id="S3.SS1.p1.3.m1.2" class="ltx_Math" alttext="a,b\in\mathbb{R}" display="inline"><semantics id="S3.SS1.p1.3.m1.2a"><mrow id="S3.SS1.p1.3.m1.2.3" xref="S3.SS1.p1.3.m1.2.3.cmml"><mrow id="S3.SS1.p1.3.m1.2.3.2.2" xref="S3.SS1.p1.3.m1.2.3.2.1.cmml"><mi id="S3.SS1.p1.3.m1.1.1" xref="S3.SS1.p1.3.m1.1.1.cmml">a</mi><mo id="S3.SS1.p1.3.m1.2.3.2.2.1" xref="S3.SS1.p1.3.m1.2.3.2.1.cmml">,</mo><mi id="S3.SS1.p1.3.m1.2.2" xref="S3.SS1.p1.3.m1.2.2.cmml">b</mi></mrow><mo id="S3.SS1.p1.3.m1.2.3.1" xref="S3.SS1.p1.3.m1.2.3.1.cmml">∈</mo><mi id="S3.SS1.p1.3.m1.2.3.3" xref="S3.SS1.p1.3.m1.2.3.3.cmml">ℝ</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m1.2b"><apply id="S3.SS1.p1.3.m1.2.3.cmml" xref="S3.SS1.p1.3.m1.2.3"><in id="S3.SS1.p1.3.m1.2.3.1.cmml" xref="S3.SS1.p1.3.m1.2.3.1"></in><list id="S3.SS1.p1.3.m1.2.3.2.1.cmml" xref="S3.SS1.p1.3.m1.2.3.2.2"><ci id="S3.SS1.p1.3.m1.1.1.cmml" xref="S3.SS1.p1.3.m1.1.1">𝑎</ci><ci id="S3.SS1.p1.3.m1.2.2.cmml" xref="S3.SS1.p1.3.m1.2.2">𝑏</ci></list><ci id="S3.SS1.p1.3.m1.2.3.3.cmml" xref="S3.SS1.p1.3.m1.2.3.3">ℝ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m1.2c">a,b\in\mathbb{R}</annotation></semantics></math> are fitted model parameters.
In order to directly predict coverage, we exponentiate both sides, ending up with the final model of:</p>
<table id="A5.EGx4" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E3.m1.2" class="ltx_Math" alttext="\displaystyle c\approx\exp(ak^{-b})" display="inline"><semantics id="S3.E3.m1.2a"><mrow id="S3.E3.m1.2.2" xref="S3.E3.m1.2.2.cmml"><mi id="S3.E3.m1.2.2.3" xref="S3.E3.m1.2.2.3.cmml">c</mi><mo id="S3.E3.m1.2.2.2" xref="S3.E3.m1.2.2.2.cmml">≈</mo><mrow id="S3.E3.m1.2.2.1.1" xref="S3.E3.m1.2.2.1.2.cmml"><mi id="S3.E3.m1.1.1" xref="S3.E3.m1.1.1.cmml">exp</mi><mo id="S3.E3.m1.2.2.1.1a" xref="S3.E3.m1.2.2.1.2.cmml">⁡</mo><mrow id="S3.E3.m1.2.2.1.1.1" xref="S3.E3.m1.2.2.1.2.cmml"><mo stretchy="false" id="S3.E3.m1.2.2.1.1.1.2" xref="S3.E3.m1.2.2.1.2.cmml">(</mo><mrow id="S3.E3.m1.2.2.1.1.1.1" xref="S3.E3.m1.2.2.1.1.1.1.cmml"><mi id="S3.E3.m1.2.2.1.1.1.1.2" xref="S3.E3.m1.2.2.1.1.1.1.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.2.2.1.1.1.1.1" xref="S3.E3.m1.2.2.1.1.1.1.1.cmml">​</mo><msup id="S3.E3.m1.2.2.1.1.1.1.3" xref="S3.E3.m1.2.2.1.1.1.1.3.cmml"><mi id="S3.E3.m1.2.2.1.1.1.1.3.2" xref="S3.E3.m1.2.2.1.1.1.1.3.2.cmml">k</mi><mrow id="S3.E3.m1.2.2.1.1.1.1.3.3" xref="S3.E3.m1.2.2.1.1.1.1.3.3.cmml"><mo id="S3.E3.m1.2.2.1.1.1.1.3.3a" xref="S3.E3.m1.2.2.1.1.1.1.3.3.cmml">−</mo><mi id="S3.E3.m1.2.2.1.1.1.1.3.3.2" xref="S3.E3.m1.2.2.1.1.1.1.3.3.2.cmml">b</mi></mrow></msup></mrow><mo stretchy="false" id="S3.E3.m1.2.2.1.1.1.3" xref="S3.E3.m1.2.2.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.2b"><apply id="S3.E3.m1.2.2.cmml" xref="S3.E3.m1.2.2"><approx id="S3.E3.m1.2.2.2.cmml" xref="S3.E3.m1.2.2.2"></approx><ci id="S3.E3.m1.2.2.3.cmml" xref="S3.E3.m1.2.2.3">𝑐</ci><apply id="S3.E3.m1.2.2.1.2.cmml" xref="S3.E3.m1.2.2.1.1"><exp id="S3.E3.m1.1.1.cmml" xref="S3.E3.m1.1.1"></exp><apply id="S3.E3.m1.2.2.1.1.1.1.cmml" xref="S3.E3.m1.2.2.1.1.1.1"><times id="S3.E3.m1.2.2.1.1.1.1.1.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1"></times><ci id="S3.E3.m1.2.2.1.1.1.1.2.cmml" xref="S3.E3.m1.2.2.1.1.1.1.2">𝑎</ci><apply id="S3.E3.m1.2.2.1.1.1.1.3.cmml" xref="S3.E3.m1.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.1.1.3.1.cmml" xref="S3.E3.m1.2.2.1.1.1.1.3">superscript</csymbol><ci id="S3.E3.m1.2.2.1.1.1.1.3.2.cmml" xref="S3.E3.m1.2.2.1.1.1.1.3.2">𝑘</ci><apply id="S3.E3.m1.2.2.1.1.1.1.3.3.cmml" xref="S3.E3.m1.2.2.1.1.1.1.3.3"><minus id="S3.E3.m1.2.2.1.1.1.1.3.3.1.cmml" xref="S3.E3.m1.2.2.1.1.1.1.3.3"></minus><ci id="S3.E3.m1.2.2.1.1.1.1.3.3.2.cmml" xref="S3.E3.m1.2.2.1.1.1.1.3.3.2">𝑏</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.2c">\displaystyle c\approx\exp(ak^{-b})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p1.4" class="ltx_p">We provide examples of fitted coverage curves in Figure <a href="#S3.F5" title="Figure 5 ‣ 3.1 Scaling Laws for Repeated Sampling ‣ 3 Characterizing the Benefits of Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, and additional curves in Appendix <a href="#A3.SS2" title="C.2 Additional results ‣ Appendix C Scaling Law Details ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.2</span></a>. While these laws are not as exact as training scaling laws (most strikingly on MiniF2F-MATH), they provide encouraging early evidence that the benefits of inference scaling can be characterized.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Similarities in Coverage Curves Across Models</h3>

<figure id="S3.F6" class="ltx_figure"><img src="/html/2407.21787/assets/x6.png" id="S3.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="138" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F6.6.3.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="S3.F6.4.2" class="ltx_text" style="font-size:90%;">Overlaying the coverage curves from different models belonging to the same family. We perform this overlay by horizontally shifting every curve (with a logarithmic x-axis) so that all curves pass through the point (1, <math id="S3.F6.3.1.m1.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.F6.3.1.m1.1b"><mi id="S3.F6.3.1.m1.1.1" xref="S3.F6.3.1.m1.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.F6.3.1.m1.1c"><ci id="S3.F6.3.1.m1.1.1.cmml" xref="S3.F6.3.1.m1.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F6.3.1.m1.1d">c</annotation></semantics></math>). We pick <math id="S3.F6.4.2.m2.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.F6.4.2.m2.1b"><mi id="S3.F6.4.2.m2.1.1" xref="S3.F6.4.2.m2.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.F6.4.2.m2.1c"><ci id="S3.F6.4.2.m2.1.1.cmml" xref="S3.F6.4.2.m2.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F6.4.2.m2.1d">c</annotation></semantics></math> to be the maximum pass@1 score over all models in the plot. We note that the similarity of the curves post-shifting shows that, within a model family, sampling scaling curves follow a similar shape.</span></figcaption>
</figure>
<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.9" class="ltx_p">Interestingly, when comparing the coverage curves (with a logarithmic x-axis) of different models from the same family on the same task (see Figure <a href="#S2.F3" title="Figure 3 ‣ 2.2 Repeated Sampling is Effective Across Model Sizes and Families ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>), it appears that the traced S-curves have the same slope, but unique horizontal offsets. To investigate this further, we overlay the coverage curves of different models from the same family in Figure <a href="#S3.F6" title="Figure 6 ‣ 3.2 Similarities in Coverage Curves Across Models ‣ 3 Characterizing the Benefits of Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>. We do this by picking an anchor coverage value <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">c</annotation></semantics></math>, and shifting every curve leftward (in log-space) so that each passes through the point (1, <math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><mi id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><ci id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">c</annotation></semantics></math>). This corresponds to a leftward shift by <math id="S3.SS2.p1.3.m3.3" class="ltx_Math" alttext="\log(\text{pass@k}^{-1}(c))" display="inline"><semantics id="S3.SS2.p1.3.m3.3a"><mrow id="S3.SS2.p1.3.m3.3.3.1" xref="S3.SS2.p1.3.m3.3.3.2.cmml"><mi id="S3.SS2.p1.3.m3.2.2" xref="S3.SS2.p1.3.m3.2.2.cmml">log</mi><mo id="S3.SS2.p1.3.m3.3.3.1a" xref="S3.SS2.p1.3.m3.3.3.2.cmml">⁡</mo><mrow id="S3.SS2.p1.3.m3.3.3.1.1" xref="S3.SS2.p1.3.m3.3.3.2.cmml"><mo stretchy="false" id="S3.SS2.p1.3.m3.3.3.1.1.2" xref="S3.SS2.p1.3.m3.3.3.2.cmml">(</mo><mrow id="S3.SS2.p1.3.m3.3.3.1.1.1" xref="S3.SS2.p1.3.m3.3.3.1.1.1.cmml"><msup id="S3.SS2.p1.3.m3.3.3.1.1.1.2" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2.cmml"><mtext id="S3.SS2.p1.3.m3.3.3.1.1.1.2.2" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2.2a.cmml">pass@k</mtext><mrow id="S3.SS2.p1.3.m3.3.3.1.1.1.2.3" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2.3.cmml"><mo id="S3.SS2.p1.3.m3.3.3.1.1.1.2.3a" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2.3.cmml">−</mo><mn id="S3.SS2.p1.3.m3.3.3.1.1.1.2.3.2" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="S3.SS2.p1.3.m3.3.3.1.1.1.1" xref="S3.SS2.p1.3.m3.3.3.1.1.1.1.cmml">​</mo><mrow id="S3.SS2.p1.3.m3.3.3.1.1.1.3.2" xref="S3.SS2.p1.3.m3.3.3.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.3.m3.3.3.1.1.1.3.2.1" xref="S3.SS2.p1.3.m3.3.3.1.1.1.cmml">(</mo><mi id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS2.p1.3.m3.3.3.1.1.1.3.2.2" xref="S3.SS2.p1.3.m3.3.3.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.SS2.p1.3.m3.3.3.1.1.3" xref="S3.SS2.p1.3.m3.3.3.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.3b"><apply id="S3.SS2.p1.3.m3.3.3.2.cmml" xref="S3.SS2.p1.3.m3.3.3.1"><log id="S3.SS2.p1.3.m3.2.2.cmml" xref="S3.SS2.p1.3.m3.2.2"></log><apply id="S3.SS2.p1.3.m3.3.3.1.1.1.cmml" xref="S3.SS2.p1.3.m3.3.3.1.1.1"><times id="S3.SS2.p1.3.m3.3.3.1.1.1.1.cmml" xref="S3.SS2.p1.3.m3.3.3.1.1.1.1"></times><apply id="S3.SS2.p1.3.m3.3.3.1.1.1.2.cmml" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.3.m3.3.3.1.1.1.2.1.cmml" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2">superscript</csymbol><ci id="S3.SS2.p1.3.m3.3.3.1.1.1.2.2a.cmml" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2.2"><mtext id="S3.SS2.p1.3.m3.3.3.1.1.1.2.2.cmml" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2.2">pass@k</mtext></ci><apply id="S3.SS2.p1.3.m3.3.3.1.1.1.2.3.cmml" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2.3"><minus id="S3.SS2.p1.3.m3.3.3.1.1.1.2.3.1.cmml" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2.3"></minus><cn type="integer" id="S3.SS2.p1.3.m3.3.3.1.1.1.2.3.2.cmml" xref="S3.SS2.p1.3.m3.3.3.1.1.1.2.3.2">1</cn></apply></apply><ci id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.3c">\log(\text{pass@k}^{-1}(c))</annotation></semantics></math>, where <math id="S3.SS2.p1.4.m4.1" class="ltx_Math" alttext="\text{pass@k}^{-1}(c)" display="inline"><semantics id="S3.SS2.p1.4.m4.1a"><mrow id="S3.SS2.p1.4.m4.1.2" xref="S3.SS2.p1.4.m4.1.2.cmml"><msup id="S3.SS2.p1.4.m4.1.2.2" xref="S3.SS2.p1.4.m4.1.2.2.cmml"><mtext id="S3.SS2.p1.4.m4.1.2.2.2" xref="S3.SS2.p1.4.m4.1.2.2.2a.cmml">pass@k</mtext><mrow id="S3.SS2.p1.4.m4.1.2.2.3" xref="S3.SS2.p1.4.m4.1.2.2.3.cmml"><mo id="S3.SS2.p1.4.m4.1.2.2.3a" xref="S3.SS2.p1.4.m4.1.2.2.3.cmml">−</mo><mn id="S3.SS2.p1.4.m4.1.2.2.3.2" xref="S3.SS2.p1.4.m4.1.2.2.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.2.1" xref="S3.SS2.p1.4.m4.1.2.1.cmml">​</mo><mrow id="S3.SS2.p1.4.m4.1.2.3.2" xref="S3.SS2.p1.4.m4.1.2.cmml"><mo stretchy="false" id="S3.SS2.p1.4.m4.1.2.3.2.1" xref="S3.SS2.p1.4.m4.1.2.cmml">(</mo><mi id="S3.SS2.p1.4.m4.1.1" xref="S3.SS2.p1.4.m4.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS2.p1.4.m4.1.2.3.2.2" xref="S3.SS2.p1.4.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m4.1b"><apply id="S3.SS2.p1.4.m4.1.2.cmml" xref="S3.SS2.p1.4.m4.1.2"><times id="S3.SS2.p1.4.m4.1.2.1.cmml" xref="S3.SS2.p1.4.m4.1.2.1"></times><apply id="S3.SS2.p1.4.m4.1.2.2.cmml" xref="S3.SS2.p1.4.m4.1.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.1.2.2.1.cmml" xref="S3.SS2.p1.4.m4.1.2.2">superscript</csymbol><ci id="S3.SS2.p1.4.m4.1.2.2.2a.cmml" xref="S3.SS2.p1.4.m4.1.2.2.2"><mtext id="S3.SS2.p1.4.m4.1.2.2.2.cmml" xref="S3.SS2.p1.4.m4.1.2.2.2">pass@k</mtext></ci><apply id="S3.SS2.p1.4.m4.1.2.2.3.cmml" xref="S3.SS2.p1.4.m4.1.2.2.3"><minus id="S3.SS2.p1.4.m4.1.2.2.3.1.cmml" xref="S3.SS2.p1.4.m4.1.2.2.3"></minus><cn type="integer" id="S3.SS2.p1.4.m4.1.2.2.3.2.cmml" xref="S3.SS2.p1.4.m4.1.2.2.3.2">1</cn></apply></apply><ci id="S3.SS2.p1.4.m4.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m4.1c">\text{pass@k}^{-1}(c)</annotation></semantics></math> denotes the closest natural number <math id="S3.SS2.p1.5.m5.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS2.p1.5.m5.1a"><mi id="S3.SS2.p1.5.m5.1.1" xref="S3.SS2.p1.5.m5.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.5.m5.1b"><ci id="S3.SS2.p1.5.m5.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.5.m5.1c">k</annotation></semantics></math> such that <math id="S3.SS2.p1.6.m6.1" class="ltx_Math" alttext="\text{pass@k}=c" display="inline"><semantics id="S3.SS2.p1.6.m6.1a"><mrow id="S3.SS2.p1.6.m6.1.1" xref="S3.SS2.p1.6.m6.1.1.cmml"><mtext id="S3.SS2.p1.6.m6.1.1.2" xref="S3.SS2.p1.6.m6.1.1.2a.cmml">pass@k</mtext><mo id="S3.SS2.p1.6.m6.1.1.1" xref="S3.SS2.p1.6.m6.1.1.1.cmml">=</mo><mi id="S3.SS2.p1.6.m6.1.1.3" xref="S3.SS2.p1.6.m6.1.1.3.cmml">c</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.6.m6.1b"><apply id="S3.SS2.p1.6.m6.1.1.cmml" xref="S3.SS2.p1.6.m6.1.1"><eq id="S3.SS2.p1.6.m6.1.1.1.cmml" xref="S3.SS2.p1.6.m6.1.1.1"></eq><ci id="S3.SS2.p1.6.m6.1.1.2a.cmml" xref="S3.SS2.p1.6.m6.1.1.2"><mtext id="S3.SS2.p1.6.m6.1.1.2.cmml" xref="S3.SS2.p1.6.m6.1.1.2">pass@k</mtext></ci><ci id="S3.SS2.p1.6.m6.1.1.3.cmml" xref="S3.SS2.p1.6.m6.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.6.m6.1c">\text{pass@k}=c</annotation></semantics></math>. We pick <math id="S3.SS2.p1.7.m7.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.SS2.p1.7.m7.1a"><mi id="S3.SS2.p1.7.m7.1.1" xref="S3.SS2.p1.7.m7.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.7.m7.1b"><ci id="S3.SS2.p1.7.m7.1.1.cmml" xref="S3.SS2.p1.7.m7.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.7.m7.1c">c</annotation></semantics></math> to be the maximum pass@1 score over all models from the same family. These similarities demonstrate that across models from the same family, the increase in the log-sample-budget (or equivalently, the multiplicative increase in the sample budget) needed to improve coverage from <math id="S3.SS2.p1.8.m8.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.SS2.p1.8.m8.1a"><mi id="S3.SS2.p1.8.m8.1.1" xref="S3.SS2.p1.8.m8.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.8.m8.1b"><ci id="S3.SS2.p1.8.m8.1.1.cmml" xref="S3.SS2.p1.8.m8.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.8.m8.1c">c</annotation></semantics></math> to <math id="S3.SS2.p1.9.m9.1" class="ltx_Math" alttext="c^{\prime}" display="inline"><semantics id="S3.SS2.p1.9.m9.1a"><msup id="S3.SS2.p1.9.m9.1.1" xref="S3.SS2.p1.9.m9.1.1.cmml"><mi id="S3.SS2.p1.9.m9.1.1.2" xref="S3.SS2.p1.9.m9.1.1.2.cmml">c</mi><mo id="S3.SS2.p1.9.m9.1.1.3" xref="S3.SS2.p1.9.m9.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.9.m9.1b"><apply id="S3.SS2.p1.9.m9.1.1.cmml" xref="S3.SS2.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.9.m9.1.1.1.cmml" xref="S3.SS2.p1.9.m9.1.1">superscript</csymbol><ci id="S3.SS2.p1.9.m9.1.1.2.cmml" xref="S3.SS2.p1.9.m9.1.1.2">𝑐</ci><ci id="S3.SS2.p1.9.m9.1.1.3.cmml" xref="S3.SS2.p1.9.m9.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.9.m9.1c">c^{\prime}</annotation></semantics></math> is approximately constant.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Harnessing Repeated Sampling Requires Precision</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">So far, we have focused on measuring model coverage, characterizing the benefits of repeated sampling under the best-case scenario where we can always identify correct model samples. We now turn to the complementary problem of precision: given a collection of model samples, can we identify the correct ones? In Section <a href="#S4.SS1" title="4.1 Common Verification Methods Don’t Always Scale with the Sample Budget ‣ 4 Harnessing Repeated Sampling Requires Precision ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>, we evaluate two common verification methods (majority voting and reward model scoring) on GSM8K and MATH. Additionally, in Section <a href="#S4.SS2" title="4.2 Verifiers and Software Tasks: Two Cautionary Tales ‣ 4 Harnessing Repeated Sampling Requires Precision ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>, we discuss potential pitfalls when relying on unit tests to identify correct software programs.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Common Verification Methods Don’t Always Scale with the Sample Budget</h3>

<figure id="S4.F7" class="ltx_figure"><img src="" id="S4.F7.g1" class="ltx_graphics ltx_centering ltx_missing ltx_missing_image" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F7.2.1.1" class="ltx_text" style="font-size:90%;">Figure 7</span>: </span><span id="S4.F7.3.2" class="ltx_text" style="font-size:90%;">Comparing coverage (performance with an oracle verifier) to mainstream methods available for picking the correct answer (majority voting, reward model selection and reward model majority voting) as we increase the number of samples. Although near-perfect coverage is achieved, all sample selection methods fail to reach the coverage upper bound and saturate before reaching 100 samples. For every k value, we calculate the metric on 100 subsets of size k then plot the mean and one standard deviation across subsets.</span></figcaption>
</figure>
<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">Of the five tasks we evaluate, only GSM8K and MATH lack tools for automatically verifying solutions. Here, we evaluate two common approaches to deciding on a final answer: calculating a majority vote across samples and using a reward model to assign a score to each sample. We test these techniques on their ability to identify correct solutions from the 10,000 samples we generated with Llama-3-8B-Instruct and Llama-3-70B-Instruct in Section <a href="#S2" title="2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. We benchmark three methods:</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<ol id="S4.I1" class="ltx_enumerate">
<li id="S4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S4.I1.i1.p1" class="ltx_para">
<p id="S4.I1.i1.p1.1" class="ltx_p"><span id="S4.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Majority Vote:</span> We pick the most common final answer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>.</p>
</div>
</li>
<li id="S4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S4.I1.i2.p1" class="ltx_para">
<p id="S4.I1.i2.p1.1" class="ltx_p"><span id="S4.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Reward Model + Best-of-N:</span> We use a reward model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite> to score each solution, and pick the answer from the highest-scoring sample.</p>
</div>
</li>
<li id="S4.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S4.I1.i3.p1" class="ltx_para">
<p id="S4.I1.i3.p1.1" class="ltx_p"><span id="S4.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Reward Model + Majority Vote:</span> We calculate a majority vote where each sample is weighted by its reward model score.</p>
</div>
</li>
</ol>
</div>
<div id="S4.SS1.p3" class="ltx_para">
<p id="S4.SS1.p3.1" class="ltx_p">We use ArmoRM-Llama3-8B-v0.1 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite> as a reward model, which currently has the highest reasoning score on the RewardBench leaderboard <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite> among open-weight models. We report our results in Figure <a href="#S4.F7" title="Figure 7 ‣ 4.1 Common Verification Methods Don’t Always Scale with the Sample Budget ‣ 4 Harnessing Repeated Sampling Requires Precision ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> as we increase the number of samples. While success rate initially increases with the number of samples for all three methods, it plateaus around 100 samples. Meanwhile, coverage continues to increase with the number of samples and exceeds 95%.</p>
</div>
<div id="S4.SS1.p4" class="ltx_para">
<p id="S4.SS1.p4.1" class="ltx_p">In the case of majority voting, this success rate saturation is easily explainable. As the number of samples increases, the proportion of votes allocated to each answer stabilizes, and therefore the success rate plateaus. For some GSM8K and MATH problems, correct solutions are sampled with a probability of 1% or lower (see Figure <a href="#S4.F8" title="Figure 8 ‣ 4.2.2 False Negatives in CodeContests ‣ 4.2 Verifiers and Software Tasks: Two Cautionary Tales ‣ 4 Harnessing Repeated Sampling Requires Precision ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>), making them a minority of samples. As the number of samples increases, rare correct solutions will appear for more problems, increasing coverage but not the success rate with majority voting. In order to fully benefit from repeated sampling, sample identification methods must be able to solve these “needle-in-a-haystack” cases and identify rare, correct samples.</p>
</div>
<div id="S4.SS1.p5" class="ltx_para">
<p id="S4.SS1.p5.1" class="ltx_p">Given the poor performance of the reward model, it is reasonable to wonder how “hard” it is to verify a candidate solution. With GSM8K and MATH, only a sample’s final answer is used for assessing correctness, with the intermediate chains of thought being excluded. If models generated only non-sensical chains of thought before guessing a correct final answer, verification may not be any easier than solving the problem in the first place. We investigate this question by manually evaluating 105 chains of thought from correct Llama-3-8B-Instruct solutions to GSM8K problems, reporting our results in Table <a href="#S4.T2" title="Table 2 ‣ 4.1 Common Verification Methods Don’t Always Scale with the Sample Budget ‣ 4 Harnessing Repeated Sampling Requires Precision ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
We find that over 90% of the chains of thought that we graded are faithful, even among problems where correct answers are generated infrequently.
These correct reasoning steps indicate that there is signal for a verifier to exploit when identifying correct samples. Interestingly, during this process we also identified one GSM8K problem that has an incorrect ground truth answer (see Appendix <a href="#A5" title="Appendix E GSM8K incorrect answer ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>).
This incorrect GSM8K problem is also the only one that Llama-3-70B-Instruct did not generate a “correct” sample for across 10,000 attempts.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<div id="S4.T2.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:390.3pt;height:79.1pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-26.9pt,5.5pt) scale(0.878732352101022,0.878732352101022) ;">
<table id="S4.T2.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.4.1.1.1" class="ltx_tr">
<th id="S4.T2.4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.4.1.1.1.1.1" class="ltx_text ltx_font_bold">Pass@1</span></th>
<th id="S4.T2.4.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.4.1.1.1.2.1" class="ltx_text ltx_font_bold"># Problems</span></th>
<th id="S4.T2.4.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.4.1.1.1.3.1" class="ltx_text ltx_font_bold"># CoT Graded</span></th>
<th id="S4.T2.4.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.4.1.1.1.4.1" class="ltx_text ltx_font_bold">Correct CoT</span></th>
<th id="S4.T2.4.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.4.1.1.1.5.1" class="ltx_text ltx_font_bold">Incorrect CoT</span></th>
<th id="S4.T2.4.1.1.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.4.1.1.1.6.1" class="ltx_text ltx_font_bold">Incorrect Ground Truth</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.4.1.2.1" class="ltx_tr">
<td id="S4.T2.4.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t">0-10%</td>
<td id="S4.T2.4.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">5</td>
<td id="S4.T2.4.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">15</td>
<td id="S4.T2.4.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">11</td>
<td id="S4.T2.4.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">1</td>
<td id="S4.T2.4.1.2.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">1 problem, 3 CoTs</td>
</tr>
<tr id="S4.T2.4.1.3.2" class="ltx_tr">
<td id="S4.T2.4.1.3.2.1" class="ltx_td ltx_align_left">10-25%</td>
<td id="S4.T2.4.1.3.2.2" class="ltx_td ltx_align_center">10</td>
<td id="S4.T2.4.1.3.2.3" class="ltx_td ltx_align_center">30</td>
<td id="S4.T2.4.1.3.2.4" class="ltx_td ltx_align_center">27</td>
<td id="S4.T2.4.1.3.2.5" class="ltx_td ltx_align_center">3</td>
<td id="S4.T2.4.1.3.2.6" class="ltx_td ltx_nopad_r ltx_align_center">0 problems</td>
</tr>
<tr id="S4.T2.4.1.4.3" class="ltx_tr">
<td id="S4.T2.4.1.4.3.1" class="ltx_td ltx_align_left">25-75%</td>
<td id="S4.T2.4.1.4.3.2" class="ltx_td ltx_align_center">29</td>
<td id="S4.T2.4.1.4.3.3" class="ltx_td ltx_align_center">30</td>
<td id="S4.T2.4.1.4.3.4" class="ltx_td ltx_align_center">28</td>
<td id="S4.T2.4.1.4.3.5" class="ltx_td ltx_align_center">2</td>
<td id="S4.T2.4.1.4.3.6" class="ltx_td ltx_nopad_r ltx_align_center">0 problems</td>
</tr>
<tr id="S4.T2.4.1.5.4" class="ltx_tr">
<td id="S4.T2.4.1.5.4.1" class="ltx_td ltx_align_left ltx_border_bb">75-100%</td>
<td id="S4.T2.4.1.5.4.2" class="ltx_td ltx_align_center ltx_border_bb">84</td>
<td id="S4.T2.4.1.5.4.3" class="ltx_td ltx_align_center ltx_border_bb">30</td>
<td id="S4.T2.4.1.5.4.4" class="ltx_td ltx_align_center ltx_border_bb">30</td>
<td id="S4.T2.4.1.5.4.5" class="ltx_td ltx_align_center ltx_border_bb">0</td>
<td id="S4.T2.4.1.5.4.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0 problems</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S4.T2.5.2.1" class="ltx_text" style="font-size:90%;">Table 2</span>: </span><span id="S4.T2.2.1" class="ltx_text" style="font-size:90%;">Human evaluation of the validity of the Chain-of-Thought reasoning in Llama-3-8B-Instruct answers to GSM8K problems. 3 chains of thought were graded per problem. Even for difficult questions, where the model only gets <math id="S4.T2.2.1.m1.1" class="ltx_Math" alttext="\leq 10\%" display="inline"><semantics id="S4.T2.2.1.m1.1b"><mrow id="S4.T2.2.1.m1.1.1" xref="S4.T2.2.1.m1.1.1.cmml"><mi id="S4.T2.2.1.m1.1.1.2" xref="S4.T2.2.1.m1.1.1.2.cmml"></mi><mo id="S4.T2.2.1.m1.1.1.1" xref="S4.T2.2.1.m1.1.1.1.cmml">≤</mo><mrow id="S4.T2.2.1.m1.1.1.3" xref="S4.T2.2.1.m1.1.1.3.cmml"><mn id="S4.T2.2.1.m1.1.1.3.2" xref="S4.T2.2.1.m1.1.1.3.2.cmml">10</mn><mo id="S4.T2.2.1.m1.1.1.3.1" xref="S4.T2.2.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.2.1.m1.1c"><apply id="S4.T2.2.1.m1.1.1.cmml" xref="S4.T2.2.1.m1.1.1"><leq id="S4.T2.2.1.m1.1.1.1.cmml" xref="S4.T2.2.1.m1.1.1.1"></leq><csymbol cd="latexml" id="S4.T2.2.1.m1.1.1.2.cmml" xref="S4.T2.2.1.m1.1.1.2">absent</csymbol><apply id="S4.T2.2.1.m1.1.1.3.cmml" xref="S4.T2.2.1.m1.1.1.3"><csymbol cd="latexml" id="S4.T2.2.1.m1.1.1.3.1.cmml" xref="S4.T2.2.1.m1.1.1.3.1">percent</csymbol><cn type="integer" id="S4.T2.2.1.m1.1.1.3.2.cmml" xref="S4.T2.2.1.m1.1.1.3.2">10</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.1.m1.1d">\leq 10\%</annotation></semantics></math> of samples correct, the CoTs almost always follow valid logical steps. For the model generations and human labels, <a target="_blank" href="https://docs.google.com/spreadsheets/d/1D-suvkheNA4fjLsO2TuwHNqwx2TIECmp" title="" class="ltx_ref ltx_href">see here</a>.</span></figcaption>
</figure>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Verifiers and Software Tasks: Two Cautionary Tales</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">Software development tasks can occupy a middle-ground with respect to available verification tools.
On one hand, the ability to execute and test code allows for a higher degree of automatic verification than is possible with unstructured language tasks.
However, tools like unit tests take a black-box approach to verifying a piece of code and are not as comprehensive as methods like proof checkers.
These imperfections in the verification process can lead to false positives and/or false negatives that are important to consider when applying repeated sampling. Below we provide two examples of software verifier imperfections that we encountered when generating our results from Section <a href="#S2.SS1" title="2.1 Repeated Sampling is Effective Across Tasks ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>.</p>
</div>
<section id="S4.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.2.1 </span>Flaky Tests in SWE-bench Lite</h4>

<div id="S4.SS2.SSS1.p1" class="ltx_para">
<p id="S4.SS2.SSS1.p1.1" class="ltx_p">When producing our results on SWE-bench Lite, we identified that 11.3% of problems have flaky test suites that do not produce consistent results when running them on the same candidate solution.
These flaky tests occasionally classify even the dataset’s ground-truth issue solutions as incorrect. Additionally, the test suites for some issues can be non-determinstic depending on the candidate solution.
For example, two SWE-bench Lite issues involve manipulating Python sets, which are naturally unordered.
The gold solutions for these issues explicitly order the items in the set and pass the test suites reliably. However, some model-generated candidate solutions do not impose such an ordering, and therefore pass the tests on some “lucky” runs and not others.
In Appendix <a href="#A2" title="Appendix B SWE-bench Lite ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>, we list all of the problem IDs where we identified flaky tests. We also report our SWE-bench Lite results from Figure <a href="#S2.F2" title="Figure 2 ‣ 2.1 Repeated Sampling is Effective Across Tasks ‣ 2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> with the problematic issues removed, finding similar results to our evaluations on the whole dataset.</p>
</div>
</section>
<section id="S4.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.2.2 </span>False Negatives in CodeContests</h4>

<div id="S4.SS2.SSS2.p1" class="ltx_para">
<p id="S4.SS2.SSS2.p1.1" class="ltx_p">Each problem from the CodeContests dataset comes with a set of input-output test cases used to asses the correctness of solutions. These test cases are more comprehensive than those from earlier coding benchmarks like APPS <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>, cutting down on the frequency of false positive solutions that pass all test cases but do not fully solve the described problem. However, the construction of the CodeContests test suites leads to false negative solutions that are correct but fail the tests.</p>
</div>
<div id="S4.SS2.SSS2.p2" class="ltx_para">
<p id="S4.SS2.SSS2.p2.1" class="ltx_p">For some CodeContests problems, the problem description allows for multiple distinct correct outputs for a given test input. However, the corresponding test cases do not handle these scenarios, instead requiring that one particular correct output is emitted. Additionally, many CodeContests test cases have been programmatically generated by mutating original test cases from the problem. Some mutated inputs violate the problem’s input specifications (e.g. a mutated input being zero when the description promises a positive integer).
These malformed test cases can lead to inconsistent behaviour between different correct solutions.</p>
</div>
<div id="S4.SS2.SSS2.p3" class="ltx_para">
<p id="S4.SS2.SSS2.p3.1" class="ltx_p">We assess the prevalence of these issues by running each problem’s test suite on the list of correct solutions that CodeContests provides. Of the 122 problems in the test set that have Python3 solutions, we find that 35 problems have “correct” solutions that fail the corresponding tests. Since we do not allow models to view all of a problem’s test cases (and their peculiarities), applying repeated sampling to these problems contains an element of “rolling the dice” to generate a solution that is not only correct, but emits the particular outputs that pass the tests.</p>
</div>
<figure id="S4.F8" class="ltx_figure"><img src="/html/2407.21787/assets/figures/hist2.png" id="S4.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="170" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F8.2.1.1" class="ltx_text" style="font-size:90%;">Figure 8</span>: </span><span id="S4.F8.3.2" class="ltx_text" style="font-size:90%;">Bar charts showing the fraction of samples (out of 10,000 samples) that are correct, for each problem in the subsets of GSM8K and MATH we evaluate on. There is one bar per problem, and the height of the bar corresponds to the fraction of samples that arrive at the
correct answer. Bars are green if self-consistency picked the correct answer and are red otherwise.
We highlight that there are many problems with correct solutions, where the correct solutions are sampled infrequently.</span></figcaption>
</figure>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Discussion and Limitations</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">In this work, we explore repeated sampling as an axis for scaling compute at inference time in order to improve model performance. Across a range of models and tasks, repeated sampling can significantly improve the fraction of problems solved using any generated sample (i.e. coverage). When correct solutions can be identified (either with automatic verification tools or other verification algorithms), repeated sampling can amplify model capabilities during inference. This amplification can make the combination of a weaker model and many samples more performant and cost-effective than using fewer attempts from a stronger, more expensive model.</p>
</div>
<div id="S5.p2" class="ltx_para ltx_noindent">
<p id="S5.p2.1" class="ltx_p"><span id="S5.p2.1.1" class="ltx_text ltx_font_bold">Improving Repeated Sampling:</span> In our experiments, we explore only a simple version of repeated sampling where all attempts to a problem are generated independently of one another using the exact same prompt and hyperparameters. We believe that this setup can be refined to improve performance, particularly along the following directions:</p>
</div>
<div id="S5.p3" class="ltx_para">
<ol id="S5.I1" class="ltx_enumerate">
<li id="S5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S5.I1.i1.p1" class="ltx_para">
<p id="S5.I1.i1.p1.1" class="ltx_p"><span id="S5.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Solution Diversity:</span> We currently rely on a positive sampling temperature as the sole mechanism for creating diversity among samples. Combining this token-level sampling with other, higher-level approaches may be able to further increase diversity. For example, AlphaCode conditions different samples with different metadata tags.</p>
</div>
</li>
<li id="S5.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S5.I1.i2.p1" class="ltx_para">
<p id="S5.I1.i2.p1.1" class="ltx_p"><span id="S5.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Multi-Turn Interactions</span>: Despite automatic verification tools being available when solving CodeContests and MiniF2F problems, we use only a single-turn setup where models generate a solution without any ability to iterate on it. Providing models with execution feedback from these tools should improve solution quality. We are interested in the tradeoffs associated with multi-turn interactions, since each attempt becomes more expensive, but also may be more likely to succeed.</p>
</div>
</li>
<li id="S5.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S5.I1.i3.p1" class="ltx_para">
<p id="S5.I1.i3.p1.1" class="ltx_p"><span id="S5.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Learning From Previous Attempts:</span> Currently, our experiments fully isolate attempts from each other. Access to existing samples, particularly if verification tools can provide feedback on them, may be helpful when generating future attempts.</p>
</div>
</li>
</ol>
</div>
<div id="S5.p4" class="ltx_para ltx_noindent">
<p id="S5.p4.1" class="ltx_p"><span id="S5.p4.1.1" class="ltx_text ltx_font_bold">Repeated Sampling and Inference Systems:</span> Repeated sampling is a distinct LLM inference workload from serving chatbot requests. Production chatbot deployments place an emphasis on low response latencies, and adhering to latency targets can force a lower per-device batch size and reduce hardware utilization. In contrast, when sampling many completions to a single prompt, a larger emphasis can be placed on overall throughput and maximizing hardware utilization. Additionally, repeated sampling can benefit from specialized attention optimizations that exploit overlaps in prompts across sequences <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>, <a href="#bib.bib6" title="" class="ltx_ref">6</a>, <a href="#bib.bib61" title="" class="ltx_ref">61</a>]</cite>. Repeated sampling inference can therefore be accomplished at a lower cost than naively making many parallel requests to a chatbot-oriented API. These cost savings can further motivate choosing to sample many times from a cheaper model instead of fewer times from a more expensive one.</p>
</div>
<div id="S5.p5" class="ltx_para ltx_noindent">
<p id="S5.p5.1" class="ltx_p"><span id="S5.p5.1.1" class="ltx_text ltx_font_bold">Verifiers:</span> Our results from Section <a href="#S4" title="4 Harnessing Repeated Sampling Requires Precision ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> highlight the importance of improving sample verification methods when tools for automatically doing so are unavailable. Equipping models with the ability to assess their own outputs will allow repeated sampling to be scaled to far more tasks. Of particular interest is applying repeated sampling to unstructured tasks like creative writing, which can require a more subjective comparison between different samples than the pass-fail tasks we consider. An alternative direction to developing model-based verifiers is to design converters that can make an unstructured task verifiable, for example by formalizing an informal math statement into a language like Lean so that proof checkers can be applied.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Related Work</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p"><span id="S6.p1.1.1" class="ltx_text ltx_font_bold">Scaling Inference Compute:</span> Methods that perform additional computation during inference have been successful across many areas of deep learning. Across a variety of game environments, state-of-the-art methods leverage inference-time search that examines many possible future game states before deciding on a move <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>, <a href="#bib.bib45" title="" class="ltx_ref">45</a>, <a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>. Similar tree-based methods can also be effective in combination with LLMs, allowing models to better plan and explore different approaches <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref">58</a>, <a href="#bib.bib8" title="" class="ltx_ref">8</a>, <a href="#bib.bib49" title="" class="ltx_ref">49</a>, <a href="#bib.bib50" title="" class="ltx_ref">50</a>]</cite>. Another axis for increasing LLM inference compute allows models to spend tokens deliberating on a problem before coming to a solution <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib57" title="" class="ltx_ref">57</a>, <a href="#bib.bib56" title="" class="ltx_ref">56</a>, <a href="#bib.bib59" title="" class="ltx_ref">59</a>]</cite>. Additionally, multiple models can be ensembled together at inference time to combine their strengths <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>, <a href="#bib.bib14" title="" class="ltx_ref">14</a>, <a href="#bib.bib40" title="" class="ltx_ref">40</a>, <a href="#bib.bib52" title="" class="ltx_ref">52</a>, <a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>. Yet another approach involves using LLMs to critique and refine their own responses <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib39" title="" class="ltx_ref">39</a>, <a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite>.</p>
</div>
<div id="S6.p2" class="ltx_para ltx_noindent">
<p id="S6.p2.1" class="ltx_p"><span id="S6.p2.1.1" class="ltx_text ltx_font_bold">Repeated Sampling:</span> Previous work has demonstrated that repeated sampling can improve LLM capabilities in multiple domains. One of the most effective use cases is coding <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>, <a href="#bib.bib15" title="" class="ltx_ref">15</a>, <a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite>, where performance continues to scale up to a million samples and verification tools (e.g. unit tests) are often available to automatically score every candidate solution.
Recently, <cite class="ltx_cite ltx_citemacro_citet">Greenblatt [<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite> shows that repeated sampling is effective when solving puzzles from the ARC challenge <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>, observing log-linear scaling as the number of samples increases.
In chat applications, repeated sampling combined with best-of-N ranking with a reward model can outperform greedily sampling a single response <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref">28</a>]</cite>.
In domains without automatic verification tools, existing work shows that using majority voting <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite> or a trained model-based verifier <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>, <a href="#bib.bib38" title="" class="ltx_ref">38</a>, <a href="#bib.bib27" title="" class="ltx_ref">27</a>]</cite> to decide on a final answer can improve performance on reasoning tasks relative to taking a single sample. Concurrent with our work, <cite class="ltx_cite ltx_citemacro_citet">Song et al. [<a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite> finds that using the best available sample improves LLM performance on chat, math, and code tasks, sweeping up to a max of 128 samples.</p>
</div>
<div id="S6.p3" class="ltx_para ltx_noindent">
<p id="S6.p3.1" class="ltx_p"><span id="S6.p3.1.1" class="ltx_text ltx_font_bold">Scaling Laws:</span> Characterizing how scaling affects model performance can lead to more informed decisions on how to allocate resources. Scaling laws for LLM training find a power law relationship between loss and the amount of training compute and provide estimates for the optimal model and dataset size given a fixed compute budget  <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>, <a href="#bib.bib33" title="" class="ltx_ref">33</a>, <a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite>.
<cite class="ltx_cite ltx_citemacro_citet">Jones [<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite> finds scaling laws in the context of the board game Hex, observing that performance scales predictably with model size and the difficulty of the problem.
Interestingly, they also show that performance scales with the amount of test-time compute spent while performing tree search.
Recently, <cite class="ltx_cite ltx_citemacro_citet">Shao et al. [<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite> observe scaling laws when augmenting LLMs with external retrieval datasets, finding that performance on retrieval tasks scales smoothly with the size of the retrieval corpus.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Acknowledgements</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">We thank Together AI for partially sponsoring the compute for this project, as well as Rahul Chalamala and Ben Athiwaratkun for their help managing this infrastructure. We thank John Yang for his advice and support when running our SWE-bench experiments. Finally, we are grateful to Mayee Chen, Neel Guha, Quinn McIntyre, Jon Saad-Falcon, and Benjamin Spector for their helpful discussions and feedback throughout this project.</p>
</div>
<div id="S7.p2" class="ltx_para">
<p id="S7.p2.1" class="ltx_p">We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF2247015 (Hardware-Aware), CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under Nos. W911NF-23-2-0184 (Long-context) and W911NF-21-2-0251 (Interactive Human-AI Teaming); ONR under Nos. N000142312633 (Deep Signal Processing); Stanford HAI under No. 247183; NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Meta, Google, and VMWare. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.</p>
</div>
<div id="S7.p3" class="ltx_para">
<p id="S7.p3.1" class="ltx_p">This work was completed with the support of the Clarendon Fund Scholarships.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">aid [2024]</span>
<span class="ltx_bibblock">
Aide.dev, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aide.dev/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aide.dev/</a>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">gpt [2024]</span>
<span class="ltx_bibblock">
Hello gpt-4o, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openai.com/index/hello-gpt-4o/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openai.com/index/hello-gpt-4o/</a>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">met [2024]</span>
<span class="ltx_bibblock">
Meta llama 3, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://llama.meta.com/llama3/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://llama.meta.com/llama3/</a>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">son [2024]</span>
<span class="ltx_bibblock">
Claude 3.5 sonnet, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.anthropic.com/news/claude-3-5-sonnet" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.anthropic.com/news/claude-3-5-sonnet</a>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">voy [2024]</span>
<span class="ltx_bibblock">
Voyage ai, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.voyageai.com/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.voyageai.com/</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Athiwaratkun et al. [2024]</span>
<span class="ltx_bibblock">
Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Haifeng Qian, Hantian Ding, Qing Sun, Jun Wang, Jiacheng Guo, Liangfu Chen, Parminder Bhatia, Ramesh Nallapati, Sudipta Sengupta, and Bing Xiang.

</span>
<span class="ltx_bibblock">Bifurcated attention: Accelerating massively parallel decoding with shared prefixes in llms, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2403.08845" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2403.08845</a>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bai et al. [2022]</span>
<span class="ltx_bibblock">
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.

</span>
<span class="ltx_bibblock">Constitutional ai: Harmlessness from ai feedback, 2022.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Besta et al. [2024]</span>
<span class="ltx_bibblock">
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler.

</span>
<span class="ltx_bibblock">Graph of thoughts: Solving elaborate problems with large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI Conference on Artificial Intelligence</em>, 38(16):17682–17690, March 2024.

</span>
<span class="ltx_bibblock">ISSN 2159-5399.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1609/aaai.v38i16.29720</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://dx.doi.org/10.1609/aaai.v38i16.29720" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://dx.doi.org/10.1609/aaai.v38i16.29720</a>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Biderman et al. [2023]</span>
<span class="ltx_bibblock">
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal.

</span>
<span class="ltx_bibblock">Pythia: A suite for analyzing large language models across training and scaling, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2304.01373" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2304.01373</a>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. [2020a]</span>
<span class="ltx_bibblock">
Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong.

</span>
<span class="ltx_bibblock">Combining deep reinforcement learning and search for imperfect-information games.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 34th International Conference on Neural Information Processing Systems</em>, NIPS ’20, Red Hook, NY, USA, 2020a. Curran Associates Inc.

</span>
<span class="ltx_bibblock">ISBN 9781713829546.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. [2020b]</span>
<span class="ltx_bibblock">
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.

</span>
<span class="ltx_bibblock">Language models are few-shot learners, 2020b.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2005.14165" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2005.14165</a>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Campbell et al. [2002]</span>
<span class="ltx_bibblock">
Murray Campbell, A. Joseph Hoane, and Feng-hsiung Hsu.

</span>
<span class="ltx_bibblock">Deep blue.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Artif. Intell.</em>, 134(1–2):57–83, jan 2002.

</span>
<span class="ltx_bibblock">ISSN 0004-3702.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1016/S0004-3702(01)00129-1</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1016/S0004-3702(01)00129-1" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1016/S0004-3702(01)00129-1</a>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2024a]</span>
<span class="ltx_bibblock">
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan.

</span>
<span class="ltx_bibblock">Alphamath almost zero: process supervision without process, 2024a.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2024b]</span>
<span class="ltx_bibblock">
Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou.

</span>
<span class="ltx_bibblock">Are more llm calls all you need? towards scaling laws of compound inference systems, 2024b.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2403.02419" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2403.02419</a>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2021]</span>
<span class="ltx_bibblock">
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.

</span>
<span class="ltx_bibblock">Evaluating large language models trained on code, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2107.03374" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2107.03374</a>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chollet [2019]</span>
<span class="ltx_bibblock">
François Chollet.

</span>
<span class="ltx_bibblock">On the measure of intelligence, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/1911.01547" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/1911.01547</a>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Christiano et al. [2017]</span>
<span class="ltx_bibblock">
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei.

</span>
<span class="ltx_bibblock">Deep reinforcement learning from human preferences, 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/1706.03741" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/1706.03741</a>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. [2021]</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.

</span>
<span class="ltx_bibblock">Training verifiers to solve math word problems, 2021.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">DeepSeek-AI et al. [2024]</span>
<span class="ltx_bibblock">
DeepSeek-AI et al.

</span>
<span class="ltx_bibblock">Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2405.04434" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2405.04434</a>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dehghani et al. [2022]</span>
<span class="ltx_bibblock">
Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay.

</span>
<span class="ltx_bibblock">The efficiency misnomer, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2110.12894" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2110.12894</a>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. [2023]</span>
<span class="ltx_bibblock">
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.

</span>
<span class="ltx_bibblock">A framework for few-shot language model evaluation, 12 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://zenodo.org/records/10256836" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://zenodo.org/records/10256836</a>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Greenblatt [2024]</span>
<span class="ltx_bibblock">
Ryan Greenblatt.

</span>
<span class="ltx_bibblock">Geting 50
</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.lesswrong.com/posts/Rdwui3wHxCeKb7feK/getting-50-sota-on-arc-agi-with-gpt-4o" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.lesswrong.com/posts/Rdwui3wHxCeKb7feK/getting-50-sota-on-arc-agi-with-gpt-4o</a>, 2024.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. [2021a]</span>
<span class="ltx_bibblock">
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring coding challenge competence with apps, 2021a.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. [2021b]</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring mathematical problem solving with the math dataset, 2021b.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hestness et al. [2017]</span>
<span class="ltx_bibblock">
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.

</span>
<span class="ltx_bibblock">Deep learning scaling is predictable, empirically, 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/1712.00409" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/1712.00409</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hoffmann et al. [2022]</span>
<span class="ltx_bibblock">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.

</span>
<span class="ltx_bibblock">Training compute-optimal large language models, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2203.15556" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2203.15556</a>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hosseini et al. [2024]</span>
<span class="ltx_bibblock">
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal.

</span>
<span class="ltx_bibblock">V-star: Training verifiers for self-taught reasoners, 2024.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Irvine et al. [2023]</span>
<span class="ltx_bibblock">
Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, Xiaoding Lu, Thomas Rialan, and William Beauchamp.

</span>
<span class="ltx_bibblock">Rewarding chatbots for real-world engagement with millions of users, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2303.06135" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2303.06135</a>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. [2023]</span>
<span class="ltx_bibblock">
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin.

</span>
<span class="ltx_bibblock">Llm-blender: Ensembling large language models with pairwise ranking and generative fusion, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2306.02561" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2306.02561</a>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jimenez et al. [2024]</span>
<span class="ltx_bibblock">
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.

</span>
<span class="ltx_bibblock">Swe-bench: Can language models resolve real-world github issues?, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2310.06770" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2310.06770</a>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jones [2021]</span>
<span class="ltx_bibblock">
Andy L. Jones.

</span>
<span class="ltx_bibblock">Scaling scaling laws with board games, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2104.03113" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2104.03113</a>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Juravsky et al. [2024]</span>
<span class="ltx_bibblock">
Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y Fu, Christopher Ré, and Azalia Mirhoseini.

</span>
<span class="ltx_bibblock">Hydragen: High-throughput llm inference with shared prefixes.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.05099</em>, 2024.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaplan et al. [2020]</span>
<span class="ltx_bibblock">
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.

</span>
<span class="ltx_bibblock">Scaling laws for neural language models, 2020.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kulal et al. [2019]</span>
<span class="ltx_bibblock">
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang.

</span>
<span class="ltx_bibblock">Spoc: Search-based pseudocode to code, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/1906.04908" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/1906.04908</a>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lambert et al. [2024]</span>
<span class="ltx_bibblock">
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Rewardbench: Evaluating reward models for language modeling, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2403.13787" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2403.13787</a>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lewkowycz et al. [2022]</span>
<span class="ltx_bibblock">
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra.

</span>
<span class="ltx_bibblock">Solving quantitative reasoning problems with language models, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2206.14858" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2206.14858</a>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2022]</span>
<span class="ltx_bibblock">
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.

</span>
<span class="ltx_bibblock">Competition-level code generation with alphacode.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Science</em>, 378(6624):1092–1097, December 2022.

</span>
<span class="ltx_bibblock">ISSN 1095-9203.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1126/science.abq1158</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://dx.doi.org/10.1126/science.abq1158" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://dx.doi.org/10.1126/science.abq1158</a>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lightman et al. [2023]</span>
<span class="ltx_bibblock">
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.

</span>
<span class="ltx_bibblock">Let’s verify step by step, 2023.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Madaan et al. [2023]</span>
<span class="ltx_bibblock">
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.

</span>
<span class="ltx_bibblock">Self-refine: Iterative refinement with self-feedback, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2303.17651" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2303.17651</a>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ong et al. [2024]</span>
<span class="ltx_bibblock">
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica.

</span>
<span class="ltx_bibblock">Routellm: Learning to route llms with preference data, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2406.18665" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2406.18665</a>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI et al. [2024]</span>
<span class="ltx_bibblock">
OpenAI et al.

</span>
<span class="ltx_bibblock">Gpt-4 technical report, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2303.08774" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2303.08774</a>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. [2019]</span>
<span class="ltx_bibblock">
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock">2019.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rozière et al. [2023]</span>
<span class="ltx_bibblock">
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve.

</span>
<span class="ltx_bibblock">Code llama: Open foundation models for code, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2308.12950" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2308.12950</a>.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shao et al. [2024]</span>
<span class="ltx_bibblock">
Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei Koh.

</span>
<span class="ltx_bibblock">Scaling retrieval-based language models with a trillion-token datastore, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2407.12854" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2407.12854</a>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Silver et al. [2017]</span>
<span class="ltx_bibblock">
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis.

</span>
<span class="ltx_bibblock">Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Song et al. [2024]</span>
<span class="ltx_bibblock">
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin.

</span>
<span class="ltx_bibblock">The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2407.10457" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2407.10457</a>.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Team et al. [2024a]</span>
<span class="ltx_bibblock">
Gemini Team et al.

</span>
<span class="ltx_bibblock">Gemini: A family of highly capable multimodal models, 2024a.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2312.11805" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2312.11805</a>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Team et al. [2024b]</span>
<span class="ltx_bibblock">
Gemma Team et al.

</span>
<span class="ltx_bibblock">Gemma: Open models based on gemini research and technology, 2024b.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2403.08295" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2403.08295</a>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tian et al. [2024]</span>
<span class="ltx_bibblock">
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu.

</span>
<span class="ltx_bibblock">Toward self-improvement of llms via imagination, searching, and criticizing, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2404.12253" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2404.12253</a>.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Trinh et al. [2024]</span>
<span class="ltx_bibblock">
Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong.

</span>
<span class="ltx_bibblock">Solving olympiad geometry without human demonstrations.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">Nature</em>, 625(7995):476–482, 2024.

</span>
<span class="ltx_bibblock">ISSN 1476-4687.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1038/s41586-023-06747-5</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1038/s41586-023-06747-5" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1038/s41586-023-06747-5</a>.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Virtanen et al. [2020]</span>
<span class="ltx_bibblock">
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors.

</span>
<span class="ltx_bibblock">SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.

</span>
<span class="ltx_bibblock"><em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">Nature Methods</em>, 17:261–272, 2020.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1038/s41592-019-0686-2</span>.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wan et al. [2024]</span>
<span class="ltx_bibblock">
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi.

</span>
<span class="ltx_bibblock">Knowledge fusion of large language models, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2401.10491" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2401.10491</a>.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2024a]</span>
<span class="ltx_bibblock">
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang.

</span>
<span class="ltx_bibblock">Interpretable preferences via multi-objective reward modeling and mixture-of-experts, 2024a.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2406.12845" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2406.12845</a>.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2024b]</span>
<span class="ltx_bibblock">
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou.

</span>
<span class="ltx_bibblock">Mixture-of-agents enhances large language model capabilities, 2024b.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2406.04692" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2406.04692</a>.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2023]</span>
<span class="ltx_bibblock">
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.

</span>
<span class="ltx_bibblock">Self-consistency improves chain of thought reasoning in language models, 2023.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. [2023]</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.

</span>
<span class="ltx_bibblock">Chain-of-thought prompting elicits reasoning in large language models, 2023.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. [2022]</span>
<span class="ltx_bibblock">
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.

</span>
<span class="ltx_bibblock">React: Synergizing reasoning and acting in language models, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2210.03629" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2210.03629</a>.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. [2023]</span>
<span class="ltx_bibblock">
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.

</span>
<span class="ltx_bibblock">Tree of thoughts: Deliberate problem solving with large language models, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2305.10601" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2305.10601</a>.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zelikman et al. [2024]</span>
<span class="ltx_bibblock">
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman.

</span>
<span class="ltx_bibblock">Quiet-star: Language models can teach themselves to think before speaking, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2403.09629" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2403.09629</a>.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. [2021]</span>
<span class="ltx_bibblock">
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu.

</span>
<span class="ltx_bibblock">Minif2f: a cross-system benchmark for formal olympiad-level mathematics.

</span>
<span class="ltx_bibblock"><em id="bib.bib60.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2109.00110</em>, 2021.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. [2024]</span>
<span class="ltx_bibblock">
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng.

</span>
<span class="ltx_bibblock">Sglang: Efficient execution of structured language model programs, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2312.07104" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2312.07104</a>.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Örwall [2024]</span>
<span class="ltx_bibblock">
Albert Örwall.

</span>
<span class="ltx_bibblock">Moatless tools.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/aorwall/moatless-tools/tree/a1017b78e3e69e7d205b1a3faa83a7d19fce3fa6" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/aorwall/moatless-tools/tree/a1017b78e3e69e7d205b1a3faa83a7d19fce3fa6</a>, 2024.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Sampling Experimental Setup</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Lean Formal Proofs</h3>

<div id="A1.SS1.p1" class="ltx_para">
<p id="A1.SS1.p1.3" class="ltx_p">We report results on the <math id="A1.SS1.p1.1.m1.1" class="ltx_Math" alttext="130" display="inline"><semantics id="A1.SS1.p1.1.m1.1a"><mn id="A1.SS1.p1.1.m1.1.1" xref="A1.SS1.p1.1.m1.1.1.cmml">130</mn><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.1.m1.1b"><cn type="integer" id="A1.SS1.p1.1.m1.1.1.cmml" xref="A1.SS1.p1.1.m1.1.1">130</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.1.m1.1c">130</annotation></semantics></math> questions in the test set of the <a target="_blank" href="https://github.com/rah4927/lean-dojo-mew/blob/main/MiniF2F/Test.lean" title="" class="ltx_ref ltx_href">lean4 MiniF2F dataset</a> that correspond to formalized MATH problems. This dataset is derived from the <a target="_blank" href="https://github.com/facebookresearch/miniF2F" title="" class="ltx_ref ltx_href">fixed version</a> of the original MiniF2F dataset created by <cite class="ltx_cite ltx_citemacro_citet">Zheng et al. [<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>. We sample with a temperature of <math id="A1.SS1.p1.2.m2.1" class="ltx_Math" alttext="0.5" display="inline"><semantics id="A1.SS1.p1.2.m2.1a"><mn id="A1.SS1.p1.2.m2.1.1" xref="A1.SS1.p1.2.m2.1.1.cmml">0.5</mn><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.2.m2.1b"><cn type="float" id="A1.SS1.p1.2.m2.1.1.cmml" xref="A1.SS1.p1.2.m2.1.1">0.5</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.2.m2.1c">0.5</annotation></semantics></math> and do not use nucleus sampling. We generated <math id="A1.SS1.p1.3.m3.2" class="ltx_Math" alttext="10,000" display="inline"><semantics id="A1.SS1.p1.3.m3.2a"><mrow id="A1.SS1.p1.3.m3.2.3.2" xref="A1.SS1.p1.3.m3.2.3.1.cmml"><mn id="A1.SS1.p1.3.m3.1.1" xref="A1.SS1.p1.3.m3.1.1.cmml">10</mn><mo id="A1.SS1.p1.3.m3.2.3.2.1" xref="A1.SS1.p1.3.m3.2.3.1.cmml">,</mo><mn id="A1.SS1.p1.3.m3.2.2" xref="A1.SS1.p1.3.m3.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.3.m3.2b"><list id="A1.SS1.p1.3.m3.2.3.1.cmml" xref="A1.SS1.p1.3.m3.2.3.2"><cn type="integer" id="A1.SS1.p1.3.m3.1.1.cmml" xref="A1.SS1.p1.3.m3.1.1">10</cn><cn type="integer" id="A1.SS1.p1.3.m3.2.2.cmml" xref="A1.SS1.p1.3.m3.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.3.m3.2c">10,000</annotation></semantics></math> samples per problem. We use proofs of the following 5 theorems from the <a target="_blank" href="https://github.com/rah4927/lean-dojo-mew/blob/main/MiniF2F/Validation.lean" title="" class="ltx_ref ltx_href">validation set</a> as few-shot examples:</p>
<ul id="A1.I1" class="ltx_itemize">
<li id="A1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i1.p1" class="ltx_para">
<p id="A1.I1.i1.p1.1" class="ltx_p"><code id="A1.I1.i1.p1.1.1" class="ltx_verbatim ltx_font_typewriter">mathd_algebra_116</code><span id="A1.I1.i1.p1.1.2" class="ltx_text"></span></p>
</div>
</li>
<li id="A1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i2.p1" class="ltx_para">
<p id="A1.I1.i2.p1.1" class="ltx_p"><code id="A1.I1.i2.p1.1.1" class="ltx_verbatim ltx_font_typewriter">amc12_2000_p5</code><span id="A1.I1.i2.p1.1.2" class="ltx_text"></span></p>
</div>
</li>
<li id="A1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i3.p1" class="ltx_para">
<p id="A1.I1.i3.p1.1" class="ltx_p"><code id="A1.I1.i3.p1.1.1" class="ltx_verbatim ltx_font_typewriter">mathd_algebra_132</code><span id="A1.I1.i3.p1.1.2" class="ltx_text"></span></p>
</div>
</li>
<li id="A1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i4.p1" class="ltx_para">
<p id="A1.I1.i4.p1.1" class="ltx_p"><code id="A1.I1.i4.p1.1.1" class="ltx_verbatim ltx_font_typewriter">mathd_algebra_11</code><span id="A1.I1.i4.p1.1.2" class="ltx_text"></span></p>
</div>
</li>
<li id="A1.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i5.p1" class="ltx_para">
<p id="A1.I1.i5.p1.1" class="ltx_p"><code id="A1.I1.i5.p1.1.1" class="ltx_verbatim ltx_font_typewriter">mathd_numbertheory_84</code><span id="A1.I1.i5.p1.1.2" class="ltx_text"></span></p>
</div>
</li>
</ul>
</div>
<div id="A1.SS1.p2" class="ltx_para ltx_noindent">
<p id="A1.SS1.p2.1" class="ltx_p">Our prompt consists of:</p>
<ol id="A1.I2" class="ltx_enumerate">
<li id="A1.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="A1.I2.i1.p1" class="ltx_para">
<p id="A1.I2.i1.p1.1" class="ltx_p">Few shot examples.</p>
</div>
</li>
<li id="A1.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="A1.I2.i2.p1" class="ltx_para">
<p id="A1.I2.i2.p1.1" class="ltx_p">Header imports present in each problem in the HuggingFace dataset <code id="A1.I2.i2.p1.1.1" class="ltx_verbatim ltx_font_typewriter">cat-searcher/minif2f-lean4</code> dataset, an upload of the lean4 MiniF2F dataset.</p>
</div>
</li>
<li id="A1.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="A1.I2.i3.p1" class="ltx_para">
<p id="A1.I2.i3.p1.2" class="ltx_p">The theorem definition. In order to avoid leaking information about how to solve the theorem from its name, we replace the name of the theorem with <code id="A1.I2.i3.p1.2.1" class="ltx_verbatim ltx_font_typewriter">theorem_i</code>. <math id="A1.I2.i3.p1.1.m1.5" class="ltx_Math" alttext="i\in\{1,2,3,4,5\}" display="inline"><semantics id="A1.I2.i3.p1.1.m1.5a"><mrow id="A1.I2.i3.p1.1.m1.5.6" xref="A1.I2.i3.p1.1.m1.5.6.cmml"><mi id="A1.I2.i3.p1.1.m1.5.6.2" xref="A1.I2.i3.p1.1.m1.5.6.2.cmml">i</mi><mo id="A1.I2.i3.p1.1.m1.5.6.1" xref="A1.I2.i3.p1.1.m1.5.6.1.cmml">∈</mo><mrow id="A1.I2.i3.p1.1.m1.5.6.3.2" xref="A1.I2.i3.p1.1.m1.5.6.3.1.cmml"><mo stretchy="false" id="A1.I2.i3.p1.1.m1.5.6.3.2.1" xref="A1.I2.i3.p1.1.m1.5.6.3.1.cmml">{</mo><mn id="A1.I2.i3.p1.1.m1.1.1" xref="A1.I2.i3.p1.1.m1.1.1.cmml">1</mn><mo id="A1.I2.i3.p1.1.m1.5.6.3.2.2" xref="A1.I2.i3.p1.1.m1.5.6.3.1.cmml">,</mo><mn id="A1.I2.i3.p1.1.m1.2.2" xref="A1.I2.i3.p1.1.m1.2.2.cmml">2</mn><mo id="A1.I2.i3.p1.1.m1.5.6.3.2.3" xref="A1.I2.i3.p1.1.m1.5.6.3.1.cmml">,</mo><mn id="A1.I2.i3.p1.1.m1.3.3" xref="A1.I2.i3.p1.1.m1.3.3.cmml">3</mn><mo id="A1.I2.i3.p1.1.m1.5.6.3.2.4" xref="A1.I2.i3.p1.1.m1.5.6.3.1.cmml">,</mo><mn id="A1.I2.i3.p1.1.m1.4.4" xref="A1.I2.i3.p1.1.m1.4.4.cmml">4</mn><mo id="A1.I2.i3.p1.1.m1.5.6.3.2.5" xref="A1.I2.i3.p1.1.m1.5.6.3.1.cmml">,</mo><mn id="A1.I2.i3.p1.1.m1.5.5" xref="A1.I2.i3.p1.1.m1.5.5.cmml">5</mn><mo stretchy="false" id="A1.I2.i3.p1.1.m1.5.6.3.2.6" xref="A1.I2.i3.p1.1.m1.5.6.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I2.i3.p1.1.m1.5b"><apply id="A1.I2.i3.p1.1.m1.5.6.cmml" xref="A1.I2.i3.p1.1.m1.5.6"><in id="A1.I2.i3.p1.1.m1.5.6.1.cmml" xref="A1.I2.i3.p1.1.m1.5.6.1"></in><ci id="A1.I2.i3.p1.1.m1.5.6.2.cmml" xref="A1.I2.i3.p1.1.m1.5.6.2">𝑖</ci><set id="A1.I2.i3.p1.1.m1.5.6.3.1.cmml" xref="A1.I2.i3.p1.1.m1.5.6.3.2"><cn type="integer" id="A1.I2.i3.p1.1.m1.1.1.cmml" xref="A1.I2.i3.p1.1.m1.1.1">1</cn><cn type="integer" id="A1.I2.i3.p1.1.m1.2.2.cmml" xref="A1.I2.i3.p1.1.m1.2.2">2</cn><cn type="integer" id="A1.I2.i3.p1.1.m1.3.3.cmml" xref="A1.I2.i3.p1.1.m1.3.3">3</cn><cn type="integer" id="A1.I2.i3.p1.1.m1.4.4.cmml" xref="A1.I2.i3.p1.1.m1.4.4">4</cn><cn type="integer" id="A1.I2.i3.p1.1.m1.5.5.cmml" xref="A1.I2.i3.p1.1.m1.5.5">5</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I2.i3.p1.1.m1.5c">i\in\{1,2,3,4,5\}</annotation></semantics></math> for the few-shot examples and <math id="A1.I2.i3.p1.2.m2.1" class="ltx_Math" alttext="i=6" display="inline"><semantics id="A1.I2.i3.p1.2.m2.1a"><mrow id="A1.I2.i3.p1.2.m2.1.1" xref="A1.I2.i3.p1.2.m2.1.1.cmml"><mi id="A1.I2.i3.p1.2.m2.1.1.2" xref="A1.I2.i3.p1.2.m2.1.1.2.cmml">i</mi><mo id="A1.I2.i3.p1.2.m2.1.1.1" xref="A1.I2.i3.p1.2.m2.1.1.1.cmml">=</mo><mn id="A1.I2.i3.p1.2.m2.1.1.3" xref="A1.I2.i3.p1.2.m2.1.1.3.cmml">6</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.I2.i3.p1.2.m2.1b"><apply id="A1.I2.i3.p1.2.m2.1.1.cmml" xref="A1.I2.i3.p1.2.m2.1.1"><eq id="A1.I2.i3.p1.2.m2.1.1.1.cmml" xref="A1.I2.i3.p1.2.m2.1.1.1"></eq><ci id="A1.I2.i3.p1.2.m2.1.1.2.cmml" xref="A1.I2.i3.p1.2.m2.1.1.2">𝑖</ci><cn type="integer" id="A1.I2.i3.p1.2.m2.1.1.3.cmml" xref="A1.I2.i3.p1.2.m2.1.1.3">6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I2.i3.p1.2.m2.1c">i=6</annotation></semantics></math> for the current problem.</p>
</div>
</li>
</ol>
</div>
<div id="A1.SS1.p3" class="ltx_para">
<p id="A1.SS1.p3.2" class="ltx_p">We set <math id="A1.SS1.p3.1.m1.1" class="ltx_Math" alttext="200" display="inline"><semantics id="A1.SS1.p3.1.m1.1a"><mn id="A1.SS1.p3.1.m1.1.1" xref="A1.SS1.p3.1.m1.1.1.cmml">200</mn><annotation-xml encoding="MathML-Content" id="A1.SS1.p3.1.m1.1b"><cn type="integer" id="A1.SS1.p3.1.m1.1.1.cmml" xref="A1.SS1.p3.1.m1.1.1">200</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p3.1.m1.1c">200</annotation></semantics></math> as the max token length for the generated solution. To grade solutions, we use the <code id="A1.SS1.p3.2.1" class="ltx_verbatim ltx_font_typewriter">lean-dojo 1.1.2</code> library with lean version <code id="A1.SS1.p3.2.2" class="ltx_verbatim ltx_font_typewriter">4.3.0-rc2</code>. We set a timeout of <math id="A1.SS1.p3.2.m2.1" class="ltx_Math" alttext="10" display="inline"><semantics id="A1.SS1.p3.2.m2.1a"><mn id="A1.SS1.p3.2.m2.1.1" xref="A1.SS1.p3.2.m2.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="A1.SS1.p3.2.m2.1b"><cn type="integer" id="A1.SS1.p3.2.m2.1.1.cmml" xref="A1.SS1.p3.2.m2.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p3.2.m2.1c">10</annotation></semantics></math> seconds for every tactic step.</p>
</div>
<div id="A1.SS1.p4" class="ltx_para ltx_noindent">
<svg id="A1.SS1.p4.pic1" class="ltx_picture" height="428.83" overflow="visible" version="1.1" width="600"><g transform="translate(0,428.83) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#9F9F9F" fill-opacity="1.0"><path d="M 0 5.91 L 0 422.93 C 0 426.19 2.64 428.83 5.91 428.83 L 594.09 428.83 C 597.36 428.83 600 426.19 600 422.93 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F9F9F9" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 404.72 L 598.03 404.72 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 410.63)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="A1.SS1.p4.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS1.p4.pic1.2.2.2.1.1.1" class="ltx_p"><span id="A1.SS1.p4.pic1.2.2.2.1.1.1.1" class="ltx_text ltx_font_bold">Few-Shot Example</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="379.13" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" class="ltx_p">Write a lean4 proof to the provided formal statement. You have access to the standard mathlib4 library.</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" class="ltx_p">```import Mathlib.Algebra.BigOperators.Basic</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.4" class="ltx_p">import Mathlib.Data.Real.Basic</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.5" class="ltx_p">import Mathlib.Data.Complex.Basic</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.6" class="ltx_p">import Mathlib.Data.Nat.Log</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.7" class="ltx_p">import Mathlib.Data.Complex.Exponential</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.8" class="ltx_p">import Mathlib.NumberTheory.Divisors</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.9" class="ltx_p">import Mathlib.Data.ZMod.Defs</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.10" class="ltx_p">import Mathlib.Data.ZMod.Basic</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.11" class="ltx_p">import Mathlib.Topology.Basic</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.12" class="ltx_p">import Mathlib.Data.Nat.Digits</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.13" class="ltx_p">open BigOperators</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.14" class="ltx_p">open Real</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.15" class="ltx_p">open Nat</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.16" class="ltx_p">open Topology</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.17" class="ltx_p">theorem theorem1</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" class="ltx_p">Int.floor ((9:<math id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\mathbb{R}" display="inline"><semantics id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mi id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">ℝ</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><ci id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">ℝ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1c">\mathbb{R}</annotation></semantics></math>) / 160 * 100) = 5 :=</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.18" class="ltx_p">by (</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.19" class="ltx_p">rw [Int.floor_eq_iff]</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.20" class="ltx_p">constructor</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.21" class="ltx_p">all_goals norm_num</span>
<span id="A1.SS1.p4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.22" class="ltx_p">)```</span>
</span></foreignObject></g></g></svg>
</div>
<div id="A1.SS1.p5" class="ltx_para ltx_noindent">
<svg id="A1.SS1.p5.pic1" class="ltx_picture" height="448.21" overflow="visible" version="1.1" width="600"><g transform="translate(0,448.21) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#9F9F9F" fill-opacity="1.0"><path d="M 0 5.91 L 0 442.3 C 0 445.56 2.64 448.21 5.91 448.21 L 594.09 448.21 C 597.36 448.21 600 445.56 600 442.3 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F9F9F9" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 424.1 L 598.03 424.1 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 430)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="A1.SS1.p5.pic1.11.11.11.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS1.p5.pic1.11.11.11.1.1.1" class="ltx_p"><span id="A1.SS1.p5.pic1.11.11.11.1.1.1.1" class="ltx_text ltx_font_bold">Example Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="398.51" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.11" class="ltx_p">Write a lean4 proof to the provided formal statement. You have access to the standard mathlib4 library.</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.12" class="ltx_p">```import Mathlib.Algebra.BigOperators.Basic</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.13" class="ltx_p">import Mathlib.Data.Real.Basic</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.14" class="ltx_p">import Mathlib.Data.Complex.Basic</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.15" class="ltx_p">import Mathlib.Data.Nat.Log</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.16" class="ltx_p">import Mathlib.Data.Complex.Exponential</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.17" class="ltx_p">import Mathlib.NumberTheory.Divisors</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.18" class="ltx_p">import Mathlib.Data.ZMod.Defs</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.19" class="ltx_p">import Mathlib.Data.ZMod.Basic</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.20" class="ltx_p">import Mathlib.Topology.Basic</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.21" class="ltx_p">import Mathlib.Data.Nat.Digits</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.22" class="ltx_p">open BigOperators</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.23" class="ltx_p">open Real</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.24" class="ltx_p">open Nat</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.25" class="ltx_p">open Topology</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.26" class="ltx_p">theorem theorem6</span>
<span id="A1.SS1.p5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" class="ltx_p">(b h v : <math id="A1.SS1.p5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\mathbb{R}" display="inline"><semantics id="A1.SS1.p5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mi id="A1.SS1.p5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="A1.SS1.p5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">ℝ</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.p5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><ci id="A1.SS1.p5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="A1.SS1.p5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">ℝ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1c">\mathbb{R}</annotation></semantics></math>)</span>
<span id="A1.SS1.p5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7" class="ltx_p">(<math id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1" class="ltx_Math" alttext="h_{0}" display="inline"><semantics id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1a"><msub id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1" xref="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.cmml"><mi id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.2" xref="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.2.cmml">h</mi><mn id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.3" xref="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1b"><apply id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.cmml" xref="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.1.cmml" xref="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1">subscript</csymbol><ci id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.2.cmml" xref="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.2">ℎ</ci><cn type="integer" id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.3.cmml" xref="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m1.1c">h_{0}</annotation></semantics></math> : 0 <math id="A1.SS1.p5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m2.1" class="ltx_Math" alttext="&lt;" display="inline"><semantics id="A1.SS1.p5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m2.1a"><mo id="A1.SS1.p5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m2.1.1" xref="A1.SS1.p5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m2.1.1.cmml">&lt;</mo><annotation-xml encoding="MathML-Content" id="A1.SS1.p5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m2.1b"><lt id="A1.SS1.p5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m2.1.1.cmml" xref="A1.SS1.p5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m2.1.1"></lt></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m2.1c">&lt;</annotation></semantics></math> b <math id="A1.SS1.p5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m3.1" class="ltx_Math" alttext="\wedge" display="inline"><semantics id="A1.SS1.p5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m3.1a"><mo id="A1.SS1.p5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m3.1.1" xref="A1.SS1.p5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m3.1.1.cmml">∧</mo><annotation-xml encoding="MathML-Content" id="A1.SS1.p5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m3.1b"><and id="A1.SS1.p5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m3.1.1.cmml" xref="A1.SS1.p5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m3.1.1"></and></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m3.1c">\wedge</annotation></semantics></math> 0 <math id="A1.SS1.p5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m4.1" class="ltx_Math" alttext="&lt;" display="inline"><semantics id="A1.SS1.p5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m4.1a"><mo id="A1.SS1.p5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m4.1.1" xref="A1.SS1.p5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m4.1.1.cmml">&lt;</mo><annotation-xml encoding="MathML-Content" id="A1.SS1.p5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m4.1b"><lt id="A1.SS1.p5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m4.1.1.cmml" xref="A1.SS1.p5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m4.1.1"></lt></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m4.1c">&lt;</annotation></semantics></math> h <math id="A1.SS1.p5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m5.1" class="ltx_Math" alttext="\wedge" display="inline"><semantics id="A1.SS1.p5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m5.1a"><mo id="A1.SS1.p5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m5.1.1" xref="A1.SS1.p5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m5.1.1.cmml">∧</mo><annotation-xml encoding="MathML-Content" id="A1.SS1.p5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m5.1b"><and id="A1.SS1.p5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m5.1.1.cmml" xref="A1.SS1.p5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m5.1.1"></and></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m5.1c">\wedge</annotation></semantics></math> 0 <math id="A1.SS1.p5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m6.1" class="ltx_Math" alttext="&lt;" display="inline"><semantics id="A1.SS1.p5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m6.1a"><mo id="A1.SS1.p5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m6.1.1" xref="A1.SS1.p5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m6.1.1.cmml">&lt;</mo><annotation-xml encoding="MathML-Content" id="A1.SS1.p5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m6.1b"><lt id="A1.SS1.p5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m6.1.1.cmml" xref="A1.SS1.p5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m6.1.1"></lt></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m6.1c">&lt;</annotation></semantics></math> v)</span>
<span id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8" class="ltx_p">(<math id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1" class="ltx_Math" alttext="h_{1}" display="inline"><semantics id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1a"><msub id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1" xref="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.cmml"><mi id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.2" xref="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.2.cmml">h</mi><mn id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.3" xref="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1b"><apply id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.cmml" xref="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.1.cmml" xref="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1">subscript</csymbol><ci id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.2.cmml" xref="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.2">ℎ</ci><cn type="integer" id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.3.cmml" xref="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m1.1c">h_{1}</annotation></semantics></math> : v = 1 / 3 * (b * h))</span>
<span id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9" class="ltx_p">(<math id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1" class="ltx_Math" alttext="h_{2}" display="inline"><semantics id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1a"><msub id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1" xref="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.cmml"><mi id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.2" xref="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.2.cmml">h</mi><mn id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.3" xref="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1b"><apply id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.cmml" xref="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.1.cmml" xref="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1">subscript</csymbol><ci id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.2.cmml" xref="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.2">ℎ</ci><cn type="integer" id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.3.cmml" xref="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1c">h_{2}</annotation></semantics></math> : b = 30)</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10" class="ltx_p">(<math id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1" class="ltx_Math" alttext="h_{3}" display="inline"><semantics id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1a"><msub id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1" xref="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.cmml"><mi id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.2" xref="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.2.cmml">h</mi><mn id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.3" xref="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1b"><apply id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.cmml" xref="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.1.cmml" xref="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1">subscript</csymbol><ci id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.2.cmml" xref="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.2">ℎ</ci><cn type="integer" id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.3.cmml" xref="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m1.1c">h_{3}</annotation></semantics></math> : h = 13 / 2) :</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.27" class="ltx_p">v = 65 :=</span>
<span id="A1.SS1.p5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.28" class="ltx_p">by (</span>
</span></foreignObject></g></g></svg>
</div>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>CodeContests</h3>

<div id="A1.SS2.p1" class="ltx_para">
<p id="A1.SS2.p1.4" class="ltx_p">We report results on the <math id="A1.SS2.p1.1.m1.1" class="ltx_Math" alttext="140" display="inline"><semantics id="A1.SS2.p1.1.m1.1a"><mn id="A1.SS2.p1.1.m1.1.1" xref="A1.SS2.p1.1.m1.1.1.cmml">140</mn><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.1.m1.1b"><cn type="integer" id="A1.SS2.p1.1.m1.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1">140</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.1.m1.1c">140</annotation></semantics></math> test set questions that do not include image tags in the problem description. We sample with a temperature of <math id="A1.SS2.p1.2.m2.1" class="ltx_Math" alttext="0.6" display="inline"><semantics id="A1.SS2.p1.2.m2.1a"><mn id="A1.SS2.p1.2.m2.1.1" xref="A1.SS2.p1.2.m2.1.1.cmml">0.6</mn><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.2.m2.1b"><cn type="float" id="A1.SS2.p1.2.m2.1.1.cmml" xref="A1.SS2.p1.2.m2.1.1">0.6</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.2.m2.1c">0.6</annotation></semantics></math> and a top-p value of <math id="A1.SS2.p1.3.m3.1" class="ltx_Math" alttext="0.95" display="inline"><semantics id="A1.SS2.p1.3.m3.1a"><mn id="A1.SS2.p1.3.m3.1.1" xref="A1.SS2.p1.3.m3.1.1.cmml">0.95</mn><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.3.m3.1b"><cn type="float" id="A1.SS2.p1.3.m3.1.1.cmml" xref="A1.SS2.p1.3.m3.1.1">0.95</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.3.m3.1c">0.95</annotation></semantics></math> following the experiments in CodeLlama <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite>. We generate 10,000 samples per problem. We use two few-shot examples from the training set that are randomly sampled per-problem. We set <math id="A1.SS2.p1.4.m4.1" class="ltx_Math" alttext="1024" display="inline"><semantics id="A1.SS2.p1.4.m4.1a"><mn id="A1.SS2.p1.4.m4.1.1" xref="A1.SS2.p1.4.m4.1.1.cmml">1024</mn><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.4.m4.1b"><cn type="integer" id="A1.SS2.p1.4.m4.1.1.cmml" xref="A1.SS2.p1.4.m4.1.1">1024</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.4.m4.1c">1024</annotation></semantics></math> as the max token length for the generated solution. We use the same answer comparison function as <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite> and use the concatenation of public, private, and generated tests to validate correctness of solutions.</p>
</div>
<div id="A1.SS2.p2" class="ltx_para ltx_noindent">
<svg id="A1.SS2.p2.pic1" class="ltx_picture" height="229.58" overflow="visible" version="1.1" width="600"><g transform="translate(0,229.58) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#9F9F9F" fill-opacity="1.0"><path d="M 0 5.91 L 0 223.68 C 0 226.94 2.64 229.58 5.91 229.58 L 594.09 229.58 C 597.36 229.58 600 226.94 600 223.68 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F9F9F9" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 205.47 L 598.03 205.47 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 211.38)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="A1.SS2.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS2.p2.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.SS2.p2.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Few-Shot Example</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="179.88" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.SS2.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS2.p2.pic1.2.2.2.1.1.1" class="ltx_p">Q: Write python code to solve the following coding problem that obeys the constraints and passes the example test cases. The output code needs to read from and write to standard IO. Please wrap your code answer using ```:</span>
<span id="A1.SS2.p2.pic1.2.2.2.1.1.2" class="ltx_p">Petya has equal…</span>
<span id="A1.SS2.p2.pic1.2.2.2.1.1.3" class="ltx_p">…</span>
<span id="A1.SS2.p2.pic1.2.2.2.1.1.4" class="ltx_p">(Rest of question and input/output examples)</span>
<span id="A1.SS2.p2.pic1.2.2.2.1.1.5" class="ltx_p">…</span>
<span id="A1.SS2.p2.pic1.2.2.2.1.1.6" class="ltx_p">A: ```</span><pre id="A1.SS2.p2.pic1.2.2.2.1.1.7" class="ltx_verbatim ltx_font_typewriter">
n = int(input())
a = int(input())
b = int(input())
ans = 6
cnt = 0
cur = 2
cnt += 2 * ((n - b) // a)
while cnt &lt; 4:
    cur += 1
    cnt += (n // a)
ans = min(ans, cur)
if b * 2 &lt;= n:
    cur, cnt = 0, 0
    cur = 1
    cnt += ((n - 2 * b) // a)
    while cnt &lt; 4:
        cur += 1
        cnt += (n // a)
    ans = min(ans, cur)
print(ans)
</pre>
<span id="A1.SS2.p2.pic1.2.2.2.1.1.8" class="ltx_p">```</span>
</span></foreignObject></g></g></svg>
</div>
<div id="A1.SS2.p3" class="ltx_para ltx_noindent">
<svg id="A1.SS2.p3.pic1" class="ltx_picture" height="175.39" overflow="visible" version="1.1" width="600"><g transform="translate(0,175.39) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#9F9F9F" fill-opacity="1.0"><path d="M 0 5.91 L 0 169.48 C 0 172.74 2.64 175.39 5.91 175.39 L 594.09 175.39 C 597.36 175.39 600 172.74 600 169.48 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F9F9F9" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 151.28 L 598.03 151.28 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 157.18)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="A1.SS2.p3.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS2.p3.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.SS2.p3.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Example Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="125.69" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.SS2.p3.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS2.p3.pic1.2.2.2.1.1.1" class="ltx_p">Q: Write python code to solve the following coding problem that obeys the constraints and passes the example test cases. The output code needs to read from and write to standard IO. Please wrap your code answer using ```:</span>
<span id="A1.SS2.p3.pic1.2.2.2.1.1.2" class="ltx_p">This is the…</span>
<span id="A1.SS2.p3.pic1.2.2.2.1.1.3" class="ltx_p">…</span>
<span id="A1.SS2.p3.pic1.2.2.2.1.1.4" class="ltx_p">(Rest of question and input/output examples)</span>
<span id="A1.SS2.p3.pic1.2.2.2.1.1.5" class="ltx_p">…</span>
<span id="A1.SS2.p3.pic1.2.2.2.1.1.6" class="ltx_p">A:</span>
</span></foreignObject></g></g></svg>
</div>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>MATH</h3>

<div id="A1.SS3.p1" class="ltx_para">
<p id="A1.SS3.p1.5" class="ltx_p">We report results on <math id="A1.SS3.p1.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="A1.SS3.p1.1.m1.1a"><mn id="A1.SS3.p1.1.m1.1.1" xref="A1.SS3.p1.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.1.m1.1b"><cn type="integer" id="A1.SS3.p1.1.m1.1.1.cmml" xref="A1.SS3.p1.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.1.m1.1c">128</annotation></semantics></math> randomly selected test-set problems. We sample with a temperature of <math id="A1.SS3.p1.2.m2.1" class="ltx_Math" alttext="0.6" display="inline"><semantics id="A1.SS3.p1.2.m2.1a"><mn id="A1.SS3.p1.2.m2.1.1" xref="A1.SS3.p1.2.m2.1.1.cmml">0.6</mn><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.2.m2.1b"><cn type="float" id="A1.SS3.p1.2.m2.1.1.cmml" xref="A1.SS3.p1.2.m2.1.1">0.6</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.2.m2.1c">0.6</annotation></semantics></math> and do not use nucleus sampling. We use the fixed <math id="A1.SS3.p1.3.m3.1" class="ltx_Math" alttext="5" display="inline"><semantics id="A1.SS3.p1.3.m3.1a"><mn id="A1.SS3.p1.3.m3.1.1" xref="A1.SS3.p1.3.m3.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.3.m3.1b"><cn type="integer" id="A1.SS3.p1.3.m3.1.1.cmml" xref="A1.SS3.p1.3.m3.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.3.m3.1c">5</annotation></semantics></math> few-shot example from <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite> for each problem. We generate <math id="A1.SS3.p1.4.m4.2" class="ltx_Math" alttext="10,000" display="inline"><semantics id="A1.SS3.p1.4.m4.2a"><mrow id="A1.SS3.p1.4.m4.2.3.2" xref="A1.SS3.p1.4.m4.2.3.1.cmml"><mn id="A1.SS3.p1.4.m4.1.1" xref="A1.SS3.p1.4.m4.1.1.cmml">10</mn><mo id="A1.SS3.p1.4.m4.2.3.2.1" xref="A1.SS3.p1.4.m4.2.3.1.cmml">,</mo><mn id="A1.SS3.p1.4.m4.2.2" xref="A1.SS3.p1.4.m4.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.4.m4.2b"><list id="A1.SS3.p1.4.m4.2.3.1.cmml" xref="A1.SS3.p1.4.m4.2.3.2"><cn type="integer" id="A1.SS3.p1.4.m4.1.1.cmml" xref="A1.SS3.p1.4.m4.1.1">10</cn><cn type="integer" id="A1.SS3.p1.4.m4.2.2.cmml" xref="A1.SS3.p1.4.m4.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.4.m4.2c">10,000</annotation></semantics></math> samples per problem. We set <math id="A1.SS3.p1.5.m5.1" class="ltx_Math" alttext="512" display="inline"><semantics id="A1.SS3.p1.5.m5.1a"><mn id="A1.SS3.p1.5.m5.1.1" xref="A1.SS3.p1.5.m5.1.1.cmml">512</mn><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.5.m5.1b"><cn type="integer" id="A1.SS3.p1.5.m5.1.1.cmml" xref="A1.SS3.p1.5.m5.1.1">512</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.5.m5.1c">512</annotation></semantics></math> as the max token length for the generated solution. To grade solutions, we use the <code id="A1.SS3.p1.5.1" class="ltx_verbatim ltx_font_typewriter">minerva_math</code> functions from LMEval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>.</p>
</div>
<div id="A1.SS3.p2" class="ltx_para ltx_noindent">
<svg id="A1.SS3.p2.pic1" class="ltx_picture" height="111.81" overflow="visible" version="1.1" width="600"><g transform="translate(0,111.81) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#9F9F9F" fill-opacity="1.0"><path d="M 0 5.91 L 0 105.91 C 0 109.17 2.64 111.81 5.91 111.81 L 594.09 111.81 C 597.36 111.81 600 109.17 600 105.91 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F9F9F9" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 87.7 L 598.03 87.7 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 93.61)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="A1.SS3.p2.pic1.6.6.6.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS3.p2.pic1.6.6.6.1.1.1" class="ltx_p"><span id="A1.SS3.p2.pic1.6.6.6.1.1.1.1" class="ltx_text ltx_font_bold">Few-Shot Example</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="62.11" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.6" class="ltx_p">Problem:</span>
<span id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3" class="ltx_p">If <math id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\det\mathbf{A}=2" display="inline"><semantics id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mrow id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml"><mrow id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2.1" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2.1.cmml">det</mo><mi id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2.2" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2.2.cmml">𝐀</mi></mrow><mo id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.1" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.1.cmml">=</mo><mn id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.3" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><apply id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1"><eq id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.1.cmml" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.1"></eq><apply id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2.cmml" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2"><determinant id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2.1.cmml" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2.1"></determinant><ci id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2.2.cmml" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.2.2">𝐀</ci></apply><cn type="integer" id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.3.cmml" xref="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1c">\det\mathbf{A}=2</annotation></semantics></math> and <math id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1" class="ltx_Math" alttext="\det\mathbf{B}=12," display="inline"><semantics id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1a"><mrow id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.cmml"><mrow id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.cmml"><mrow id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2.1" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2.1.cmml">det</mo><mi id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2.2" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2.2.cmml">𝐁</mi></mrow><mo id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.1" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.1.cmml">=</mo><mn id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.3" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.3.cmml">12</mn></mrow><mo id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.2" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1b"><apply id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.cmml" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1"><eq id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.1.cmml" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.1"></eq><apply id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2.cmml" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2"><determinant id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2.1.cmml" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2.1"></determinant><ci id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2.2.cmml" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.2.2">𝐁</ci></apply><cn type="integer" id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.3.cmml" xref="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.1.1.3">12</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1c">\det\mathbf{B}=12,</annotation></semantics></math> then find <math id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2" class="ltx_Math" alttext="\det(\mathbf{A}\mathbf{B})." display="inline"><semantics id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2a"><mrow id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.cmml"><mrow id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.cmml"><mo rspace="0em" id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.1" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.1.cmml">det</mo><mrow id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.2.2" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.cmml"><mo stretchy="false" id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.2.2.1" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.cmml">(</mo><mi id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1.1" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1.1.cmml">𝐀𝐁</mi><mo stretchy="false" id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.2.2.2" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.cmml">)</mo></mrow></mrow><mo lspace="0em" id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.2" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2b"><apply id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.cmml" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1"><determinant id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.1.cmml" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2.2.1.1.1"></determinant><ci id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1.1.cmml" xref="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1.1">𝐀𝐁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.2c">\det(\mathbf{A}\mathbf{B}).</annotation></semantics></math></span>
<span id="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.7" class="ltx_p">Solution:</span>
<span id="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5" class="ltx_p">We have that <math id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4" class="ltx_Math" alttext="\det(\mathbf{A}\mathbf{B})=(\det\mathbf{A})(\det\mathbf{B})=(2)(12)=\boxed{24}." display="inline"><semantics id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4a"><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.cmml"><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.cmml"><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.cmml"><mo rspace="0em" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.1.cmml">det</mo><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.2.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.cmml"><mo stretchy="false" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.2.2.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.cmml">(</mo><mi id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.1.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.1.1.cmml">𝐀𝐁</mi><mo stretchy="false" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.2.2.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.cmml">)</mo></mrow></mrow><mo id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.5" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.5.cmml">=</mo><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.cmml"><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.cmml"><mo lspace="0em" rspace="0.167em" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.1.cmml">det</mo><mi id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.2.cmml">𝐀</mi></mrow><mo stretchy="false" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.3" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.3" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.3.cmml">​</mo><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.cmml"><mo stretchy="false" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.cmml">(</mo><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.cmml"><mo lspace="0em" rspace="0.167em" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.1.cmml">det</mo><mi id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.2.cmml">𝐁</mi></mrow><mo stretchy="false" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.3" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.cmml">)</mo></mrow></mrow><mo id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.6" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.6.cmml">=</mo><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.cmml"><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.2.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.cmml"><mo stretchy="false" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.2.2.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.cmml">(</mo><mn id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.2.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.2.2.cmml">2</mn><mo stretchy="false" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.2.2.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.1.cmml">​</mo><mrow id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.3.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.cmml"><mo stretchy="false" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.3.2.1" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.cmml">(</mo><mn id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.3.3" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.3.3.cmml">12</mn><mo stretchy="false" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.3.2.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.cmml">)</mo></mrow></mrow><mo id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.8" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.8.cmml">=</mo><menclose notation="box" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.9" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.9.cmml"><mn id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.9a" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.9.cmml">24</mn></menclose></mrow><mo lspace="0em" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.2" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4b"><apply id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1"><and id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1a.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1"></and><apply id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1b.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1"><eq id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.5.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.5"></eq><apply id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4"><determinant id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.1.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.4.1"></determinant><ci id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.1.1.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.1.1">𝐀𝐁</ci></apply><apply id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2"><times id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.3.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.3"></times><apply id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1"><determinant id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.1.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.1"></determinant><ci id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.2.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.1.1.1.1.2">𝐀</ci></apply><apply id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1"><determinant id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.1.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.1"></determinant><ci id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.2.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.2.1.1.2">𝐁</ci></apply></apply></apply><apply id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1c.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1"><eq id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.6.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.6"></eq><share href="#A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.2.cmml" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1d.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1"></share><apply id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7"><times id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.1.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.1"></times><cn type="integer" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.2.2.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.2.2">2</cn><cn type="integer" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.3.3.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.3.3">12</cn></apply></apply><apply id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1e.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1"><eq id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.8.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.8"></eq><share href="#A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.7.cmml" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1f.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1"></share><cn type="integer" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.9.cmml" xref="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4.4.1.1.9">24</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p2.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m1.4c">\det(\mathbf{A}\mathbf{B})=(\det\mathbf{A})(\det\mathbf{B})=(2)(12)=\boxed{24}.</annotation></semantics></math>
Final Answer: The final answer is <math id="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m2.1" class="ltx_Math" alttext="24" display="inline"><semantics id="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m2.1a"><mn id="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m2.1.1" xref="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m2.1.1.cmml">24</mn><annotation-xml encoding="MathML-Content" id="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m2.1b"><cn type="integer" id="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m2.1.1.cmml" xref="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m2.1.1">24</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p2.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m2.1c">24</annotation></semantics></math>. I hope it is correct.</span>
</span></foreignObject></g></g></svg>
</div>
<div id="A1.SS3.p3" class="ltx_para ltx_noindent">
<svg id="A1.SS3.p3.pic1" class="ltx_picture" height="104.97" overflow="visible" version="1.1" width="600"><g transform="translate(0,104.97) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#9F9F9F" fill-opacity="1.0"><path d="M 0 5.91 L 0 99.07 C 0 102.33 2.64 104.97 5.91 104.97 L 594.09 104.97 C 597.36 104.97 600 102.33 600 99.07 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F9F9F9" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 80.86 L 598.03 80.86 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 86.77)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="A1.SS3.p3.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS3.p3.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.SS3.p3.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Example Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="55.27" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.SS3.p3.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS3.p3.pic1.2.2.2.1.1.1" class="ltx_p">Problem:</span>
<span id="A1.SS3.p3.pic1.2.2.2.1.1.2" class="ltx_p">What is the domain of the function</span>
<span id="A1.Ex4" class="ltx_equation ltx_eqn_table">

<span><span class="ltx_equation ltx_eqn_row ltx_align_baseline">
<span class="ltx_eqn_cell ltx_eqn_center_padleft"></span>
<span class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex4.m1.5" class="ltx_Math" alttext="f(x)=\frac{(2x-3)(2x+5)}{(3x-9)(3x+6)}~{}?" display="block"><semantics id="A1.Ex4.m1.5a"><mrow id="A1.Ex4.m1.5.6" xref="A1.Ex4.m1.5.6.cmml"><mrow id="A1.Ex4.m1.5.6.2" xref="A1.Ex4.m1.5.6.2.cmml"><mi id="A1.Ex4.m1.5.6.2.2" xref="A1.Ex4.m1.5.6.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.5.6.2.1" xref="A1.Ex4.m1.5.6.2.1.cmml">​</mo><mrow id="A1.Ex4.m1.5.6.2.3.2" xref="A1.Ex4.m1.5.6.2.cmml"><mo stretchy="false" id="A1.Ex4.m1.5.6.2.3.2.1" xref="A1.Ex4.m1.5.6.2.cmml">(</mo><mi id="A1.Ex4.m1.5.5" xref="A1.Ex4.m1.5.5.cmml">x</mi><mo stretchy="false" id="A1.Ex4.m1.5.6.2.3.2.2" xref="A1.Ex4.m1.5.6.2.cmml">)</mo></mrow></mrow><mo id="A1.Ex4.m1.5.6.1" xref="A1.Ex4.m1.5.6.1.cmml">=</mo><mrow id="A1.Ex4.m1.5.6.3" xref="A1.Ex4.m1.5.6.3.cmml"><mfrac id="A1.Ex4.m1.4.4" xref="A1.Ex4.m1.4.4.cmml"><mrow id="A1.Ex4.m1.2.2.2" xref="A1.Ex4.m1.2.2.2.cmml"><mrow id="A1.Ex4.m1.1.1.1.1.1" xref="A1.Ex4.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.Ex4.m1.1.1.1.1.1.2" xref="A1.Ex4.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.Ex4.m1.1.1.1.1.1.1" xref="A1.Ex4.m1.1.1.1.1.1.1.cmml"><mrow id="A1.Ex4.m1.1.1.1.1.1.1.2" xref="A1.Ex4.m1.1.1.1.1.1.1.2.cmml"><mn id="A1.Ex4.m1.1.1.1.1.1.1.2.2" xref="A1.Ex4.m1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.1.1.1.1.1.1.2.1" xref="A1.Ex4.m1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.Ex4.m1.1.1.1.1.1.1.2.3" xref="A1.Ex4.m1.1.1.1.1.1.1.2.3.cmml">x</mi></mrow><mo id="A1.Ex4.m1.1.1.1.1.1.1.1" xref="A1.Ex4.m1.1.1.1.1.1.1.1.cmml">−</mo><mn id="A1.Ex4.m1.1.1.1.1.1.1.3" xref="A1.Ex4.m1.1.1.1.1.1.1.3.cmml">3</mn></mrow><mo stretchy="false" id="A1.Ex4.m1.1.1.1.1.1.3" xref="A1.Ex4.m1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.2.2.2.3" xref="A1.Ex4.m1.2.2.2.3.cmml">​</mo><mrow id="A1.Ex4.m1.2.2.2.2.1" xref="A1.Ex4.m1.2.2.2.2.1.1.cmml"><mo stretchy="false" id="A1.Ex4.m1.2.2.2.2.1.2" xref="A1.Ex4.m1.2.2.2.2.1.1.cmml">(</mo><mrow id="A1.Ex4.m1.2.2.2.2.1.1" xref="A1.Ex4.m1.2.2.2.2.1.1.cmml"><mrow id="A1.Ex4.m1.2.2.2.2.1.1.2" xref="A1.Ex4.m1.2.2.2.2.1.1.2.cmml"><mn id="A1.Ex4.m1.2.2.2.2.1.1.2.2" xref="A1.Ex4.m1.2.2.2.2.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.2.2.2.2.1.1.2.1" xref="A1.Ex4.m1.2.2.2.2.1.1.2.1.cmml">​</mo><mi id="A1.Ex4.m1.2.2.2.2.1.1.2.3" xref="A1.Ex4.m1.2.2.2.2.1.1.2.3.cmml">x</mi></mrow><mo id="A1.Ex4.m1.2.2.2.2.1.1.1" xref="A1.Ex4.m1.2.2.2.2.1.1.1.cmml">+</mo><mn id="A1.Ex4.m1.2.2.2.2.1.1.3" xref="A1.Ex4.m1.2.2.2.2.1.1.3.cmml">5</mn></mrow><mo stretchy="false" id="A1.Ex4.m1.2.2.2.2.1.3" xref="A1.Ex4.m1.2.2.2.2.1.1.cmml">)</mo></mrow></mrow><mrow id="A1.Ex4.m1.4.4.4" xref="A1.Ex4.m1.4.4.4.cmml"><mrow id="A1.Ex4.m1.3.3.3.1.1" xref="A1.Ex4.m1.3.3.3.1.1.1.cmml"><mo stretchy="false" id="A1.Ex4.m1.3.3.3.1.1.2" xref="A1.Ex4.m1.3.3.3.1.1.1.cmml">(</mo><mrow id="A1.Ex4.m1.3.3.3.1.1.1" xref="A1.Ex4.m1.3.3.3.1.1.1.cmml"><mrow id="A1.Ex4.m1.3.3.3.1.1.1.2" xref="A1.Ex4.m1.3.3.3.1.1.1.2.cmml"><mn id="A1.Ex4.m1.3.3.3.1.1.1.2.2" xref="A1.Ex4.m1.3.3.3.1.1.1.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.3.3.3.1.1.1.2.1" xref="A1.Ex4.m1.3.3.3.1.1.1.2.1.cmml">​</mo><mi id="A1.Ex4.m1.3.3.3.1.1.1.2.3" xref="A1.Ex4.m1.3.3.3.1.1.1.2.3.cmml">x</mi></mrow><mo id="A1.Ex4.m1.3.3.3.1.1.1.1" xref="A1.Ex4.m1.3.3.3.1.1.1.1.cmml">−</mo><mn id="A1.Ex4.m1.3.3.3.1.1.1.3" xref="A1.Ex4.m1.3.3.3.1.1.1.3.cmml">9</mn></mrow><mo stretchy="false" id="A1.Ex4.m1.3.3.3.1.1.3" xref="A1.Ex4.m1.3.3.3.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.4.4.4.3" xref="A1.Ex4.m1.4.4.4.3.cmml">​</mo><mrow id="A1.Ex4.m1.4.4.4.2.1" xref="A1.Ex4.m1.4.4.4.2.1.1.cmml"><mo stretchy="false" id="A1.Ex4.m1.4.4.4.2.1.2" xref="A1.Ex4.m1.4.4.4.2.1.1.cmml">(</mo><mrow id="A1.Ex4.m1.4.4.4.2.1.1" xref="A1.Ex4.m1.4.4.4.2.1.1.cmml"><mrow id="A1.Ex4.m1.4.4.4.2.1.1.2" xref="A1.Ex4.m1.4.4.4.2.1.1.2.cmml"><mn id="A1.Ex4.m1.4.4.4.2.1.1.2.2" xref="A1.Ex4.m1.4.4.4.2.1.1.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.4.4.4.2.1.1.2.1" xref="A1.Ex4.m1.4.4.4.2.1.1.2.1.cmml">​</mo><mi id="A1.Ex4.m1.4.4.4.2.1.1.2.3" xref="A1.Ex4.m1.4.4.4.2.1.1.2.3.cmml">x</mi></mrow><mo id="A1.Ex4.m1.4.4.4.2.1.1.1" xref="A1.Ex4.m1.4.4.4.2.1.1.1.cmml">+</mo><mn id="A1.Ex4.m1.4.4.4.2.1.1.3" xref="A1.Ex4.m1.4.4.4.2.1.1.3.cmml">6</mn></mrow><mo stretchy="false" id="A1.Ex4.m1.4.4.4.2.1.3" xref="A1.Ex4.m1.4.4.4.2.1.1.cmml">)</mo></mrow></mrow></mfrac><mo lspace="0.330em" rspace="0em" id="A1.Ex4.m1.5.6.3.1" xref="A1.Ex4.m1.5.6.3.1.cmml">​</mo><mi mathvariant="normal" id="A1.Ex4.m1.5.6.3.2" xref="A1.Ex4.m1.5.6.3.2.cmml">?</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex4.m1.5b"><apply id="A1.Ex4.m1.5.6.cmml" xref="A1.Ex4.m1.5.6"><eq id="A1.Ex4.m1.5.6.1.cmml" xref="A1.Ex4.m1.5.6.1"></eq><apply id="A1.Ex4.m1.5.6.2.cmml" xref="A1.Ex4.m1.5.6.2"><times id="A1.Ex4.m1.5.6.2.1.cmml" xref="A1.Ex4.m1.5.6.2.1"></times><ci id="A1.Ex4.m1.5.6.2.2.cmml" xref="A1.Ex4.m1.5.6.2.2">𝑓</ci><ci id="A1.Ex4.m1.5.5.cmml" xref="A1.Ex4.m1.5.5">𝑥</ci></apply><apply id="A1.Ex4.m1.5.6.3.cmml" xref="A1.Ex4.m1.5.6.3"><times id="A1.Ex4.m1.5.6.3.1.cmml" xref="A1.Ex4.m1.5.6.3.1"></times><apply id="A1.Ex4.m1.4.4.cmml" xref="A1.Ex4.m1.4.4"><divide id="A1.Ex4.m1.4.4.5.cmml" xref="A1.Ex4.m1.4.4"></divide><apply id="A1.Ex4.m1.2.2.2.cmml" xref="A1.Ex4.m1.2.2.2"><times id="A1.Ex4.m1.2.2.2.3.cmml" xref="A1.Ex4.m1.2.2.2.3"></times><apply id="A1.Ex4.m1.1.1.1.1.1.1.cmml" xref="A1.Ex4.m1.1.1.1.1.1"><minus id="A1.Ex4.m1.1.1.1.1.1.1.1.cmml" xref="A1.Ex4.m1.1.1.1.1.1.1.1"></minus><apply id="A1.Ex4.m1.1.1.1.1.1.1.2.cmml" xref="A1.Ex4.m1.1.1.1.1.1.1.2"><times id="A1.Ex4.m1.1.1.1.1.1.1.2.1.cmml" xref="A1.Ex4.m1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="A1.Ex4.m1.1.1.1.1.1.1.2.2.cmml" xref="A1.Ex4.m1.1.1.1.1.1.1.2.2">2</cn><ci id="A1.Ex4.m1.1.1.1.1.1.1.2.3.cmml" xref="A1.Ex4.m1.1.1.1.1.1.1.2.3">𝑥</ci></apply><cn type="integer" id="A1.Ex4.m1.1.1.1.1.1.1.3.cmml" xref="A1.Ex4.m1.1.1.1.1.1.1.3">3</cn></apply><apply id="A1.Ex4.m1.2.2.2.2.1.1.cmml" xref="A1.Ex4.m1.2.2.2.2.1"><plus id="A1.Ex4.m1.2.2.2.2.1.1.1.cmml" xref="A1.Ex4.m1.2.2.2.2.1.1.1"></plus><apply id="A1.Ex4.m1.2.2.2.2.1.1.2.cmml" xref="A1.Ex4.m1.2.2.2.2.1.1.2"><times id="A1.Ex4.m1.2.2.2.2.1.1.2.1.cmml" xref="A1.Ex4.m1.2.2.2.2.1.1.2.1"></times><cn type="integer" id="A1.Ex4.m1.2.2.2.2.1.1.2.2.cmml" xref="A1.Ex4.m1.2.2.2.2.1.1.2.2">2</cn><ci id="A1.Ex4.m1.2.2.2.2.1.1.2.3.cmml" xref="A1.Ex4.m1.2.2.2.2.1.1.2.3">𝑥</ci></apply><cn type="integer" id="A1.Ex4.m1.2.2.2.2.1.1.3.cmml" xref="A1.Ex4.m1.2.2.2.2.1.1.3">5</cn></apply></apply><apply id="A1.Ex4.m1.4.4.4.cmml" xref="A1.Ex4.m1.4.4.4"><times id="A1.Ex4.m1.4.4.4.3.cmml" xref="A1.Ex4.m1.4.4.4.3"></times><apply id="A1.Ex4.m1.3.3.3.1.1.1.cmml" xref="A1.Ex4.m1.3.3.3.1.1"><minus id="A1.Ex4.m1.3.3.3.1.1.1.1.cmml" xref="A1.Ex4.m1.3.3.3.1.1.1.1"></minus><apply id="A1.Ex4.m1.3.3.3.1.1.1.2.cmml" xref="A1.Ex4.m1.3.3.3.1.1.1.2"><times id="A1.Ex4.m1.3.3.3.1.1.1.2.1.cmml" xref="A1.Ex4.m1.3.3.3.1.1.1.2.1"></times><cn type="integer" id="A1.Ex4.m1.3.3.3.1.1.1.2.2.cmml" xref="A1.Ex4.m1.3.3.3.1.1.1.2.2">3</cn><ci id="A1.Ex4.m1.3.3.3.1.1.1.2.3.cmml" xref="A1.Ex4.m1.3.3.3.1.1.1.2.3">𝑥</ci></apply><cn type="integer" id="A1.Ex4.m1.3.3.3.1.1.1.3.cmml" xref="A1.Ex4.m1.3.3.3.1.1.1.3">9</cn></apply><apply id="A1.Ex4.m1.4.4.4.2.1.1.cmml" xref="A1.Ex4.m1.4.4.4.2.1"><plus id="A1.Ex4.m1.4.4.4.2.1.1.1.cmml" xref="A1.Ex4.m1.4.4.4.2.1.1.1"></plus><apply id="A1.Ex4.m1.4.4.4.2.1.1.2.cmml" xref="A1.Ex4.m1.4.4.4.2.1.1.2"><times id="A1.Ex4.m1.4.4.4.2.1.1.2.1.cmml" xref="A1.Ex4.m1.4.4.4.2.1.1.2.1"></times><cn type="integer" id="A1.Ex4.m1.4.4.4.2.1.1.2.2.cmml" xref="A1.Ex4.m1.4.4.4.2.1.1.2.2">3</cn><ci id="A1.Ex4.m1.4.4.4.2.1.1.2.3.cmml" xref="A1.Ex4.m1.4.4.4.2.1.1.2.3">𝑥</ci></apply><cn type="integer" id="A1.Ex4.m1.4.4.4.2.1.1.3.cmml" xref="A1.Ex4.m1.4.4.4.2.1.1.3">6</cn></apply></apply></apply><ci id="A1.Ex4.m1.5.6.3.2.cmml" xref="A1.Ex4.m1.5.6.3.2">?</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex4.m1.5c">f(x)=\frac{(2x-3)(2x+5)}{(3x-9)(3x+6)}~{}?</annotation></semantics></math></span>
<span class="ltx_eqn_cell ltx_eqn_center_padright"></span></span></span>
</span>
<span id="A1.SS3.p3.pic1.2.2.2.1.1.3" class="ltx_p">Express your answer as an interval or as a union of intervals.</span>
<span id="A1.SS3.p3.pic1.2.2.2.1.1.4" class="ltx_p">Solution:</span>
</span></foreignObject></g></g></svg>
</div>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.4 </span>GSM8K</h3>

<div id="A1.SS4.p1" class="ltx_para">
<p id="A1.SS4.p1.5" class="ltx_p">We report results on <math id="A1.SS4.p1.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="A1.SS4.p1.1.m1.1a"><mn id="A1.SS4.p1.1.m1.1.1" xref="A1.SS4.p1.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.1.m1.1b"><cn type="integer" id="A1.SS4.p1.1.m1.1.1.cmml" xref="A1.SS4.p1.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.1.m1.1c">128</annotation></semantics></math> randomly sampled test-set problems. We sample with a temperature of <math id="A1.SS4.p1.2.m2.1" class="ltx_Math" alttext="0.6" display="inline"><semantics id="A1.SS4.p1.2.m2.1a"><mn id="A1.SS4.p1.2.m2.1.1" xref="A1.SS4.p1.2.m2.1.1.cmml">0.6</mn><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.2.m2.1b"><cn type="float" id="A1.SS4.p1.2.m2.1.1.cmml" xref="A1.SS4.p1.2.m2.1.1">0.6</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.2.m2.1c">0.6</annotation></semantics></math> and do not use nucleus sampling. We use <math id="A1.SS4.p1.3.m3.1" class="ltx_Math" alttext="5" display="inline"><semantics id="A1.SS4.p1.3.m3.1a"><mn id="A1.SS4.p1.3.m3.1.1" xref="A1.SS4.p1.3.m3.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.3.m3.1b"><cn type="integer" id="A1.SS4.p1.3.m3.1.1.cmml" xref="A1.SS4.p1.3.m3.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.3.m3.1c">5</annotation></semantics></math> few-shot examples from the training set that are randomly sampled per-problem. We generate <math id="A1.SS4.p1.4.m4.2" class="ltx_Math" alttext="10,000" display="inline"><semantics id="A1.SS4.p1.4.m4.2a"><mrow id="A1.SS4.p1.4.m4.2.3.2" xref="A1.SS4.p1.4.m4.2.3.1.cmml"><mn id="A1.SS4.p1.4.m4.1.1" xref="A1.SS4.p1.4.m4.1.1.cmml">10</mn><mo id="A1.SS4.p1.4.m4.2.3.2.1" xref="A1.SS4.p1.4.m4.2.3.1.cmml">,</mo><mn id="A1.SS4.p1.4.m4.2.2" xref="A1.SS4.p1.4.m4.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.4.m4.2b"><list id="A1.SS4.p1.4.m4.2.3.1.cmml" xref="A1.SS4.p1.4.m4.2.3.2"><cn type="integer" id="A1.SS4.p1.4.m4.1.1.cmml" xref="A1.SS4.p1.4.m4.1.1">10</cn><cn type="integer" id="A1.SS4.p1.4.m4.2.2.cmml" xref="A1.SS4.p1.4.m4.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.4.m4.2c">10,000</annotation></semantics></math> samples per problem. We set <math id="A1.SS4.p1.5.m5.1" class="ltx_Math" alttext="512" display="inline"><semantics id="A1.SS4.p1.5.m5.1a"><mn id="A1.SS4.p1.5.m5.1.1" xref="A1.SS4.p1.5.m5.1.1.cmml">512</mn><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.5.m5.1b"><cn type="integer" id="A1.SS4.p1.5.m5.1.1.cmml" xref="A1.SS4.p1.5.m5.1.1">512</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.5.m5.1c">512</annotation></semantics></math> as the max token length for the generated solution. To grade solutions, we follow LMEval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>.</p>
</div>
<div id="A1.SS4.p2" class="ltx_para ltx_noindent">
<svg id="A1.SS4.p2.pic1" class="ltx_picture" height="161.63" overflow="visible" version="1.1" width="600"><g transform="translate(0,161.63) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#9F9F9F" fill-opacity="1.0"><path d="M 0 5.91 L 0 155.72 C 0 158.98 2.64 161.63 5.91 161.63 L 594.09 161.63 C 597.36 161.63 600 158.98 600 155.72 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F9F9F9" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 137.52 L 598.03 137.52 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 143.42)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="A1.SS4.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS4.p2.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.SS4.p2.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Few-Shot Example</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="111.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.SS4.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS4.p2.pic1.2.2.2.1.1.1" class="ltx_p">Question: James decides to replace his car. He sold his $20,000 car for 80% of its value and then was able to haggle to buy a $30,000 sticker price car for 90% of its value. How much was he out of pocket?</span>
<span id="A1.SS4.p2.pic1.2.2.2.1.1.2" class="ltx_p">Answer: He sold his car for 20000*.8=$<code id="A1.SS4.p2.pic1.2.2.2.1.1.2.1" class="ltx_verbatim ltx_font_typewriter">&lt;&lt;</code>20000*.8=16000<code id="A1.SS4.p2.pic1.2.2.2.1.1.2.2" class="ltx_verbatim ltx_font_typewriter">&gt;&gt;</code>16,000
He bought the new car for 30,000*.9=$<code id="A1.SS4.p2.pic1.2.2.2.1.1.2.3" class="ltx_verbatim ltx_font_typewriter">&lt;&lt;</code>30000*.9=27000<code id="A1.SS4.p2.pic1.2.2.2.1.1.2.4" class="ltx_verbatim ltx_font_typewriter">&gt;&gt;</code>27,000
That means he was out of pocket 27,000-16,000=$<code id="A1.SS4.p2.pic1.2.2.2.1.1.2.5" class="ltx_verbatim ltx_font_typewriter">&lt;&lt;</code>27000-16000=11000<code id="A1.SS4.p2.pic1.2.2.2.1.1.2.6" class="ltx_verbatim ltx_font_typewriter">&gt;&gt;</code>11,000</span>
<span id="A1.SS4.p2.pic1.2.2.2.1.1.3" class="ltx_p">#### 11000</span>
</span></foreignObject></g></g></svg>
</div>
<div id="A1.SS4.p3" class="ltx_para ltx_noindent">
<svg id="A1.SS4.p3.pic1" class="ltx_picture" height="108.97" overflow="visible" version="1.1" width="600"><g transform="translate(0,108.97) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#9F9F9F" fill-opacity="1.0"><path d="M 0 5.91 L 0 103.06 C 0 106.33 2.64 108.97 5.91 108.97 L 594.09 108.97 C 597.36 108.97 600 106.33 600 103.06 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F9F9F9" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 84.86 L 598.03 84.86 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 90.76)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="A1.SS4.p3.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS4.p3.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.SS4.p3.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Example Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="59.27" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.SS4.p3.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.SS4.p3.pic1.2.2.2.1.1.1" class="ltx_p">Question: Mary has 6 jars of sprinkles in her pantry. Each jar of sprinkles can decorate 8 cupcakes. Mary wants to bake enough cupcakes to use up all of her sprinkles. If each pan holds 12 cupcakes, how many pans worth of cupcakes should she bake?</span>
<span id="A1.SS4.p3.pic1.2.2.2.1.1.2" class="ltx_p">Answer:</span>
</span></foreignObject></g></g></svg>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>SWE-bench Lite</h2>

<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>Experimental Setup</h3>

<div id="A2.SS1.p1" class="ltx_para">
<p id="A2.SS1.p1.1" class="ltx_p">For our experiments, we use DeepSeek-V2-Coder-Instruct with the Moatless Tools agent framework (at commit <span id="A2.SS1.p1.1.1" class="ltx_text ltx_font_typewriter">a1017b78e3e69e7d205b1a3faa83a7d19fce3fa6</span>).
We use Voyage AI <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite> embeddings for retrieval, the default used by Moatless Tools. We make no modifications to the model or framework, using them entirely as off-the-shelf components.</p>
</div>
<div id="A2.SS1.p2" class="ltx_para">
<p id="A2.SS1.p2.1" class="ltx_p">With this setup, we sample 250 independent completions for each problem using standard temperature-based sampling. To determine the optimal sampling temperature, we conducted a sweep on a random subset of 50 problems from the test set, testing temperatures of 1.0, 1.4, 1.6, and 1.8. Based on these results, we selected a temperature of 1.6 for our main experiments.</p>
</div>
</section>
<section id="A2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.2 </span>Test Suite Flakiness</h3>

<div id="A2.SS2.p1" class="ltx_para">
<p id="A2.SS2.p1.1" class="ltx_p">During our analysis, we identified 34 problems in SWE-bench Lite whose test suites had flaky tests. Using the SWE-bench testing harness provided by the authors of SWE-bench, we tested each solution repeatedly: for some solutions, sometimes the solution was marked as correct, and other times it was marked as incorrect. In 30 of these 34 cases, we observed flakiness even on the correct solutions provided by the dataset authors. Table <a href="#A2.T3" title="Table 3 ‣ B.2 Test Suite Flakiness ‣ Appendix B SWE-bench Lite ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> lists the problem IDs of the 34 instances with flaky tests.</p>
</div>
<figure id="A2.T3" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A2.T3.2.1.1" class="ltx_text" style="font-size:90%;">Table 3</span>: </span><span id="A2.T3.3.2" class="ltx_text" style="font-size:90%;">Instance IDs of problems from SWE-bench Lite that have flaky tests.</span></figcaption>
<table id="A2.T3.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A2.T3.4.1.1" class="ltx_tr">
<th id="A2.T3.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_l ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T3.4.1.1.1.1" class="ltx_text ltx_font_bold">Repository</span></th>
<th id="A2.T3.4.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A2.T3.4.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T3.4.1.1.2.1.1" class="ltx_p" style="width:325.2pt;"><span id="A2.T3.4.1.1.2.1.1.1" class="ltx_text ltx_font_bold">Instance IDs</span></span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A2.T3.4.2.1" class="ltx_tr">
<th id="A2.T3.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T3.4.2.1.1.1" class="ltx_text ltx_font_typewriter">django</span></th>
<td id="A2.T3.4.2.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A2.T3.4.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T3.4.2.1.2.1.1" class="ltx_p" style="width:325.2pt;"><span id="A2.T3.4.2.1.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">

<span id="A2.T3.4.2.1.2.1.1.1.1" class="ltx_tabular ltx_align_top">
<span id="A2.T3.4.2.1.2.1.1.1.1.1" class="ltx_tr">
<span id="A2.T3.4.2.1.2.1.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">django__django-13315, django__django-13447, django__django-13590,</span></span>
<span id="A2.T3.4.2.1.2.1.1.1.1.2" class="ltx_tr">
<span id="A2.T3.4.2.1.2.1.1.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">django__django-13710, django__django-13757, django__django-13933,</span></span>
<span id="A2.T3.4.2.1.2.1.1.1.1.3" class="ltx_tr">
<span id="A2.T3.4.2.1.2.1.1.1.1.3.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">django__django-13964, django__django-14017, django__django-14238,</span></span>
<span id="A2.T3.4.2.1.2.1.1.1.1.4" class="ltx_tr">
<span id="A2.T3.4.2.1.2.1.1.1.1.4.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">django__django-14382, django__django-14608, django__django-14672,</span></span>
<span id="A2.T3.4.2.1.2.1.1.1.1.5" class="ltx_tr">
<span id="A2.T3.4.2.1.2.1.1.1.1.5.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">django__django-14752, django__django-14915, django__django-14997,</span></span>
<span id="A2.T3.4.2.1.2.1.1.1.1.6" class="ltx_tr">
<span id="A2.T3.4.2.1.2.1.1.1.1.6.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">django__django-14999, django__django-15320, django__django-15738,</span></span>
<span id="A2.T3.4.2.1.2.1.1.1.1.7" class="ltx_tr">
<span id="A2.T3.4.2.1.2.1.1.1.1.7.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">django__django-15790, django__django-15814, django__django-15819,</span></span>
<span id="A2.T3.4.2.1.2.1.1.1.1.8" class="ltx_tr">
<span id="A2.T3.4.2.1.2.1.1.1.1.8.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">django__django-16229, django__django-16379, django__django-16400,</span></span>
<span id="A2.T3.4.2.1.2.1.1.1.1.9" class="ltx_tr">
<span id="A2.T3.4.2.1.2.1.1.1.1.9.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">django__django-17051</span></span>
</span></span></span>
</span>
</td>
</tr>
<tr id="A2.T3.4.3.2" class="ltx_tr">
<th id="A2.T3.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T3.4.3.2.1.1" class="ltx_text ltx_font_typewriter">sympy</span></th>
<td id="A2.T3.4.3.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A2.T3.4.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T3.4.3.2.2.1.1" class="ltx_p" style="width:325.2pt;"><span id="A2.T3.4.3.2.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">

<span id="A2.T3.4.3.2.2.1.1.1.1" class="ltx_tabular ltx_align_top">
<span id="A2.T3.4.3.2.2.1.1.1.1.1" class="ltx_tr">
<span id="A2.T3.4.3.2.2.1.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">sympy__sympy-13146, sympy__sympy-13177, sympy__sympy-16988</span></span>
</span></span></span>
</span>
</td>
</tr>
<tr id="A2.T3.4.4.3" class="ltx_tr">
<th id="A2.T3.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T3.4.4.3.1.1" class="ltx_text ltx_font_typewriter">requests</span></th>
<td id="A2.T3.4.4.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A2.T3.4.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T3.4.4.3.2.1.1" class="ltx_p" style="width:325.2pt;"><span id="A2.T3.4.4.3.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">

<span id="A2.T3.4.4.3.2.1.1.1.1" class="ltx_tabular ltx_align_top">
<span id="A2.T3.4.4.3.2.1.1.1.1.1" class="ltx_tr">
<span id="A2.T3.4.4.3.2.1.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">psf__requests-863, psf__requests-2317,</span></span>
<span id="A2.T3.4.4.3.2.1.1.1.1.2" class="ltx_tr">
<span id="A2.T3.4.4.3.2.1.1.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">psf__requests-2674, psf__requests-3362</span></span>
</span></span></span>
</span>
</td>
</tr>
<tr id="A2.T3.4.5.4" class="ltx_tr">
<th id="A2.T3.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T3.4.5.4.1.1" class="ltx_text ltx_font_typewriter">scikit-learn</span></th>
<td id="A2.T3.4.5.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A2.T3.4.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T3.4.5.4.2.1.1" class="ltx_p" style="width:325.2pt;"><span id="A2.T3.4.5.4.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">scikit-learn__scikit-learn-13241</span></span>
</span>
</td>
</tr>
<tr id="A2.T3.4.6.5" class="ltx_tr">
<th id="A2.T3.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_b ltx_border_l ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T3.4.6.5.1.1" class="ltx_text ltx_font_typewriter">matplotlib</span></th>
<td id="A2.T3.4.6.5.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_b ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A2.T3.4.6.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T3.4.6.5.2.1.1" class="ltx_p" style="width:325.2pt;"><span id="A2.T3.4.6.5.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">matplotlib__matplotlib-23987</span></span>
</span>
</td>
</tr>
</tbody>
</table>
</figure>
<div id="A2.SS2.p2" class="ltx_para ltx_noindent">
<p id="A2.SS2.p2.1" class="ltx_p">An additional instance, <span id="A2.SS2.p2.1.1" class="ltx_text ltx_font_typewriter">astropy__astropy-6938</span>, was flaky on some machines and not others. The authors of SWE-bench were able to reproduce the flakyness; however, we were unable to. Our prelimiary investigation indicates this specific issue is due to unpinned versions of dependencies in the docker environments that run the unit tests.</p>
</div>
<div id="A2.SS2.p3" class="ltx_para">
<p id="A2.SS2.p3.1" class="ltx_p">Here, we include results on a subset with the problems in Table <a href="#A2.T3" title="Table 3 ‣ B.2 Test Suite Flakiness ‣ Appendix B SWE-bench Lite ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> removed (266 problems). For the full dataset evaluation, on any problem that has flaky tests, we run the test suite 11 times and use majority voting to determine whether a solution passed or failed. For the evaluation on the subset without flaky tests, all baselines we compare against release which problems they correctly solve, so we simply removed the problems with flaky tests and recomputed their scores.</p>
</div>
<figure id="A2.F9" class="ltx_figure"><img src="/html/2407.21787/assets/figures/with_flaky.png" id="A2.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="292" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F9.2.1.1" class="ltx_text" style="font-size:90%;">Figure 9</span>: </span><span id="A2.F9.3.2" class="ltx_text" style="font-size:90%;">SWE-bench Lite results, without and with problems that have flaky tests. For the graph on the left, all problems in Table <a href="#A2.T3" title="Table 3 ‣ B.2 Test Suite Flakiness ‣ Appendix B SWE-bench Lite ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> are excluded. For the graph on the right, all problems are included. We note that the trend is the same with or without the flaky tests.</span></figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Scaling Law Details</h2>

<section id="A3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.1 </span>Experimental details</h3>

<div id="A3.SS1.p1" class="ltx_para">
<p id="A3.SS1.p1.5" class="ltx_p">To fit exponentiated power laws to coverage curves, we first sample <math id="A3.SS1.p1.1.m1.1" class="ltx_Math" alttext="40" display="inline"><semantics id="A3.SS1.p1.1.m1.1a"><mn id="A3.SS1.p1.1.m1.1.1" xref="A3.SS1.p1.1.m1.1.1.cmml">40</mn><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.1.m1.1b"><cn type="integer" id="A3.SS1.p1.1.m1.1.1.cmml" xref="A3.SS1.p1.1.m1.1.1">40</cn></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p1.1.m1.1c">40</annotation></semantics></math> points spaced evenly along a log scale from <math id="A3.SS1.p1.2.m2.1" class="ltx_Math" alttext="0" display="inline"><semantics id="A3.SS1.p1.2.m2.1a"><mn id="A3.SS1.p1.2.m2.1.1" xref="A3.SS1.p1.2.m2.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.2.m2.1b"><cn type="integer" id="A3.SS1.p1.2.m2.1.1.cmml" xref="A3.SS1.p1.2.m2.1.1">0</cn></annotation-xml></semantics></math> to <math id="A3.SS1.p1.3.m3.2" class="ltx_Math" alttext="10,000" display="inline"><semantics id="A3.SS1.p1.3.m3.2a"><mrow id="A3.SS1.p1.3.m3.2.3.2" xref="A3.SS1.p1.3.m3.2.3.1.cmml"><mn id="A3.SS1.p1.3.m3.1.1" xref="A3.SS1.p1.3.m3.1.1.cmml">10</mn><mo id="A3.SS1.p1.3.m3.2.3.2.1" xref="A3.SS1.p1.3.m3.2.3.1.cmml">,</mo><mn id="A3.SS1.p1.3.m3.2.2" xref="A3.SS1.p1.3.m3.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.3.m3.2b"><list id="A3.SS1.p1.3.m3.2.3.1.cmml" xref="A3.SS1.p1.3.m3.2.3.2"><cn type="integer" id="A3.SS1.p1.3.m3.1.1.cmml" xref="A3.SS1.p1.3.m3.1.1">10</cn><cn type="integer" id="A3.SS1.p1.3.m3.2.2.cmml" xref="A3.SS1.p1.3.m3.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p1.3.m3.2c">10,000</annotation></semantics></math> and remove duplicates. We then use SciPy’s <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite> <code id="A3.SS1.p1.5.1" class="ltx_verbatim ltx_font_typewriter">curve_fit</code> function to find the <math id="A3.SS1.p1.4.m4.1" class="ltx_Math" alttext="a" display="inline"><semantics id="A3.SS1.p1.4.m4.1a"><mi id="A3.SS1.p1.4.m4.1.1" xref="A3.SS1.p1.4.m4.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.4.m4.1b"><ci id="A3.SS1.p1.4.m4.1.1.cmml" xref="A3.SS1.p1.4.m4.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p1.4.m4.1c">a</annotation></semantics></math> and <math id="A3.SS1.p1.5.m5.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A3.SS1.p1.5.m5.1a"><mi id="A3.SS1.p1.5.m5.1.1" xref="A3.SS1.p1.5.m5.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.5.m5.1b"><ci id="A3.SS1.p1.5.m5.1.1.cmml" xref="A3.SS1.p1.5.m5.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p1.5.m5.1c">b</annotation></semantics></math> parameters from Equation <a href="#S3.E3" title="In 3.1 Scaling Laws for Repeated Sampling ‣ 3 Characterizing the Benefits of Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> that best fit these points.</p>
</div>
</section>
<section id="A3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.2 </span>Additional results</h3>

<div id="A3.SS2.p1" class="ltx_para">
<p id="A3.SS2.p1.1" class="ltx_p">In Figure <a href="#A3.F10" title="Figure 10 ‣ C.2 Additional results ‣ Appendix C Scaling Law Details ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>, we show additional results fitting power laws to coverage curves for an expanded set of datasets and models.</p>
</div>
<figure id="A3.F10" class="ltx_figure"><img src="/html/2407.21787/assets/x8.png" id="A3.F10.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="415" height="622" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A3.F10.2.1.1" class="ltx_text" style="font-size:90%;">Figure 10</span>: </span><span id="A3.F10.3.2" class="ltx_text" style="font-size:90%;">Fitting exponentiated power laws to coverage curves for an expanded set of tasks and models.
</span></figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Precision Details</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.3" class="ltx_p">To calculate the Majority Vote, Reward Model + Best-of-N and Reward Model + Majority Vote metrics, we use the same <math id="A4.p1.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="A4.p1.1.m1.1a"><mn id="A4.p1.1.m1.1.1" xref="A4.p1.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="A4.p1.1.m1.1b"><cn type="integer" id="A4.p1.1.m1.1.1.cmml" xref="A4.p1.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.1.m1.1c">128</annotation></semantics></math> problem subsets for both MATH and GSM8K datasets introduced in Section <a href="#S2" title="2 Scaling Repeated Sampling ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
Each problem corresponds to 10,000 samples for each model we test.
For each verification method, we take <math id="A4.p1.2.m2.1" class="ltx_Math" alttext="100" display="inline"><semantics id="A4.p1.2.m2.1a"><mn id="A4.p1.2.m2.1.1" xref="A4.p1.2.m2.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="A4.p1.2.m2.1b"><cn type="integer" id="A4.p1.2.m2.1.1.cmml" xref="A4.p1.2.m2.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.2.m2.1c">100</annotation></semantics></math> random subsets of size <math id="A4.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A4.p1.3.m3.1a"><mi id="A4.p1.3.m3.1.1" xref="A4.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A4.p1.3.m3.1b"><ci id="A4.p1.3.m3.1.1.cmml" xref="A4.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.3.m3.1c">k</annotation></semantics></math> and calculate the success rate using each subset.
We report the mean and standard deviation across subsets in Figure <a href="#S4.F7" title="Figure 7 ‣ 4.1 Common Verification Methods Don’t Always Scale with the Sample Budget ‣ 4 Harnessing Repeated Sampling Requires Precision ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.
To calculate the Majority Vote answer, we take the plurality answer in each subset.
For the Reward Model + Best-of-N, we take the answer with the highest score assigned by the reward model.
For the Reward Model + Majority Vote metric, we sum the reward model score across all the samples with the same final answer, and use the final answer with the highest sum.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>GSM8K incorrect answer</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.1" class="ltx_p">As discussed in <a href="#S4.SS1" title="4.1 Common Verification Methods Don’t Always Scale with the Sample Budget ‣ 4 Harnessing Repeated Sampling Requires Precision ‣ Large Language MonkeysTitle inspired by https://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Compute with Repeated Sampling" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>, we identify that <a target="_blank" href="https://huggingface.co/datasets/openai/gsm8k/viewer/main/test?row=1042" title="" class="ltx_ref ltx_href">a problem in the GSM8K test set (index 1042 on HuggingFace)</a> has an incorrect ground truth solution.</p>
</div>
<div id="A5.p2" class="ltx_para ltx_noindent">
<svg id="A5.p2.pic1" class="ltx_picture" height="111.66" overflow="visible" version="1.1" width="600"><g transform="translate(0,111.66) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#9F9F9F" fill-opacity="1.0"><path d="M 0 5.91 L 0 105.75 C 0 109.02 2.64 111.66 5.91 111.66 L 594.09 111.66 C 597.36 111.66 600 109.02 600 105.75 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F9F9F9" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 87.7 L 598.03 87.7 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 93.61)"><foreignObject width="556.69" height="12.15" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="A5.p2.pic1.4.4.4.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A5.p2.pic1.4.4.4.1.1.1" class="ltx_p"><span id="A5.p2.pic1.4.4.4.1.1.1.1" class="ltx_text ltx_font_bold">Question</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="62.11" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A5.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A5.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3" class="ltx_p">Johnny’s dad brought him to watch some horse racing and his dad bet money. On the first race, he lost $<math id="A5.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="5" display="inline"><semantics id="A5.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="A5.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="A5.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="A5.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="A5.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="A5.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p2.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1c">5</annotation></semantics></math>. On the second race, he won $<math id="A5.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1" class="ltx_Math" alttext="1" display="inline"><semantics id="A5.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1a"><mn id="A5.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1" xref="A5.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="A5.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1b"><cn type="integer" id="A5.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1.cmml" xref="A5.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p2.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1c">1</annotation></semantics></math> more than twice the amount he previously lost. On the third race, he lost <math id="A5.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1" class="ltx_Math" alttext="1.5" display="inline"><semantics id="A5.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1a"><mn id="A5.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1.1" xref="A5.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1.1.cmml">1.5</mn><annotation-xml encoding="MathML-Content" id="A5.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1b"><cn type="float" id="A5.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1.1.cmml" xref="A5.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1.1">1.5</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p2.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m3.1c">1.5</annotation></semantics></math> times as much as he won in the second race. How much did he lose on average that day?</span>
</span></foreignObject></g></g></svg>
</div>
<div id="A5.p3" class="ltx_para ltx_noindent">
<svg id="A5.p3.pic1" class="ltx_picture" height="142.18" overflow="visible" version="1.1" width="600"><g transform="translate(0,142.18) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#FF4040" fill-opacity="1.0"><path d="M 0 5.91 L 0 136.27 C 0 139.53 2.64 142.18 5.91 142.18 L 594.09 142.18 C 597.36 142.18 600 139.53 600 136.27 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F9F9F9" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 120.91 L 598.03 120.91 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 126.82)"><foreignObject width="556.69" height="9.46" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="A5.p3.pic1.11.11.11.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A5.p3.pic1.11.11.11.1.1.1" class="ltx_p"><span id="A5.p3.pic1.11.11.11.1.1.1.1" class="ltx_text ltx_font_bold">Answer</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="95.32" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2" class="ltx_p">On the second race he won $<math id="A5.p3.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="11" display="inline"><semantics id="A5.p3.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="A5.p3.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="A5.p3.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">11</mn><annotation-xml encoding="MathML-Content" id="A5.p3.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="A5.p3.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="A5.p3.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">11</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p3.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1c">11</annotation></semantics></math> because <math id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1" class="ltx_math_unparsed" alttext="1+5\times 2=&lt;&lt;1+5*2=11&gt;&gt;11" display="inline"><semantics id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1a"><mrow id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1b"><mn id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.1">1</mn><mo id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.2">+</mo><mn id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.3">5</mn><mo lspace="0.222em" rspace="0.222em" id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.4">×</mo><mn id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.5">2</mn><mo rspace="0em" id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.6">=</mo><mo lspace="0em" rspace="0em" id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.7">&lt;</mo><mo lspace="0em" id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.8">&lt;</mo><mn id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.9">1</mn><mo id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.10">+</mo><mn id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.11">5</mn><mo lspace="0.222em" rspace="0.222em" id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.12">∗</mo><mn id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.13">2</mn><mo id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.14">=</mo><mn id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.15">11</mn><mo rspace="0em" id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.16">&gt;</mo><mo lspace="0em" id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.17">&gt;</mo><mn id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1.18">11</mn></mrow><annotation encoding="application/x-tex" id="A5.p3.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.m2.1c">1+5\times 2=&lt;&lt;1+5*2=11&gt;&gt;11</annotation></semantics></math></span>
<span id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4" class="ltx_p">On the third race he lost $<math id="A5.p3.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m1.1" class="ltx_Math" alttext="15" display="inline"><semantics id="A5.p3.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m1.1a"><mn id="A5.p3.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m1.1.1" xref="A5.p3.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m1.1.1.cmml">15</mn><annotation-xml encoding="MathML-Content" id="A5.p3.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m1.1b"><cn type="integer" id="A5.p3.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m1.1.1.cmml" xref="A5.p3.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m1.1.1">15</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p3.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.m1.1c">15</annotation></semantics></math> because <math id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1" class="ltx_math_unparsed" alttext="10\times 1.5=&lt;&lt;10*1.5=15&gt;&gt;15" display="inline"><semantics id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1a"><mrow id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1b"><mn id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.1">10</mn><mo lspace="0.222em" rspace="0.222em" id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.2">×</mo><mn id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.3">1.5</mn><mo rspace="0em" id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.4">=</mo><mo lspace="0em" rspace="0em" id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.5">&lt;</mo><mo lspace="0em" id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.6">&lt;</mo><mn id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.7">10</mn><mo lspace="0.222em" rspace="0.222em" id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.8">∗</mo><mn id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.9">1.5</mn><mo id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.10">=</mo><mn id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.11">15</mn><mo rspace="0em" id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.12">&gt;</mo><mo lspace="0em" id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.13">&gt;</mo><mn id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1.14">15</mn></mrow><annotation encoding="application/x-tex" id="A5.p3.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.m2.1c">10\times 1.5=&lt;&lt;10*1.5=15&gt;&gt;15</annotation></semantics></math></span>
<span id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6" class="ltx_p">He lost a total of $<math id="A5.p3.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="A5.p3.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m1.1a"><mn id="A5.p3.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m1.1.1" xref="A5.p3.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="A5.p3.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m1.1b"><cn type="integer" id="A5.p3.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m1.1.1.cmml" xref="A5.p3.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p3.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.m1.1c">20</annotation></semantics></math> on the first and third races because <math id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1" class="ltx_math_unparsed" alttext="15+5=&lt;&lt;15+5=20&gt;&gt;20" display="inline"><semantics id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1a"><mrow id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1b"><mn id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.1">15</mn><mo id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.2">+</mo><mn id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.3">5</mn><mo rspace="0em" id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.4">=</mo><mo lspace="0em" rspace="0em" id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.5">&lt;</mo><mo lspace="0em" id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.6">&lt;</mo><mn id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.7">15</mn><mo id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.8">+</mo><mn id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.9">5</mn><mo id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.10">=</mo><mn id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.11">20</mn><mo rspace="0em" id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.12">&gt;</mo><mo lspace="0em" id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.13">&gt;</mo><mn id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1.14">20</mn></mrow><annotation encoding="application/x-tex" id="A5.p3.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.m2.1c">15+5=&lt;&lt;15+5=20&gt;&gt;20</annotation></semantics></math></span>
<span id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8" class="ltx_p">He lost <math id="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1" class="ltx_Math" alttext="\$9" display="inline"><semantics id="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1a"><mrow id="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1" xref="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1.cmml"><mo rspace="0.167em" id="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1.1" xref="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1.1.cmml">$</mo><mn id="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1.2" xref="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1.2.cmml">9</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1b"><apply id="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1.cmml" xref="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1"><csymbol cd="latexml" id="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1.1.cmml" xref="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1.1">currency-dollar</csymbol><cn type="integer" id="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1.2.cmml" xref="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1.1.2">9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p3.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.m1.1c">\$9</annotation></semantics></math> that day because <math id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1" class="ltx_math_unparsed" alttext="11-20=&lt;&lt;11-20=-9&gt;&gt;-9" display="inline"><semantics id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1a"><mrow id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1b"><mn id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.1">11</mn><mo id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.2">−</mo><mn id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.3">20</mn><mo rspace="0em" id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.4">=</mo><mo lspace="0em" rspace="0em" id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.5">&lt;</mo><mo lspace="0em" id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.6">&lt;</mo><mn id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.7">11</mn><mo id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.8">−</mo><mn id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.9">20</mn><mo rspace="0em" id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.10">=</mo><mo lspace="0em" id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.11">−</mo><mn id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.12">9</mn><mo rspace="0em" id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.13">&gt;</mo><mo lspace="0em" rspace="0em" id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.14">&gt;</mo><mo lspace="0em" id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.15">−</mo><mn id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1.16">9</mn></mrow><annotation encoding="application/x-tex" id="A5.p3.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.m2.1c">11-20=&lt;&lt;11-20=-9&gt;&gt;-9</annotation></semantics></math></span>
<span id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10" class="ltx_p">He lost an average of $<math id="A5.p3.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1" class="ltx_Math" alttext="3" display="inline"><semantics id="A5.p3.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1a"><mn id="A5.p3.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1" xref="A5.p3.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="A5.p3.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1b"><cn type="integer" id="A5.p3.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1.cmml" xref="A5.p3.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p3.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.m1.1c">3</annotation></semantics></math> per race because <math id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1" class="ltx_math_unparsed" alttext="9/3=&lt;&lt;9/3=3&gt;&gt;3" display="inline"><semantics id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1a"><mrow id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1b"><mn id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.1">9</mn><mo id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.2">/</mo><mn id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.3">3</mn><mo rspace="0em" id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.4">=</mo><mo lspace="0em" rspace="0em" id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.5">&lt;</mo><mo lspace="0em" id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.6">&lt;</mo><mn id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.7">9</mn><mo id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.8">/</mo><mn id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.9">3</mn><mo id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.10">=</mo><mn id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.11">3</mn><mo rspace="0em" id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.12">&gt;</mo><mo lspace="0em" id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.13">&gt;</mo><mn id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1.14">3</mn></mrow><annotation encoding="application/x-tex" id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.m2.1c">9/3=&lt;&lt;9/3=3&gt;&gt;3</annotation></semantics></math></span>
<span id="A5.p3.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.11" class="ltx_p">#### 3</span>
</span></foreignObject></g></g></svg>
</div>
<div id="A5.p4" class="ltx_para ltx_noindent">
<p id="A5.p4.6" class="ltx_p">The mistake is in the second line of the answer: on the third race, Johnny’s dad lost $<math id="A5.p4.1.m1.1" class="ltx_Math" alttext="16.5" display="inline"><semantics id="A5.p4.1.m1.1a"><mn id="A5.p4.1.m1.1.1" xref="A5.p4.1.m1.1.1.cmml">16.5</mn><annotation-xml encoding="MathML-Content" id="A5.p4.1.m1.1b"><cn type="float" id="A5.p4.1.m1.1.1.cmml" xref="A5.p4.1.m1.1.1">16.5</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.1.m1.1c">16.5</annotation></semantics></math>, not $<math id="A5.p4.2.m2.1" class="ltx_Math" alttext="15" display="inline"><semantics id="A5.p4.2.m2.1a"><mn id="A5.p4.2.m2.1.1" xref="A5.p4.2.m2.1.1.cmml">15</mn><annotation-xml encoding="MathML-Content" id="A5.p4.2.m2.1b"><cn type="integer" id="A5.p4.2.m2.1.1.cmml" xref="A5.p4.2.m2.1.1">15</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.2.m2.1c">15</annotation></semantics></math>, meaning he made $<math id="A5.p4.3.m3.1" class="ltx_Math" alttext="11" display="inline"><semantics id="A5.p4.3.m3.1a"><mn id="A5.p4.3.m3.1.1" xref="A5.p4.3.m3.1.1.cmml">11</mn><annotation-xml encoding="MathML-Content" id="A5.p4.3.m3.1b"><cn type="integer" id="A5.p4.3.m3.1.1.cmml" xref="A5.p4.3.m3.1.1">11</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.3.m3.1c">11</annotation></semantics></math> and lost <math id="A5.p4.4.m4.1" class="ltx_Math" alttext="\$16.5+\$5=\$21.5" display="inline"><semantics id="A5.p4.4.m4.1a"><mrow id="A5.p4.4.m4.1.1" xref="A5.p4.4.m4.1.1.cmml"><mrow id="A5.p4.4.m4.1.1.2" xref="A5.p4.4.m4.1.1.2.cmml"><mrow id="A5.p4.4.m4.1.1.2.2" xref="A5.p4.4.m4.1.1.2.2.cmml"><mo rspace="0.167em" id="A5.p4.4.m4.1.1.2.2.1" xref="A5.p4.4.m4.1.1.2.2.1.cmml">$</mo><mn id="A5.p4.4.m4.1.1.2.2.2" xref="A5.p4.4.m4.1.1.2.2.2.cmml">16.5</mn></mrow><mo id="A5.p4.4.m4.1.1.2.1" xref="A5.p4.4.m4.1.1.2.1.cmml">+</mo><mrow id="A5.p4.4.m4.1.1.2.3" xref="A5.p4.4.m4.1.1.2.3.cmml"><mo lspace="0em" rspace="0.167em" id="A5.p4.4.m4.1.1.2.3.1" xref="A5.p4.4.m4.1.1.2.3.1.cmml">$</mo><mn id="A5.p4.4.m4.1.1.2.3.2" xref="A5.p4.4.m4.1.1.2.3.2.cmml">5</mn></mrow></mrow><mo rspace="0.1389em" id="A5.p4.4.m4.1.1.1" xref="A5.p4.4.m4.1.1.1.cmml">=</mo><mrow id="A5.p4.4.m4.1.1.3" xref="A5.p4.4.m4.1.1.3.cmml"><mo lspace="0.1389em" rspace="0.167em" id="A5.p4.4.m4.1.1.3.1" xref="A5.p4.4.m4.1.1.3.1.cmml">$</mo><mn id="A5.p4.4.m4.1.1.3.2" xref="A5.p4.4.m4.1.1.3.2.cmml">21.5</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.p4.4.m4.1b"><apply id="A5.p4.4.m4.1.1.cmml" xref="A5.p4.4.m4.1.1"><eq id="A5.p4.4.m4.1.1.1.cmml" xref="A5.p4.4.m4.1.1.1"></eq><apply id="A5.p4.4.m4.1.1.2.cmml" xref="A5.p4.4.m4.1.1.2"><plus id="A5.p4.4.m4.1.1.2.1.cmml" xref="A5.p4.4.m4.1.1.2.1"></plus><apply id="A5.p4.4.m4.1.1.2.2.cmml" xref="A5.p4.4.m4.1.1.2.2"><csymbol cd="latexml" id="A5.p4.4.m4.1.1.2.2.1.cmml" xref="A5.p4.4.m4.1.1.2.2.1">currency-dollar</csymbol><cn type="float" id="A5.p4.4.m4.1.1.2.2.2.cmml" xref="A5.p4.4.m4.1.1.2.2.2">16.5</cn></apply><apply id="A5.p4.4.m4.1.1.2.3.cmml" xref="A5.p4.4.m4.1.1.2.3"><csymbol cd="latexml" id="A5.p4.4.m4.1.1.2.3.1.cmml" xref="A5.p4.4.m4.1.1.2.3.1">currency-dollar</csymbol><cn type="integer" id="A5.p4.4.m4.1.1.2.3.2.cmml" xref="A5.p4.4.m4.1.1.2.3.2">5</cn></apply></apply><apply id="A5.p4.4.m4.1.1.3.cmml" xref="A5.p4.4.m4.1.1.3"><csymbol cd="latexml" id="A5.p4.4.m4.1.1.3.1.cmml" xref="A5.p4.4.m4.1.1.3.1">currency-dollar</csymbol><cn type="float" id="A5.p4.4.m4.1.1.3.2.cmml" xref="A5.p4.4.m4.1.1.3.2">21.5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.4.m4.1c">\$16.5+\$5=\$21.5</annotation></semantics></math>. So, the answer is an average loss of <math id="A5.p4.5.m5.1" class="ltx_Math" alttext="\$3.5" display="inline"><semantics id="A5.p4.5.m5.1a"><mrow id="A5.p4.5.m5.1.1" xref="A5.p4.5.m5.1.1.cmml"><mo rspace="0.167em" id="A5.p4.5.m5.1.1.1" xref="A5.p4.5.m5.1.1.1.cmml">$</mo><mn id="A5.p4.5.m5.1.1.2" xref="A5.p4.5.m5.1.1.2.cmml">3.5</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.p4.5.m5.1b"><apply id="A5.p4.5.m5.1.1.cmml" xref="A5.p4.5.m5.1.1"><csymbol cd="latexml" id="A5.p4.5.m5.1.1.1.cmml" xref="A5.p4.5.m5.1.1.1">currency-dollar</csymbol><cn type="float" id="A5.p4.5.m5.1.1.2.cmml" xref="A5.p4.5.m5.1.1.2">3.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.5.m5.1c">\$3.5</annotation></semantics></math> per race, not <math id="A5.p4.6.m6.1" class="ltx_Math" alttext="\$3" display="inline"><semantics id="A5.p4.6.m6.1a"><mrow id="A5.p4.6.m6.1.1" xref="A5.p4.6.m6.1.1.cmml"><mo rspace="0.167em" id="A5.p4.6.m6.1.1.1" xref="A5.p4.6.m6.1.1.1.cmml">$</mo><mn id="A5.p4.6.m6.1.1.2" xref="A5.p4.6.m6.1.1.2.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.p4.6.m6.1b"><apply id="A5.p4.6.m6.1.1.cmml" xref="A5.p4.6.m6.1.1"><csymbol cd="latexml" id="A5.p4.6.m6.1.1.1.cmml" xref="A5.p4.6.m6.1.1.1">currency-dollar</csymbol><cn type="integer" id="A5.p4.6.m6.1.1.2.cmml" xref="A5.p4.6.m6.1.1.2">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.6.m6.1c">\$3</annotation></semantics></math> per race (the answer in the dataset).</p>
</div>
<div id="A5.p5" class="ltx_para">
<pre id="A5.p5.1" class="ltx_verbatim ltx_font_typewriter">

</pre>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.21786" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.21787" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.21787">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.21787" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.21788" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 15:27:43 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.21787
Scaling the amount of compute used to train language models has dramatically improved their capabilities. However, when it comes to inference, we often limit the amount of compute to only one attempt per problem. Here, we explore inference compute as another axis for scaling by increasing the number of generated samples. Across multiple tasks and models, we observe that coverage – the fraction of problems solved by any attempt – scales with the number of samples over four orders of magnitude. In domains like coding and formal proofs, where all answers can be automatically verified, these increases in coverage directly translate into improved performance.
When we apply repeated sampling to SWE-bench Lite, the fraction of issues solved with DeepSeek-V2-Coder-Instruct increases from 15.9% with one sample to 56% with 250 samples, outperforming the single-attempt state-of-the-art of 43% which uses more capable frontier models.
Moreover, using current API pricing, amplifying the cheaper DeepSeek model with five samples is more cost-effective and solves more issues than paying a premium for one sample from GPT-4o or Claude 3.5 Sonnet. Interestingly, the relationship between coverage and the number of samples is often log-linear and can be modelled with an exponentiated power law, suggesting the existence of inference-time scaling laws.
Finally, we find that identifying correct samples out of many generations remains an important direction for future research in domains without automatic verifiers.
When solving math word problems from GSM8K and MATH, coverage with Llama-3 models grows to over 95% with 10,000 samples.
However, common methods to pick correct solutions from a sample collection, such as majority voting or reward models, plateau beyond several hundred samples and fail to fully scale with the sample budget.
abstract
Large Language Monkeys00footnotetext:Title inspired byhttps://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Computewith Repeated Sampling
Abstract
2407.21787
1IntroductionThe ability of large language models (LLMs)  to solve coding, mathematics, and other reasoning tasks has improved dramatically over the past several years. Scaling up model training has been a consistent driver of these gains. Investments in larger models, larger pre-training datasets, and more extensive post-training (e.g. through collecting human preference labels)  has led to remarkably capable generalist systems.In contrast, a comparatively limited investment has been made in scaling the amount of computation used during inference. Larger models do require more inference compute than smaller ones, and prompting techniques like chain-of-thoughtcan increase answer quality at the cost of longer (and therefore more computationally expensive)  outputs. However, when interacting with LLMs, users and developers often restrict models to making only one attempt when solving a problem. In this work, we investigate repeated sampling (depicted in Figure1)  as an alternative axis for scaling inference compute to improve LLM reasoning performance.Figure 1:The repeated sampling procedure that we follow in this paper. 1)  We generate many candidate solutions for a given problem by sampling from an LLM with a positive temperature. 2)  We use a domain-specific verifier (ex. unit tests for code)  to select a final answer from the generated samples.The effectiveness of repeated sampling is determined by two key properties:1.Coverage: As the number of samples increases, what fraction of problems can we solve using any sample that was generated?2.Precision: In settings where we must select a final answer from the collection of generated samples, can we identify correct samples?With unlimited attempts, any model that assigns a non-zero probability to every sequence will achieve perfect coverage. However, repeated sampling is only practical if we can improve coverage with a feasible budget.
Moreover, without the ability to decide between samples, the applications of repeated sampling are limited. Existing work provides encouraging evidence along both of these directions, showing examples of repeated sampling improving performance in math, coding, and puzzle-solving settings. Notably, AlphaCode, a state-of-the-art system for competitive programming, finds that performance continues to improve with a million samples per problem.Here, we show that repeated sampling is an effective methodology for improving coverage across a range of tasks, models, and sample budgets.
For example, when solving CodeContestsprogramming problems using Gemma-2B, we increase coverage by over 300x as we scale the number of samples, from 0.02% with one attempt to 7.1% with 10,000 attempts.
Interestingly, the relationship betweenand the number of samples often follows an approximate power law. With Llama-3and Gemma models, we observe that coverage grows nearly log-linearly with the number of samples over several orders of magnitude.In settings where all of a model’s solutions can be automatically verified, such as with proof checkers or unit tests, these increases in coverage translate directly into improved task performance. When applying repeated sampling to competitive programming and writing Lean proofs, models like Llama-3-8B-Instruct can exceed the single-attempt performance of much stronger ones like GPT-4o. This ability to amplify weaker models extends to the challenging SWE-bench Lite dataset of real-life GitHub issues, where the current single-attempt state-of-the-art (SOTA) , achieved by a mixture of GPT-4o and Claude 3.5 Sonnet models, is 43%. When given a single attempt, DeepSeek-Coder-V2-Instructachieves only 15.9% on the benchmark. By simply increasing the number of attempts to 250, we increase the fraction of solved problems to 56%, exceeding the single-attempt state-of-the-art by 13%.In addition to improving model quality, repeated sampling provides a new mechanism for minimizing LLM inference costs. When holding the total number of inference FLOPs constant, we find that on some datasets (e.g. MATH)  coverage is maximized with a smaller model and more attempts, while on others (e.g CodeContests)  it is better to use a larger model. We also compare API prices between DeepSeek-V2-Coder-Instruct, GPT-4o, and Claude Sonnet 3.5 in the context of solving SWE-bench Lite issues. When keeping the agent framework (Moatless Tools)  constant, sampling five times from the weaker and cheaper DeepSeek model solves more issues than single attempts from Claude or GPT, while also being over 3x cheaper.Finally, in math word problem settings, where answers cannot be automatically verified by existing tools, we identify a large gap between coverage and the performance of common methods for deciding on a final answer. When solving MATHproblems with Llama-3-8B-Instruct, coverage increases from 79.8% with 100 samples to 95.3% with 10,000 samples. However, methods such as majority voting and using reward models plateau with a lower sample budget, scaling only from 38.7% to 39.8% over the same range. These results highlight that building robust verifiers remains an open problem.In summary, our primary observations are:1.We demonstrate that scaling inference compute through repeated sampling leads to large improvements in coverage across a variety tasks, models, and sample budgets. This makes it possible, and sometimes cost-effective, to amplify weaker models with many samples and outperform single attempts from more capable models. Notably, we are able to solve 56% of issues from SWE-bench Lite by sampling 250 times from DeepSeek-V2-Coder-Instruct, exceeding the single-attempt SOTA of 43%.2.We show that the relationship between coverage and the number of samples can often be modelled using an exponentiated power law, suggesting a form of scaling laws for inference-time compute.3.In domains without automatic verifiers, we show that common approaches to verification like majority voting and reward model scoring plateau beyond approximately 100 samples. This leads to a growing gap between the performance achieved with these methods and the coverage upper bound.
introduction
Large Language Monkeys00footnotetext:Title inspired byhttps://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Computewith Repeated Sampling
1Introduction
2407.21787
2Scaling Repeated SamplingWe focus on pass-fail tasks where a candidate solution can be scored as right or wrong. The primary metric of interest for these tasks is thesuccess rate:the fraction of problems that we are able to solve. With repeated sampling, we consider a setup where a model can generate many candidate solutions while attempting to solve a problem. The success rate is therefore influenced both by the ability to generate correct samples for many problems (i.e. coverage) , as well as the ability to identify these correct samples (i.e. precision) .The difficulty of the precision problem depends on the availability of tools for sample verification. When proving formal statements in Lean, proof checkers can quickly identify whether a candidate solution is correct. Similarly, unit tests can be used to verify candidate solutions to coding tasks. In these cases, precision is handled automatically, and improving coverage directly translates into higher success rates. In contrast, the tools available for verifying solutions to math word problems from GSM8K and MATH are limited, necessitating additional verification methods that decide on a single final answer from many (often conflicting)  samples.We consider the following five tasks:1.GSM8K:A dataset of grade-school level math word problems. We evaluate on a random subset of 128 problems from the GSM8K test set.2.MATH:Another dataset of math word problems that are generally harder than those from GSM8K. Similarly, we evaluate on 128 random problems from this dataset’s test set.3.MiniF2F-MATH:A dataset of mathematics problems that have been formalized into proof checking languages. We use Lean4 as our language, and evaluate on the 130 test set problems that are formalized from the MATH dataset.4.CodeContests:A dataset of competitive programming problems. Each problem has a text description, along with a set of input-output test cases (hidden from the model)  that can be used to verify the correctness of a candidate solution. We enforce that models write their solutions using Python3.5.SWE-bench Lite:A dataset of real world Github issues, where each problem consists of a description and a snapshot of a code repository. To solve a problem, models must edit files in the codebase (in the Lite subset of SWE-bench that we use, only a single file needs to be changed) . Candidate solutions can be automatically checked using the repository’s suite of unit tests.Among these tasks, MiniF2F-MATH, CodeContests, and SWE-bench Lite have automatic verifiers (in the form of the Lean4 proof checker, test cases, and unit test suites, respectively) .
We begin by investigating how repeated sampling improves model coverage. Coverage improvements correspond directly with increased success rates for tasks with automatic verifiers and in the general case provide an upper bound on the success rate.
In coding settings, our definition of coverage is equivalent to the commonly-used pass@k metric, wheredenotes the number of samples per problem. We use this metric directly when evaluating on CodeContests and SWE-bench Lite. For MiniF2F the metric is similar, with a “pass” defined according to the Lean4 proof checker. For GSM8K and MATH, coverage corresponds to using an oracle verifier that checks if any sample “passes” by outputting the correct final answer. To reduce the variance when calculating coverage, we adopt the unbiased estimation formula from. In each experiment, we first generatesamples for each problem indexand calculate the number of correct samples. We then calculate the pass@k scores at eachof interest according to:pass@k(1) We use the numerically stable implementation of the above formula suggested in.
Results and code will be available athttps://scalyresearch.stanford.edu/pubs/large_language_monkeys/.2.1Repeated Sampling is Effective Across TasksFigure 2:Across five tasks, we find that coverage (the fraction of problems solved by at least one generated sample)  increases as we scale the number of samples. Notably, using repeated sampling, we are able to increase the solve rate of an open-source method from 15.9% to 56% on SWE-bench Lite.Here, we establish that repeated sampling improves coverage across multiple tasks and a range of sample budgets. We evaluate Llama-3-8B-Instruct and Llama-3-70B-Instruct on CodeContests, MiniF2F, GSM8K, and MATH, generating 10,000 independent samples per problem. For SWE-bench Lite, we use DeepSeek-V2-Coder-Instruct, as the required context length of this task exceeds the limits of the Llama-3 models. As is standard when solving SWE-bench issues, we equip our LLM with a software framework that provides the model with tools for navigating through and editing codebases. In our work, we use the open-source Moatless Tools library. Note that solving a SWE-bench issue involves a back-and-forth exchange between the LLM and Moatless Tools. One sample/attempt for this benchmark refers to one entire multi-turn trajectory. To minimize costs, we restrict the number of attempts per issue to 250, with all attempts made independently of one another.We report our results in Figure2. We also include the single-attempt performance of GPT-4o on each task, as well the single-attempt state-of-the-art for SWE-bench Lite (CodeStory Aidewhich uses a combination of GPT-4o and Claude 3.5 Sonnet) . Across all five tasks, we find that coverage smoothly improves as the sample budget increases. When all LLMs are given a single attempt, GPT-4o outperforms the Llama and DeepSeek models at every task. However, as the number of samples increases, all three of the weaker models exceed GPT-4o’s single-attempt performance. In the case of SWE-bench Lite, we solve 56% of problems, exceeding the single-attempt SOTA of 43%.2.2Repeated Sampling is Effective Across Model Sizes and FamiliesFigure 3:Scaling inference time compute via repeated sampling leads to consistent coverage gains across a variety of model sizes (70M-70B) , families (Llama, Gemma and Pythia)  and levels of post-training (Base and Instruct models) .The results from Section2.1indicate that repeated sampling improves coverage. However, we only show this trend for three recent, instruction-tuned models with 8B or more parameters. We now show that these trends hold across other model sizes, families, and levels of post-training. We expand our evaluation to include a broader set of models:•Llama 3:Llama-3-8B, Llama-3-8B-Instruct, Llama-3-70B-Instruct.•Gemma:Gemma-2B, Gemma-7B.•Pythia:Pythia-70M through Pythia-12B (eight models in total) .We restrict evaluation to the MATH and CodeContests datasets to minimize inference costs, reporting results in Figure3. Coverage increases across almost every model we test, with smaller models showing some of the sharpest increases in coverage when repeated sampling is applied. On CodeContests, the coverage of Gemma-2B increases by over 300x, from a pass@1 of 0.02% to a pass@10k of 7.1%.
Similarly, when solving MATH problems with Pythia-160M, coverage increases from a pass@1 of 0.27% to a pass@10k of 57%.The exception to this pattern of increasing coverage across models is with the Pythia family evaluated on CodeContests. All Pythia models achieve zero coverage on this dataset, even with a budget of 10,000 samples. We speculate that this due to Pythia being trained on less coding-specific data than Llama and Gemma.2.3Repeated Sampling Can Help Balance Performance and CostOne takeaway from the results in Sections2.1and2.2is that repeated sampling makes it possible to amplify a weaker model’s capabilities and outperform single samples from stronger models.
Here, we demonstrate that this amplification can be more cost-effective than using a stronger, more expensive model, providing practitioners with a new degree of freedom when trying to jointly optimize performance and costs.We first consider FLOPs as a cost metric, examining the Llama-3 results from Section2.1. We re-plot our results from Figure2, now visualizing coverage as a function of total inference FLOPs instead of the sample budget. Since Llama-3 models are dense transformers where the majority of parameters are used in matrix multiplications, we approximate inference FLOPs with the formula:FLOPs per tokentotal inference FLOPsWe present our re-scaled results for MiniF2F, CodeContests, MATH, and GSM8K in Figure4. Interestingly, the model that maximizes coverage varies with the compute budget and task. On MiniF2F, GSM8K and MATH, Llama-3-8B-Instruct always obtains higher coverage than the larger (and more expensive)  70B model when the FLOP budget is fixed.
However for CodeContests, the 70B model is almost always more cost effective.
We note that examining FLOPs alone can be a crude cost metric that ignores other aspects of system efficiency.
In particular, repeated sampling can make use of high batch sizes and specialized optimizations that improve system throughput relative to single-attempt inference workloads.
We discuss this in more detail in Section5.Figure 4:Comparing cost, measured in number of inference FLOPs, and coverage for Llama-3-8B-Instruct and Llama-3-70B-Instruct. We see that the ideal model size depends on the task, compute budget, and coverage requirements. Note that Llama-3-70B-Instruct does not achieve 100% coverage on GSM8K due to an incorrectly labelled ground truth answer: see AppendixE.We also examine the dollar costs of repeated sampling when solving SWE-bench Lite issues using current API pricing. Keeping the agent framework (Moatless Tools)  constant, we consider making a single attempt per issue with Claude 3.5 Sonnet and GPT-4o, as well as repeated sampling using DeepSeek-V2-Coder-Instruct. We report the average cost per issue and issue resolution rate with each approach in Table1. While the DeepSeek model is weaker than the GPT and Claude models, it is also over 10x cheaper. In this case, repeated sampling provides a cheaper alternative to paying a premium for access to strong models while achieving a superior issue solve rate.ModelCost perattempt(USD) Number ofattemptsIssuessolved (%) Total Cost(USD) RelativeTotal CostDeepSeek-V2-Coder-Instruct0.008529.62121xGPT-4o0.13124.00393.25xClaude 3.5 Sonnet0.17126.70514.25xTable 1:Comparing API cost (in US dollars)  and performance for various models on the SWE-bench Lite dataset using the Moatless Tools agent framework. When sampled more, the open-source DeepSeekCoder-V2 model can achieve the same issue solve-rate as closed-source frontier models for under a third of the price.
body
Large Language Monkeys00footnotetext:Title inspired byhttps://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Computewith Repeated Sampling
2Scaling Repeated Sampling
2407.21787
3Characterizing the Benefits of Repeated SamplingThe relationship between an LLM’s loss and its training compute has been well-characterized with training scaling laws.
These laws have empirically held over many orders of magnitude and inspire confidence in model developers that large investments in training will pay off.
Inspired by training scaling laws, here we aim to better characterize the relationship between coverage and the sample budget (i.e. the amount of inference compute) , presenting two interesting observations:1.The relationship between coverage and the number of samples can often be modelled with an exponentiated power law.2.For a given task, the coverage curves of different models from the same family resemble S-curves with similar slopes but distinct horizontal offsets.3.1Scaling Laws for Repeated SamplingFigure 5:The relationship between coverage and the number of samples can be modelled with an exponentiated power law for most tasks and models. We highlight that some curves, such as Llama-3-8B-Instruct on MiniF2F-MATH, do not follow this trend closely.Here, we develop an explicit model for the relationship between coverage and the number of samples. The GPT-4 technical reportfinds that the relationship between a model’s mean-log-pass-rate on coding problems and its training compute can be modelled well using a power law. We start by adopting the same function class, but now modelling the log of coverageas a function of the number of samples:(2) whereare fitted model parameters.
In order to directly predict coverage, we exponentiate both sides, ending up with the final model of:(3) We provide examples of fitted coverage curves in Figure5, and additional curves in AppendixC.2. While these laws are not as exact as training scaling laws (most strikingly on MiniF2F-MATH) , they provide encouraging early evidence that the benefits of inference scaling can be characterized.3.2Similarities in Coverage Curves Across ModelsFigure 6:Overlaying the coverage curves from different models belonging to the same family. We perform this overlay by horizontally shifting every curve (with a logarithmic x-axis)  so that all curves pass through the point (1,) . We pickto be the maximum pass@1 score over all models in the plot. We note that the similarity of the curves post-shifting shows that, within a model family, sampling scaling curves follow a similar shape.Interestingly, when comparing the coverage curves (with a logarithmic x-axis)  of different models from the same family on the same task (see Figure3) , it appears that the traced S-curves have the same slope, but unique horizontal offsets. To investigate this further, we overlay the coverage curves of different models from the same family in Figure6. We do this by picking an anchor coverage value, and shifting every curve leftward (in log-space)  so that each passes through the point (1,) . This corresponds to a leftward shift by, wheredenotes the closest natural numbersuch that. We pickto be the maximum pass@1 score over all models from the same family. These similarities demonstrate that across models from the same family, the increase in the log-sample-budget (or equivalently, the multiplicative increase in the sample budget)  needed to improve coverage fromtois approximately constant.
body
Large Language Monkeys00footnotetext:Title inspired byhttps://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Computewith Repeated Sampling
3Characterizing the Benefits of Repeated Sampling
2407.21787
4Harnessing Repeated Sampling Requires PrecisionSo far, we have focused on measuring model coverage, characterizing the benefits of repeated sampling under the best-case scenario where we can always identify correct model samples. We now turn to the complementary problem of precision: given a collection of model samples, can we identify the correct ones? In Section4.1, we evaluate two common verification methods (majority voting and reward model scoring)  on GSM8K and MATH. Additionally, in Section4.2, we discuss potential pitfalls when relying on unit tests to identify correct software programs.4.1Common Verification Methods Don’t Always Scale with the Sample BudgetFigure 7:Comparing coverage (performance with an oracle verifier)  to mainstream methods available for picking the correct answer (majority voting, reward model selection and reward model majority voting)  as we increase the number of samples. Although near-perfect coverage is achieved, all sample selection methods fail to reach the coverage upper bound and saturate before reaching 100 samples. For every k value, we calculate the metric on 100 subsets of size k then plot the mean and one standard deviation across subsets.Of the five tasks we evaluate, only GSM8K and MATH lack tools for automatically verifying solutions. Here, we evaluate two common approaches to deciding on a final answer: calculating a majority vote across samples and using a reward model to assign a score to each sample. We test these techniques on their ability to identify correct solutions from the 10,000 samples we generated with Llama-3-8B-Instruct and Llama-3-70B-Instruct in Section2. We benchmark three methods:1.Majority Vote:We pick the most common final answer.2.Reward Model + Best-of-N:We use a reward modelto score each solution, and pick the answer from the highest-scoring sample.3.Reward Model + Majority Vote:We calculate a majority vote where each sample is weighted by its reward model score.We use ArmoRM-Llama3-8B-v0.1as a reward model, which currently has the highest reasoning score on the RewardBench leaderboardamong open-weight models. We report our results in Figure7as we increase the number of samples. While success rate initially increases with the number of samples for all three methods, it plateaus around 100 samples. Meanwhile, coverage continues to increase with the number of samples and exceeds 95%.In the case of majority voting, this success rate saturation is easily explainable. As the number of samples increases, the proportion of votes allocated to each answer stabilizes, and therefore the success rate plateaus. For some GSM8K and MATH problems, correct solutions are sampled with a probability of 1% or lower (see Figure8) , making them a minority of samples. As the number of samples increases, rare correct solutions will appear for more problems, increasing coverage but not the success rate with majority voting. In order to fully benefit from repeated sampling, sample identification methods must be able to solve these “needle-in-a-haystack” cases and identify rare, correct samples.Given the poor performance of the reward model, it is reasonable to wonder how “hard” it is to verify a candidate solution. With GSM8K and MATH, only a sample’s final answer is used for assessing correctness, with the intermediate chains of thought being excluded. If models generated only non-sensical chains of thought before guessing a correct final answer, verification may not be any easier than solving the problem in the first place. We investigate this question by manually evaluating 105 chains of thought from correct Llama-3-8B-Instruct solutions to GSM8K problems, reporting our results in Table2.
We find that over 90% of the chains of thought that we graded are faithful, even among problems where correct answers are generated infrequently.
These correct reasoning steps indicate that there is signal for a verifier to exploit when identifying correct samples. Interestingly, during this process we also identified one GSM8K problem that has an incorrect ground truth answer (see AppendixE) .
This incorrect GSM8K problem is also the only one that Llama-3-70B-Instruct did not generate a “correct” sample for across 10,000 attempts.Pass@1# Problems# CoT GradedCorrect CoTIncorrect CoTIncorrect Ground Truth0-10%5151111 problem, 3 CoTs10-25%10302730 problems25-75%29302820 problems75-100%84303000 problemsTable 2:Human evaluation of the validity of the Chain-of-Thought reasoning in Llama-3-8B-Instruct answers to GSM8K problems. 3 chains of thought were graded per problem. Even for difficult questions, where the model only getsof samples correct, the CoTs almost always follow valid logical steps. For the model generations and human labels,see here.4.2Verifiers and Software Tasks: Two Cautionary TalesSoftware development tasks can occupy a middle-ground with respect to available verification tools.
On one hand, the ability to execute and test code allows for a higher degree of automatic verification than is possible with unstructured language tasks.
However, tools like unit tests take a black-box approach to verifying a piece of code and are not as comprehensive as methods like proof checkers.
These imperfections in the verification process can lead to false positives and/or false negatives that are important to consider when applying repeated sampling. Below we provide two examples of software verifier imperfections that we encountered when generating our results from Section2.1.4.2.1Flaky Tests in SWE-bench LiteWhen producing our results on SWE-bench Lite, we identified that 11.3% of problems have flaky test suites that do not produce consistent results when running them on the same candidate solution.
These flaky tests occasionally classify even the dataset’s ground-truth issue solutions as incorrect. Additionally, the test suites for some issues can be non-determinstic depending on the candidate solution.
For example, two SWE-bench Lite issues involve manipulating Python sets, which are naturally unordered.
The gold solutions for these issues explicitly order the items in the set and pass the test suites reliably. However, some model-generated candidate solutions do not impose such an ordering, and therefore pass the tests on some “lucky” runs and not others.
In AppendixB, we list all of the problem IDs where we identified flaky tests. We also report our SWE-bench Lite results from Figure2with the problematic issues removed, finding similar results to our evaluations on the whole dataset.4.2.2False Negatives in CodeContestsEach problem from the CodeContests dataset comes with a set of input-output test cases used to asses the correctness of solutions. These test cases are more comprehensive than those from earlier coding benchmarks like APPS, cutting down on the frequency of false positive solutions that pass all test cases but do not fully solve the described problem. However, the construction of the CodeContests test suites leads to false negative solutions that are correct but fail the tests.For some CodeContests problems, the problem description allows for multiple distinct correct outputs for a given test input. However, the corresponding test cases do not handle these scenarios, instead requiring that one particular correct output is emitted. Additionally, many CodeContests test cases have been programmatically generated by mutating original test cases from the problem. Some mutated inputs violate the problem’s input specifications (e.g. a mutated input being zero when the description promises a positive integer) .
These malformed test cases can lead to inconsistent behaviour between different correct solutions.We assess the prevalence of these issues by running each problem’s test suite on the list of correct solutions that CodeContests provides. Of the 122 problems in the test set that have Python3 solutions, we find that 35 problems have “correct” solutions that fail the corresponding tests. Since we do not allow models to view all of a problem’s test cases (and their peculiarities) , applying repeated sampling to these problems contains an element of “rolling the dice” to generate a solution that is not only correct, but emits the particular outputs that pass the tests.Figure 8:Bar charts showing the fraction of samples (out of 10,000 samples)  that are correct, for each problem in the subsets of GSM8K and MATH we evaluate on. There is one bar per problem, and the height of the bar corresponds to the fraction of samples that arrive at the
correct answer. Bars are green if self-consistency picked the correct answer and are red otherwise.
We highlight that there are many problems with correct solutions, where the correct solutions are sampled infrequently.
body
Large Language Monkeys00footnotetext:Title inspired byhttps://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Computewith Repeated Sampling
4Harnessing Repeated Sampling Requires Precision
2407.21787
5Discussion and LimitationsIn this work, we explore repeated sampling as an axis for scaling compute at inference time in order to improve model performance. Across a range of models and tasks, repeated sampling can significantly improve the fraction of problems solved using any generated sample (i.e. coverage) . When correct solutions can be identified (either with automatic verification tools or other verification algorithms) , repeated sampling can amplify model capabilities during inference. This amplification can make the combination of a weaker model and many samples more performant and cost-effective than using fewer attempts from a stronger, more expensive model.Improving Repeated Sampling:In our experiments, we explore only a simple version of repeated sampling where all attempts to a problem are generated independently of one another using the exact same prompt and hyperparameters. We believe that this setup can be refined to improve performance, particularly along the following directions:1.Solution Diversity:We currently rely on a positive sampling temperature as the sole mechanism for creating diversity among samples. Combining this token-level sampling with other, higher-level approaches may be able to further increase diversity. For example, AlphaCode conditions different samples with different metadata tags.2.Multi-Turn Interactions: Despite automatic verification tools being available when solving CodeContests and MiniF2F problems, we use only a single-turn setup where models generate a solution without any ability to iterate on it. Providing models with execution feedback from these tools should improve solution quality. We are interested in the tradeoffs associated with multi-turn interactions, since each attempt becomes more expensive, but also may be more likely to succeed.3.Learning From Previous Attempts:Currently, our experiments fully isolate attempts from each other. Access to existing samples, particularly if verification tools can provide feedback on them, may be helpful when generating future attempts.Repeated Sampling and Inference Systems:Repeated sampling is a distinct LLM inference workload from serving chatbot requests. Production chatbot deployments place an emphasis on low response latencies, and adhering to latency targets can force a lower per-device batch size and reduce hardware utilization. In contrast, when sampling many completions to a single prompt, a larger emphasis can be placed on overall throughput and maximizing hardware utilization. Additionally, repeated sampling can benefit from specialized attention optimizations that exploit overlaps in prompts across sequences. Repeated sampling inference can therefore be accomplished at a lower cost than naively making many parallel requests to a chatbot-oriented API. These cost savings can further motivate choosing to sample many times from a cheaper model instead of fewer times from a more expensive one.Verifiers:Our results from Section4highlight the importance of improving sample verification methods when tools for automatically doing so are unavailable. Equipping models with the ability to assess their own outputs will allow repeated sampling to be scaled to far more tasks. Of particular interest is applying repeated sampling to unstructured tasks like creative writing, which can require a more subjective comparison between different samples than the pass-fail tasks we consider. An alternative direction to developing model-based verifiers is to design converters that can make an unstructured task verifiable, for example by formalizing an informal math statement into a language like Lean so that proof checkers can be applied.
body
Large Language Monkeys00footnotetext:Title inspired byhttps://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Computewith Repeated Sampling
5Discussion and Limitations
2407.21787
6Related WorkScaling Inference Compute:Methods that perform additional computation during inference have been successful across many areas of deep learning. Across a variety of game environments, state-of-the-art methods leverage inference-time search that examines many possible future game states before deciding on a move. Similar tree-based methods can also be effective in combination with LLMs, allowing models to better plan and explore different approaches. Another axis for increasing LLM inference compute allows models to spend tokens deliberating on a problem before coming to a solution. Additionally, multiple models can be ensembled together at inference time to combine their strengths. Yet another approach involves using LLMs to critique and refine their own responses.Repeated Sampling:Previous work has demonstrated that repeated sampling can improve LLM capabilities in multiple domains. One of the most effective use cases is coding, where performance continues to scale up to a million samples and verification tools (e.g. unit tests)  are often available to automatically score every candidate solution.
Recently,shows that repeated sampling is effective when solving puzzles from the ARC challenge, observing log-linear scaling as the number of samples increases.
In chat applications, repeated sampling combined with best-of-N ranking with a reward model can outperform greedily sampling a single response.
In domains without automatic verification tools, existing work shows that using majority votingor a trained model-based verifierto decide on a final answer can improve performance on reasoning tasks relative to taking a single sample. Concurrent with our work,finds that using the best available sample improves LLM performance on chat, math, and code tasks, sweeping up to a max of 128 samples.Scaling Laws:Characterizing how scaling affects model performance can lead to more informed decisions on how to allocate resources. Scaling laws for LLM training find a power law relationship between loss and the amount of training compute and provide estimates for the optimal model and dataset size given a fixed compute budget.finds scaling laws in the context of the board game Hex, observing that performance scales predictably with model size and the difficulty of the problem.
Interestingly, they also show that performance scales with the amount of test-time compute spent while performing tree search.
Recently,observe scaling laws when augmenting LLMs with external retrieval datasets, finding that performance on retrieval tasks scales smoothly with the size of the retrieval corpus.
body
Large Language Monkeys00footnotetext:Title inspired byhttps://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Computewith Repeated Sampling
6Related Work
2407.21787
7AcknowledgementsWe thank Together AI for partially sponsoring the compute for this project, as well as Rahul Chalamala and Ben Athiwaratkun for their help managing this infrastructure. We thank John Yang for his advice and support when running our SWE-bench experiments. Finally, we are grateful to Mayee Chen, Neel Guha, Quinn McIntyre, Jon Saad-Falcon, and Benjamin Spector for their helpful discussions and feedback throughout this project.We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize) , NSF under Nos. CCF2247015 (Hardware-Aware) , CCF1763315 (Beyond Sparsity) , CCF1563078 (Volume to Velocity) , and 1937301 (RTML) ; US DEVCOM ARL under Nos. W911NF-23-2-0184 (Long-context)  and W911NF-21-2-0251 (Interactive Human-AI Teaming) ; ONR under Nos. N000142312633 (Deep Signal Processing) ; Stanford HAI under No. 247183; NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI) , and members of the Stanford DAWN project: Meta, Google, and VMWare. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.This work was completed with the support of the Clarendon Fund Scholarships.
conclusion
Large Language Monkeys00footnotetext:Title inspired byhttps://en.m.wikipedia.org/wiki/Infinite_monkey_theorem.: Scaling Inference Computewith Repeated Sampling
7Acknowledgements