<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2105.09938] Measuring Coding Challenge Competence With APPS</title><meta property="og:description" content="While programming is one of the most broadly applicable skills in modern society, it is unclear how well state-of-the-art machine learning models can write code. Despite its importance, there has been surprisingly litt…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Measuring Coding Challenge Competence With APPS">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Measuring Coding Challenge Competence With APPS">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2105.09938">

<!--Generated on Thu Mar  7 00:35:40 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Measuring Coding Challenge Competence With APPS</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Dan Hendrycks
<br class="ltx_break">UC Berkeley
<br class="ltx_break">&amp;Steven Basart<sup id="id3.1.id1" class="ltx_sup">*</sup>
<br class="ltx_break">UChicago
<br class="ltx_break">&amp;Saurav Kadavath
<br class="ltx_break">UC Berkeley
<br class="ltx_break">&amp;Mantas Mazeika
<br class="ltx_break">UIUC
<br class="ltx_break">&amp;Akul Arora
<br class="ltx_break">UC Berkeley
<br class="ltx_break">&amp;Ethan Guo
<br class="ltx_break">UC Berkeley
<br class="ltx_break">&amp;Collin Burns
<br class="ltx_break">UC Berkeley
<br class="ltx_break">&amp;Samir Puranik
<br class="ltx_break">UC Berkeley
<br class="ltx_break">&amp;Horace He
<br class="ltx_break">Cornell
<br class="ltx_break"><span id="id4.2.id2" class="ltx_ERROR undefined">\AND</span>Dawn Song
<br class="ltx_break">UC Berkeley
<br class="ltx_break">&amp;Jacob Steinhardt
<br class="ltx_break">UC Berkeley
<br class="ltx_break">
</span><span class="ltx_author_notes">Equal Contribution.</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id2.2" class="ltx_p">While programming is one of the most broadly applicable skills in modern society, it is unclear how well state-of-the-art machine learning models can write code. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to assess code generation performance in an accurate and rigorous manner. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we evaluate models by checking their generated code on test cases. Our benchmark includes <math id="id1.1.m1.2" class="ltx_Math" alttext="10,\!000" display="inline"><semantics id="id1.1.m1.2a"><mrow id="id1.1.m1.2.3.2" xref="id1.1.m1.2.3.1.cmml"><mn id="id1.1.m1.1.1" xref="id1.1.m1.1.1.cmml">10</mn><mpadded width="0.275em"><mo id="id1.1.m1.2.3.2.1" xref="id1.1.m1.2.3.1.cmml">,</mo></mpadded><mn id="id1.1.m1.2.2" xref="id1.1.m1.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="id1.1.m1.2b"><list id="id1.1.m1.2.3.1.cmml" xref="id1.1.m1.2.3.2"><cn type="integer" id="id1.1.m1.1.1.cmml" xref="id1.1.m1.1.1">10</cn><cn type="integer" id="id1.1.m1.2.2.cmml" xref="id1.1.m1.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="id1.1.m1.2c">10,\!000</annotation></semantics></math> problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately <math id="id2.2.m2.1" class="ltx_Math" alttext="20\%" display="inline"><semantics id="id2.2.m2.1a"><mrow id="id2.2.m2.1.1" xref="id2.2.m2.1.1.cmml"><mn id="id2.2.m2.1.1.2" xref="id2.2.m2.1.1.2.cmml">20</mn><mo id="id2.2.m2.1.1.1" xref="id2.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="id2.2.m2.1b"><apply id="id2.2.m2.1.1.cmml" xref="id2.2.m2.1.1"><csymbol cd="latexml" id="id2.2.m2.1.1.1.cmml" xref="id2.2.m2.1.1.1">percent</csymbol><cn type="integer" id="id2.2.m2.1.1.2.cmml" xref="id2.2.m2.1.1.2">20</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="id2.2.m2.1c">20\%</annotation></semantics></math> of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an objective measure for tracking advancements.</p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<p id="p1.1" class="ltx_p">“Everybody should learn to program a computer, because it teaches you how to think.” – <em id="p1.1.1" class="ltx_emph ltx_font_italic">Steve Jobs</em></p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Computer programming can be found in nearly all parts of society. Spanning entertainment, healthcare, education, and more, programming is an extraordinarily general tool with applications that are vast in scope. As computers are becoming more ubiquitous in modern life, rising demand for high-quality code draws an ever-greater number of aspiring programmers to the profession. After years of study to become proficient coders, human experts are are able to convert abstract specifications of diverse cognitive tasks into concrete programs.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">In the past few years, large-scale language models have shown promise in generalizing to various cognitive tasks, including linguistic inference <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib35" title="" class="ltx_ref">2019a</a>)</cite>, commonsense reasoning <cite class="ltx_cite ltx_citemacro_citep">(Zellers et al., <a href="#bib.bib42" title="" class="ltx_ref">2019</a>; Huang et al., <a href="#bib.bib15" title="" class="ltx_ref">2019</a>; Bisk et al., <a href="#bib.bib3" title="" class="ltx_ref">2019</a>)</cite>, logical deduction <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib21" title="" class="ltx_ref">2020b</a>)</cite>, mathematics <cite class="ltx_cite ltx_citemacro_citep">(Polu and Sutskever, <a href="#bib.bib26" title="" class="ltx_ref">2020</a>; Hendrycks et al., <a href="#bib.bib14" title="" class="ltx_ref">2021c</a>)</cite>, and general understanding of multiple domains of human knowledge <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib13" title="" class="ltx_ref">2021b</a>)</cite>. However, whether large-scale language models can reliably write code remains an open question.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">Motivated by the potential of language models and the need for thorough code generation evaluation, we introduce APPS, a benchmark for code generation from natural language specifications. Unlike prior work on code generation with Transformer language models <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib34" title="" class="ltx_ref">2017</a>)</cite>, which mostly focuses on code translation <cite class="ltx_cite ltx_citemacro_citep">(Lachaux et al., <a href="#bib.bib18" title="" class="ltx_ref">2020</a>)</cite> and pseudocode-to-code <cite class="ltx_cite ltx_citemacro_citep">(Kulal et al., <a href="#bib.bib17" title="" class="ltx_ref">2019</a>)</cite>, we evaluate models on their ability to take specifications given in natural language and write code that meets these specifications. This setting mirrors how human coders are evaluated and is a more realistic and informative setting in which to benchmark models.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_missing ltx_missing_image" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>An example problem from APPS (left) along with possible generated code (middle) and two example test cases we use to evaluate the generated code (right). Our evaluation framework has test cases and <math id="S1.F1.2.m1.2" class="ltx_Math" alttext="10,\!000" display="inline"><semantics id="S1.F1.2.m1.2b"><mrow id="S1.F1.2.m1.2.3.2" xref="S1.F1.2.m1.2.3.1.cmml"><mn id="S1.F1.2.m1.1.1" xref="S1.F1.2.m1.1.1.cmml">10</mn><mpadded width="0.275em"><mo id="S1.F1.2.m1.2.3.2.1" xref="S1.F1.2.m1.2.3.1.cmml">,</mo></mpadded><mn id="S1.F1.2.m1.2.2" xref="S1.F1.2.m1.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.F1.2.m1.2c"><list id="S1.F1.2.m1.2.3.1.cmml" xref="S1.F1.2.m1.2.3.2"><cn type="integer" id="S1.F1.2.m1.1.1.cmml" xref="S1.F1.2.m1.1.1">10</cn><cn type="integer" id="S1.F1.2.m1.2.2.cmml" xref="S1.F1.2.m1.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.2.m1.2d">10,\!000</annotation></semantics></math> code generation problems of varying difficulty levels.</figcaption>
</figure>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">APPS provides a precise and comprehensive view of code generation. APPS evaluates models not only on their ability to code syntactically correct programs, but also on their ability to understand task descriptions and devise algorithms to solve these tasks. It contains <math id="S1.p4.1.m1.2" class="ltx_Math" alttext="10,\!000" display="inline"><semantics id="S1.p4.1.m1.2a"><mrow id="S1.p4.1.m1.2.3.2" xref="S1.p4.1.m1.2.3.1.cmml"><mn id="S1.p4.1.m1.1.1" xref="S1.p4.1.m1.1.1.cmml">10</mn><mpadded width="0.275em"><mo id="S1.p4.1.m1.2.3.2.1" xref="S1.p4.1.m1.2.3.1.cmml">,</mo></mpadded><mn id="S1.p4.1.m1.2.2" xref="S1.p4.1.m1.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.p4.1.m1.2b"><list id="S1.p4.1.m1.2.3.1.cmml" xref="S1.p4.1.m1.2.3.2"><cn type="integer" id="S1.p4.1.m1.1.1.cmml" xref="S1.p4.1.m1.1.1">10</cn><cn type="integer" id="S1.p4.1.m1.2.2.cmml" xref="S1.p4.1.m1.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.1.m1.2c">10,\!000</annotation></semantics></math> programming problems at various levels of difficulty, covering simple introductory problems, interview-level problems, and coding competition challenges. If a model were to perform well on APPS, this would indicate an ability to flexibly use data structures and programming techniques, as well as an ability to correctly interpret diverse task specifications, follow instructions, and understand human intent <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib12" title="" class="ltx_ref">2021a</a>)</cite>.</p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">For most text generation tasks, high-quality evaluation requires human feedback, which can be time-consuming or carry pecuniary costs. As a result, automatic metrics such as BLEU <cite class="ltx_cite ltx_citemacro_citep">(Papineni et al., <a href="#bib.bib25" title="" class="ltx_ref">2002</a>)</cite> are often used to compare methods, but these metrics do not necessarily track program correctness. Since the objective for code generation is to produce correct programs, we assess programs not with BLEU but with test cases and error catching. Evaluating code generation on APPS is facilitated by a large bank of over <math id="S1.p5.1.m1.2" class="ltx_Math" alttext="130,\!000" display="inline"><semantics id="S1.p5.1.m1.2a"><mrow id="S1.p5.1.m1.2.3.2" xref="S1.p5.1.m1.2.3.1.cmml"><mn id="S1.p5.1.m1.1.1" xref="S1.p5.1.m1.1.1.cmml">130</mn><mpadded width="0.275em"><mo id="S1.p5.1.m1.2.3.2.1" xref="S1.p5.1.m1.2.3.1.cmml">,</mo></mpadded><mn id="S1.p5.1.m1.2.2" xref="S1.p5.1.m1.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.1.m1.2b"><list id="S1.p5.1.m1.2.3.1.cmml" xref="S1.p5.1.m1.2.3.2"><cn type="integer" id="S1.p5.1.m1.1.1.cmml" xref="S1.p5.1.m1.1.1">130</cn><cn type="integer" id="S1.p5.1.m1.2.2.cmml" xref="S1.p5.1.m1.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.1.m1.2c">130,\!000</annotation></semantics></math> test cases. The test cases are specifically chosen to probe correct functionality across the input space. By using test cases, we provide a gold-standard metric for code generation quality.</p>
</div>
<div id="S1.p6" class="ltx_para ltx_noindent">
<p id="S1.p6.1" class="ltx_p">In our experiments, we find that models are now starting to exhibit nonzero accuracy and solve some coding problems. Additionally, as models improve, we observe that syntax errors are exponentially decreasing. We also find further evidence that BLEU is a problematic metric for code generation, sometimes being anticorrelated with gold-standard accuracy. We find that accuracy decreases with difficulty level and improves through fine-tuning and model size increases. The strongest model that we evaluate on introductory problems passes almost <math id="S1.p6.1.m1.1" class="ltx_Math" alttext="20\%" display="inline"><semantics id="S1.p6.1.m1.1a"><mrow id="S1.p6.1.m1.1.1" xref="S1.p6.1.m1.1.1.cmml"><mn id="S1.p6.1.m1.1.1.2" xref="S1.p6.1.m1.1.1.2.cmml">20</mn><mo id="S1.p6.1.m1.1.1.1" xref="S1.p6.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S1.p6.1.m1.1b"><apply id="S1.p6.1.m1.1.1.cmml" xref="S1.p6.1.m1.1.1"><csymbol cd="latexml" id="S1.p6.1.m1.1.1.1.cmml" xref="S1.p6.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S1.p6.1.m1.1.1.2.cmml" xref="S1.p6.1.m1.1.1.2">20</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p6.1.m1.1c">20\%</annotation></semantics></math> of test cases given five attempts. These results position code generation as a challenging but now tractable testbed for large-scale language models.</p>
</div>
<div id="S1.p7" class="ltx_para ltx_noindent">
<p id="S1.p7.1" class="ltx_p">Writing code to meet specifications in natural language is an economically valuable task with widespread social implications should it be solved, as it could eventually facilitate malicious code generation and one day result in job automation. As large-scale language models have the potential to make significant progress on code generation, it is essential that we begin to track advancements on this task. Our new benchmark facilitates measuring performance in an accurate and rigorous manner. Using APPS, we find that programming is very difficult for modern language models, though performance is improving. Thus, the APPS benchmark can provide foresight about the performance of future large-scale language models at the critical task of program synthesis from natural language. The dataset is available at <a target="_blank" href="https://github.com/hendrycks/apps" title="" class="ltx_ref ltx_href">https://github.com/hendrycks/apps</a>.</p>
</div>
<figure id="S1.T1" class="ltx_table">
<table id="S1.T1.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S1.T1.2.3" class="ltx_tr">
<td id="S1.T1.2.3.1" class="ltx_td" style="padding-left:9.0pt;padding-right:9.0pt;"></td>
<td id="S1.T1.2.3.2" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">PY150</td>
<td id="S1.T1.2.3.3" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">CONCODE</td>
<td id="S1.T1.2.3.4" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">SPoC</td>
<td id="S1.T1.2.3.5" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">APPS</td>
</tr>
<tr id="S1.T1.2.4" class="ltx_tr">
<td id="S1.T1.2.4.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:9.0pt;padding-right:9.0pt;">Programming Language</td>
<td id="S1.T1.2.4.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:9.0pt;padding-right:9.0pt;">Python</td>
<td id="S1.T1.2.4.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:9.0pt;padding-right:9.0pt;">Java</td>
<td id="S1.T1.2.4.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:9.0pt;padding-right:9.0pt;">C++</td>
<td id="S1.T1.2.4.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:9.0pt;padding-right:9.0pt;">Python</td>
</tr>
<tr id="S1.T1.2.2" class="ltx_tr">
<td id="S1.T1.2.2.3" class="ltx_td ltx_align_left" style="padding-left:9.0pt;padding-right:9.0pt;">Test Cases</td>
<td id="S1.T1.2.2.4" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;"><span id="S1.T1.2.2.4.1" class="ltx_text" style="color:#FF0000;">✕</span></td>
<td id="S1.T1.2.2.5" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;"><span id="S1.T1.2.2.5.1" class="ltx_text" style="color:#FF0000;">✕</span></td>
<td id="S1.T1.1.1.1" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;"><span id="S1.T1.1.1.1.1" class="ltx_text" style="color:#009A18;"><svg id="S1.T1.1.1.1.1.pic1" class="ltx_picture" height="11.02" overflow="visible" version="1.1" width="15.75"><g transform="translate(0,11.02) matrix(1 0 0 -1 0 0)" fill="#009A18" stroke="#009A18" stroke-width="0.4pt"><path d="M 0 5.51 L 3.94 0 L 15.75 11.02 L 3.94 2.36 Z" style="stroke:none"></path></g></svg></span></td>
<td id="S1.T1.2.2.2" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;"><span id="S1.T1.2.2.2.1" class="ltx_text" style="color:#009A18;"><svg id="S1.T1.2.2.2.1.pic1" class="ltx_picture" height="11.02" overflow="visible" version="1.1" width="15.75"><g transform="translate(0,11.02) matrix(1 0 0 -1 0 0)" fill="#009A18" stroke="#009A18" stroke-width="0.4pt"><path d="M 0 5.51 L 3.94 0 L 15.75 11.02 L 3.94 2.36 Z" style="stroke:none"></path></g></svg></span></td>
</tr>
<tr id="S1.T1.2.5" class="ltx_tr">
<td id="S1.T1.2.5.1" class="ltx_td ltx_align_left" style="padding-left:9.0pt;padding-right:9.0pt;">Number of Programs</td>
<td id="S1.T1.2.5.2" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">N/A</td>
<td id="S1.T1.2.5.3" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">104,000</td>
<td id="S1.T1.2.5.4" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">18,356</td>
<td id="S1.T1.2.5.5" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">232,421</td>
</tr>
<tr id="S1.T1.2.6" class="ltx_tr">
<td id="S1.T1.2.6.1" class="ltx_td ltx_align_left" style="padding-left:9.0pt;padding-right:9.0pt;">Lines per Program (Avg.)</td>
<td id="S1.T1.2.6.2" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">1</td>
<td id="S1.T1.2.6.3" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">26.3</td>
<td id="S1.T1.2.6.4" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">14.7</td>
<td id="S1.T1.2.6.5" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">18.0</td>
</tr>
<tr id="S1.T1.2.7" class="ltx_tr">
<td id="S1.T1.2.7.1" class="ltx_td ltx_align_left" style="padding-left:9.0pt;padding-right:9.0pt;">Number of Exercises</td>
<td id="S1.T1.2.7.2" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">3,000</td>
<td id="S1.T1.2.7.3" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">104,000</td>
<td id="S1.T1.2.7.4" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">677</td>
<td id="S1.T1.2.7.5" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">10,000</td>
</tr>
<tr id="S1.T1.2.8" class="ltx_tr">
<td id="S1.T1.2.8.1" class="ltx_td ltx_align_left ltx_border_b" style="padding-left:9.0pt;padding-right:9.0pt;">Text Input</td>
<td id="S1.T1.2.8.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:9.0pt;padding-right:9.0pt;">Python</td>
<td id="S1.T1.2.8.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:9.0pt;padding-right:9.0pt;">Docstrings</td>
<td id="S1.T1.2.8.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:9.0pt;padding-right:9.0pt;">Pseudocode</td>
<td id="S1.T1.2.8.5" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:9.0pt;padding-right:9.0pt;">Problem Descriptions</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>A comparison of the APPS dataset to existing datasets for converting between text and code. APPS has over an order of magnitude more ground-truth solutions than these datasets, test cases, and natural language problem descriptions.</figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p"><span id="S2.p1.1.1" class="ltx_text ltx_font_bold">Program Synthesis.</span> Program synthesis is the task of generating a computer program that satisfies given specifications.
Deductive program synthesis uses formal logic specifications to define a search problem. Complex optimization techniques are used to generate programs satisfying these specifications <cite class="ltx_cite ltx_citemacro_citep">(Alur et al., <a href="#bib.bib2" title="" class="ltx_ref">2018</a>)</cite>. Because specifications must be converted into a formal language, these approaches can be rigid. Inductive synthesis from example input-output behavior can provide an alternative to formal specification <cite class="ltx_cite ltx_citemacro_citep">(Cai et al., <a href="#bib.bib6" title="" class="ltx_ref">2017</a>; Gulwani et al., <a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite>, but it is often hard to full specify behavior with examples, as any machine learning practitioner is well-aware.</p>
</div>
<div id="S2.p2" class="ltx_para ltx_noindent">
<p id="S2.p2.1" class="ltx_p">An alternative to formal or inductive specification is to specify program behavior in natural language, which prior work has considered in constrained settings.
<cite class="ltx_cite ltx_citemacro_citet">Raza et al. (<a href="#bib.bib31" title="" class="ltx_ref">2015</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Desai et al. (<a href="#bib.bib8" title="" class="ltx_ref">2016</a>)</cite> generate short programs using ad-hoc programming languages to solve specifications such as “Any 2 letters followed by any combination of 6 whole numbers.” <cite class="ltx_cite ltx_citemacro_citet">Yu et al. (<a href="#bib.bib39" title="" class="ltx_ref">2018</a>)</cite> introduce the Spider dataset for converting natural language queries into short SQL database commands. In contrast, we consider long natural language specifications and general-purpose programming languages.</p>
</div>
<div id="S2.p3" class="ltx_para ltx_noindent">
<p id="S2.p3.2" class="ltx_p"><span id="S2.p3.2.1" class="ltx_text ltx_font_bold">Code Understanding Datasets.</span> Language modeling is a compelling tool for code generation, and several works have achieved success generating code with language models in limited settings. <cite class="ltx_cite ltx_citemacro_citet">Lachaux et al. (<a href="#bib.bib18" title="" class="ltx_ref">2020</a>)</cite> use unsupervised machine translation techniques to translate functions across programming languages, attaining identical behavior after translation in many cases. <cite class="ltx_cite ltx_citemacro_citet">Kulal et al. (<a href="#bib.bib17" title="" class="ltx_ref">2019</a>)</cite> introduce SPoC, a method for converting pseudocode to code utilizing seq2seq machine translation with an additional search step. To train SPoC, they collect line-by-line descriptions of C++ programs using Amazon Mechanical Turk. Recently, <cite class="ltx_cite ltx_citemacro_citet">Lu et al. (<a href="#bib.bib23" title="" class="ltx_ref">2021</a>)</cite> introduce the CodeXGLUE benchmark which aggregates various previous benchmarks and use CodeBLEU <cite class="ltx_cite ltx_citemacro_citep">(Ren et al., <a href="#bib.bib32" title="" class="ltx_ref">2020</a>)</cite> and CONCODE. <cite class="ltx_cite ltx_citemacro_citet">Iyer et al. (<a href="#bib.bib16" title="" class="ltx_ref">2018</a>)</cite> investigate generating Java code from docstrings and evaluate performance with BLEU. The docstrings are often incomplete specifications of what should be coded and only <math id="S2.p3.1.m1.1" class="ltx_Math" alttext="14.7" display="inline"><semantics id="S2.p3.1.m1.1a"><mn id="S2.p3.1.m1.1.1" xref="S2.p3.1.m1.1.1.cmml">14.7</mn><annotation-xml encoding="MathML-Content" id="S2.p3.1.m1.1b"><cn type="float" id="S2.p3.1.m1.1.1.cmml" xref="S2.p3.1.m1.1.1">14.7</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.1.m1.1c">14.7</annotation></semantics></math> words long on average, e.g. “Convert mixed case to underscores.” By comparison, problem specifications in our new APPS benchmark are self-contained and have a much larger average length of <math id="S2.p3.2.m2.1" class="ltx_Math" alttext="293.2" display="inline"><semantics id="S2.p3.2.m2.1a"><mn id="S2.p3.2.m2.1.1" xref="S2.p3.2.m2.1.1.cmml">293.2</mn><annotation-xml encoding="MathML-Content" id="S2.p3.2.m2.1b"><cn type="float" id="S2.p3.2.m2.1.1.cmml" xref="S2.p3.2.m2.1.1">293.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.2.m2.1c">293.2</annotation></semantics></math> words. Unlike <cite class="ltx_cite ltx_citemacro_citet">Iyer et al. (<a href="#bib.bib16" title="" class="ltx_ref">2018</a>)</cite>, APPS contains test cases for every exercise, enabling a high-quality evaluation of code correctness. Further comparisons are in the Appendix.</p>
</div>
<div id="S2.p4" class="ltx_para ltx_noindent">
<p id="S2.p4.1" class="ltx_p"><span id="S2.p4.1.1" class="ltx_text ltx_font_bold">Evaluating Large-Scale Language Models.</span> Modern large-scale language models have demonstrated impressive capabilities across a variety of text-based tasks. On the SuperGLUE benchmark <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib36" title="" class="ltx_ref">2019b</a>)</cite>, some models now exceed human performance. On many commonsense reasoning benchmarks, performance is rising quickly <cite class="ltx_cite ltx_citemacro_citep">(Zellers et al., <a href="#bib.bib42" title="" class="ltx_ref">2019</a>; Huang et al., <a href="#bib.bib15" title="" class="ltx_ref">2019</a>; Bisk et al., <a href="#bib.bib3" title="" class="ltx_ref">2019</a>)</cite>. Even when language models are evaluated across diverse technical areas such as law and medicine, performance is surprisingly high and poised to improve as models are scaled up further <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib13" title="" class="ltx_ref">2021b</a>)</cite>. With rapid improvements across numerous datasets, finding resilient benchmarks on which models significantly underperform humans is challenging. APPS represents an attempt to fill this gap and cleanly separate model performance from that of expert humans.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>The APPS Dataset</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.1" class="ltx_p">The APPS dataset consists of problems collected from different open-access coding websites such as Codeforces, Kattis, and more. The APPS benchmark attempts to mirror how humans programmers are evaluated by posing coding problems in unrestricted natural language and using test cases to evaluate solution correctness. The problems range in difficulty from introductory to collegiate competition level and measure coding and problem-solving ability.</p>
</div>
<div id="S3.p2" class="ltx_para ltx_noindent">
<p id="S3.p2.6" class="ltx_p">The Automated Programming Progress Standard, abbreviated APPS, consists of <math id="S3.p2.1.m1.2" class="ltx_Math" alttext="10,\!000" display="inline"><semantics id="S3.p2.1.m1.2a"><mrow id="S3.p2.1.m1.2.3.2" xref="S3.p2.1.m1.2.3.1.cmml"><mn id="S3.p2.1.m1.1.1" xref="S3.p2.1.m1.1.1.cmml">10</mn><mpadded width="0.275em"><mo id="S3.p2.1.m1.2.3.2.1" xref="S3.p2.1.m1.2.3.1.cmml">,</mo></mpadded><mn id="S3.p2.1.m1.2.2" xref="S3.p2.1.m1.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.1.m1.2b"><list id="S3.p2.1.m1.2.3.1.cmml" xref="S3.p2.1.m1.2.3.2"><cn type="integer" id="S3.p2.1.m1.1.1.cmml" xref="S3.p2.1.m1.1.1">10</cn><cn type="integer" id="S3.p2.1.m1.2.2.cmml" xref="S3.p2.1.m1.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.1.m1.2c">10,\!000</annotation></semantics></math> coding problems in total, with <math id="S3.p2.2.m2.2" class="ltx_Math" alttext="131,\!777" display="inline"><semantics id="S3.p2.2.m2.2a"><mrow id="S3.p2.2.m2.2.3.2" xref="S3.p2.2.m2.2.3.1.cmml"><mn id="S3.p2.2.m2.1.1" xref="S3.p2.2.m2.1.1.cmml">131</mn><mpadded width="0.275em"><mo id="S3.p2.2.m2.2.3.2.1" xref="S3.p2.2.m2.2.3.1.cmml">,</mo></mpadded><mn id="S3.p2.2.m2.2.2" xref="S3.p2.2.m2.2.2.cmml">777</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.2.m2.2b"><list id="S3.p2.2.m2.2.3.1.cmml" xref="S3.p2.2.m2.2.3.2"><cn type="integer" id="S3.p2.2.m2.1.1.cmml" xref="S3.p2.2.m2.1.1">131</cn><cn type="integer" id="S3.p2.2.m2.2.2.cmml" xref="S3.p2.2.m2.2.2">777</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.2.m2.2c">131,\!777</annotation></semantics></math> test cases for checking solutions and <math id="S3.p2.3.m3.2" class="ltx_Math" alttext="232,\!421" display="inline"><semantics id="S3.p2.3.m3.2a"><mrow id="S3.p2.3.m3.2.3.2" xref="S3.p2.3.m3.2.3.1.cmml"><mn id="S3.p2.3.m3.1.1" xref="S3.p2.3.m3.1.1.cmml">232</mn><mpadded width="0.275em"><mo id="S3.p2.3.m3.2.3.2.1" xref="S3.p2.3.m3.2.3.1.cmml">,</mo></mpadded><mn id="S3.p2.3.m3.2.2" xref="S3.p2.3.m3.2.2.cmml">421</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.3.m3.2b"><list id="S3.p2.3.m3.2.3.1.cmml" xref="S3.p2.3.m3.2.3.2"><cn type="integer" id="S3.p2.3.m3.1.1.cmml" xref="S3.p2.3.m3.1.1">232</cn><cn type="integer" id="S3.p2.3.m3.2.2.cmml" xref="S3.p2.3.m3.2.2">421</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.3.m3.2c">232,\!421</annotation></semantics></math> ground-truth solutions written by humans. Problems can be complicated, as the average length of a problem is <math id="S3.p2.4.m4.1" class="ltx_Math" alttext="293.2" display="inline"><semantics id="S3.p2.4.m4.1a"><mn id="S3.p2.4.m4.1.1" xref="S3.p2.4.m4.1.1.cmml">293.2</mn><annotation-xml encoding="MathML-Content" id="S3.p2.4.m4.1b"><cn type="float" id="S3.p2.4.m4.1.1.cmml" xref="S3.p2.4.m4.1.1">293.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.4.m4.1c">293.2</annotation></semantics></math> words.
The data are split evenly into training and test sets, with <math id="S3.p2.5.m5.2" class="ltx_Math" alttext="5,\!000" display="inline"><semantics id="S3.p2.5.m5.2a"><mrow id="S3.p2.5.m5.2.3.2" xref="S3.p2.5.m5.2.3.1.cmml"><mn id="S3.p2.5.m5.1.1" xref="S3.p2.5.m5.1.1.cmml">5</mn><mpadded width="0.275em"><mo id="S3.p2.5.m5.2.3.2.1" xref="S3.p2.5.m5.2.3.1.cmml">,</mo></mpadded><mn id="S3.p2.5.m5.2.2" xref="S3.p2.5.m5.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.5.m5.2b"><list id="S3.p2.5.m5.2.3.1.cmml" xref="S3.p2.5.m5.2.3.2"><cn type="integer" id="S3.p2.5.m5.1.1.cmml" xref="S3.p2.5.m5.1.1">5</cn><cn type="integer" id="S3.p2.5.m5.2.2.cmml" xref="S3.p2.5.m5.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.5.m5.2c">5,\!000</annotation></semantics></math> problems each. In the test set, every problem has multiple test cases, and the average number of test cases is <math id="S3.p2.6.m6.1" class="ltx_Math" alttext="21.2" display="inline"><semantics id="S3.p2.6.m6.1a"><mn id="S3.p2.6.m6.1.1" xref="S3.p2.6.m6.1.1.cmml">21.2</mn><annotation-xml encoding="MathML-Content" id="S3.p2.6.m6.1b"><cn type="float" id="S3.p2.6.m6.1.1.cmml" xref="S3.p2.6.m6.1.1">21.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.6.m6.1c">21.2</annotation></semantics></math>. Each test case is specifically designed for the corresponding problem, enabling us to rigorously evaluate program functionality.</p>
</div>
<div id="S3.p3" class="ltx_para ltx_noindent">
<p id="S3.p3.1" class="ltx_p"><span id="S3.p3.1.1" class="ltx_text ltx_font_bold">Dataset Construction.</span> To create the APPS dataset, we manually curate problems from open-access sites where programmers share problems with each other, including Codewars, AtCoder, Kattis, and Codeforces. Problems are posed as natural language specifications of what should be coded, and they come in various formats. To improve quality and consistency, we wrote custom HTML parsers for each source of problems, which allows us to properly format LaTeX expressions, lists, and sections in the question text. Where necessary, we convert equation images to LaTeX using the MathPix API, and we remove problems that rely on image figures. We also perform deduplication using tf-idf features with SVD dimensionality reduction and cosine similarity. Several graduate and undergraduate student authors polished and refined this dataset over the course of six months, ensuring a high-quality set of problems.</p>
</div>
<div id="S3.p4" class="ltx_para ltx_noindent">
<p id="S3.p4.1" class="ltx_p">Executing and evaluating arbitrary Python code is challenging. On the websites we source data from, human solutions are allowed to run arbitrary code, including import statements for common modules and libraries. To handle this, each website implements a custom judging system for solutions. We design a testing framework with this in mind, which merges the judging functionality of several websites. We also standardize the format of test cases. The end result is that solutions are allowed to execute arbitrary Python code, and the results are compared against test cases for a given problem.</p>
</div>
<div id="S3.p5" class="ltx_para ltx_noindent">
<p id="S3.p5.4" class="ltx_p"><span id="S3.p5.4.1" class="ltx_text ltx_font_bold">Dataset Difficulty.</span> Each of our problem sources uses a separate scale for measuring difficulty. We place problems from these different sources into three categories. For example, problems from Kattis with difficulty less than <math id="S3.p5.1.m1.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S3.p5.1.m1.1a"><mn id="S3.p5.1.m1.1.1" xref="S3.p5.1.m1.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S3.p5.1.m1.1b"><cn type="integer" id="S3.p5.1.m1.1.1.cmml" xref="S3.p5.1.m1.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p5.1.m1.1c">3</annotation></semantics></math> are categorized as “introductory,” problems with difficulty between <math id="S3.p5.2.m2.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S3.p5.2.m2.1a"><mn id="S3.p5.2.m2.1.1" xref="S3.p5.2.m2.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S3.p5.2.m2.1b"><cn type="integer" id="S3.p5.2.m2.1.1.cmml" xref="S3.p5.2.m2.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p5.2.m2.1c">3</annotation></semantics></math> and <math id="S3.p5.3.m3.1" class="ltx_Math" alttext="5" display="inline"><semantics id="S3.p5.3.m3.1a"><mn id="S3.p5.3.m3.1.1" xref="S3.p5.3.m3.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="S3.p5.3.m3.1b"><cn type="integer" id="S3.p5.3.m3.1.1.cmml" xref="S3.p5.3.m3.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p5.3.m3.1c">5</annotation></semantics></math> as “interview,” and problems with difficulty greater than <math id="S3.p5.4.m4.1" class="ltx_Math" alttext="5" display="inline"><semantics id="S3.p5.4.m4.1a"><mn id="S3.p5.4.m4.1.1" xref="S3.p5.4.m4.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="S3.p5.4.m4.1b"><cn type="integer" id="S3.p5.4.m4.1.1.cmml" xref="S3.p5.4.m4.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p5.4.m4.1c">5</annotation></semantics></math> as “competition.”</p>
</div>
<div id="S3.p6" class="ltx_para ltx_noindent">
<ol id="S3.I1" class="ltx_enumerate">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.2" class="ltx_p"><span id="S3.I1.i1.p1.2.1" class="ltx_text ltx_font_bold">Introductory Level</span>. These are problems that most programmers with 1-2 years of experience can answer without requiring complicated algorithms. Examples of such problems include counting the number of vowels in a string, or returning the running sum of a list of integers. There are <math id="S3.I1.i1.p1.1.m1.2" class="ltx_Math" alttext="3,\!639" display="inline"><semantics id="S3.I1.i1.p1.1.m1.2a"><mrow id="S3.I1.i1.p1.1.m1.2.3.2" xref="S3.I1.i1.p1.1.m1.2.3.1.cmml"><mn id="S3.I1.i1.p1.1.m1.1.1" xref="S3.I1.i1.p1.1.m1.1.1.cmml">3</mn><mpadded width="0.275em"><mo id="S3.I1.i1.p1.1.m1.2.3.2.1" xref="S3.I1.i1.p1.1.m1.2.3.1.cmml">,</mo></mpadded><mn id="S3.I1.i1.p1.1.m1.2.2" xref="S3.I1.i1.p1.1.m1.2.2.cmml">639</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.1.m1.2b"><list id="S3.I1.i1.p1.1.m1.2.3.1.cmml" xref="S3.I1.i1.p1.1.m1.2.3.2"><cn type="integer" id="S3.I1.i1.p1.1.m1.1.1.cmml" xref="S3.I1.i1.p1.1.m1.1.1">3</cn><cn type="integer" id="S3.I1.i1.p1.1.m1.2.2.cmml" xref="S3.I1.i1.p1.1.m1.2.2">639</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.1.m1.2c">3,\!639</annotation></semantics></math> problems classified as introductory level and <math id="S3.I1.i1.p1.2.m2.2" class="ltx_Math" alttext="1,\!000" display="inline"><semantics id="S3.I1.i1.p1.2.m2.2a"><mrow id="S3.I1.i1.p1.2.m2.2.3.2" xref="S3.I1.i1.p1.2.m2.2.3.1.cmml"><mn id="S3.I1.i1.p1.2.m2.1.1" xref="S3.I1.i1.p1.2.m2.1.1.cmml">1</mn><mpadded width="0.275em"><mo id="S3.I1.i1.p1.2.m2.2.3.2.1" xref="S3.I1.i1.p1.2.m2.2.3.1.cmml">,</mo></mpadded><mn id="S3.I1.i1.p1.2.m2.2.2" xref="S3.I1.i1.p1.2.m2.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.2.m2.2b"><list id="S3.I1.i1.p1.2.m2.2.3.1.cmml" xref="S3.I1.i1.p1.2.m2.2.3.2"><cn type="integer" id="S3.I1.i1.p1.2.m2.1.1.cmml" xref="S3.I1.i1.p1.2.m2.1.1">1</cn><cn type="integer" id="S3.I1.i1.p1.2.m2.2.2.cmml" xref="S3.I1.i1.p1.2.m2.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.2.m2.2c">1,\!000</annotation></semantics></math> in the test set.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.2" class="ltx_p"><span id="S3.I1.i2.p1.2.1" class="ltx_text ltx_font_bold">Interview Level</span>. These are problems that are more algorithmic and difficult in nature and would be at the level of questions asked in programming technical interviews. Examples of such problems might include those involving data structures such as trees or graphs, or problems that requiring nontrivial algorithms. There are <math id="S3.I1.i2.p1.1.m1.2" class="ltx_Math" alttext="5,\!000" display="inline"><semantics id="S3.I1.i2.p1.1.m1.2a"><mrow id="S3.I1.i2.p1.1.m1.2.3.2" xref="S3.I1.i2.p1.1.m1.2.3.1.cmml"><mn id="S3.I1.i2.p1.1.m1.1.1" xref="S3.I1.i2.p1.1.m1.1.1.cmml">5</mn><mpadded width="0.275em"><mo id="S3.I1.i2.p1.1.m1.2.3.2.1" xref="S3.I1.i2.p1.1.m1.2.3.1.cmml">,</mo></mpadded><mn id="S3.I1.i2.p1.1.m1.2.2" xref="S3.I1.i2.p1.1.m1.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.1.m1.2b"><list id="S3.I1.i2.p1.1.m1.2.3.1.cmml" xref="S3.I1.i2.p1.1.m1.2.3.2"><cn type="integer" id="S3.I1.i2.p1.1.m1.1.1.cmml" xref="S3.I1.i2.p1.1.m1.1.1">5</cn><cn type="integer" id="S3.I1.i2.p1.1.m1.2.2.cmml" xref="S3.I1.i2.p1.1.m1.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.1.m1.2c">5,\!000</annotation></semantics></math> problems classified as interview level and <math id="S3.I1.i2.p1.2.m2.2" class="ltx_Math" alttext="3,\!000" display="inline"><semantics id="S3.I1.i2.p1.2.m2.2a"><mrow id="S3.I1.i2.p1.2.m2.2.3.2" xref="S3.I1.i2.p1.2.m2.2.3.1.cmml"><mn id="S3.I1.i2.p1.2.m2.1.1" xref="S3.I1.i2.p1.2.m2.1.1.cmml">3</mn><mpadded width="0.275em"><mo id="S3.I1.i2.p1.2.m2.2.3.2.1" xref="S3.I1.i2.p1.2.m2.2.3.1.cmml">,</mo></mpadded><mn id="S3.I1.i2.p1.2.m2.2.2" xref="S3.I1.i2.p1.2.m2.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.2.m2.2b"><list id="S3.I1.i2.p1.2.m2.2.3.1.cmml" xref="S3.I1.i2.p1.2.m2.2.3.2"><cn type="integer" id="S3.I1.i2.p1.2.m2.1.1.cmml" xref="S3.I1.i2.p1.2.m2.1.1">3</cn><cn type="integer" id="S3.I1.i2.p1.2.m2.2.2.cmml" xref="S3.I1.i2.p1.2.m2.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.2.m2.2c">3,\!000</annotation></semantics></math> in the test set.</p>
</div>
</li>
<li id="S3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S3.I1.i3.p1" class="ltx_para ltx_noindent">
<p id="S3.I1.i3.p1.2" class="ltx_p"><span id="S3.I1.i3.p1.2.1" class="ltx_text ltx_font_bold">Competition Level</span>. These are problems are the most challenging and are at the level of the most advanced high school and collegiate programming competitions, including USACO, IOI, and ACM. There are <math id="S3.I1.i3.p1.1.m1.2" class="ltx_Math" alttext="1,\!361" display="inline"><semantics id="S3.I1.i3.p1.1.m1.2a"><mrow id="S3.I1.i3.p1.1.m1.2.3.2" xref="S3.I1.i3.p1.1.m1.2.3.1.cmml"><mn id="S3.I1.i3.p1.1.m1.1.1" xref="S3.I1.i3.p1.1.m1.1.1.cmml">1</mn><mpadded width="0.275em"><mo id="S3.I1.i3.p1.1.m1.2.3.2.1" xref="S3.I1.i3.p1.1.m1.2.3.1.cmml">,</mo></mpadded><mn id="S3.I1.i3.p1.1.m1.2.2" xref="S3.I1.i3.p1.1.m1.2.2.cmml">361</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i3.p1.1.m1.2b"><list id="S3.I1.i3.p1.1.m1.2.3.1.cmml" xref="S3.I1.i3.p1.1.m1.2.3.2"><cn type="integer" id="S3.I1.i3.p1.1.m1.1.1.cmml" xref="S3.I1.i3.p1.1.m1.1.1">1</cn><cn type="integer" id="S3.I1.i3.p1.1.m1.2.2.cmml" xref="S3.I1.i3.p1.1.m1.2.2">361</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i3.p1.1.m1.2c">1,\!361</annotation></semantics></math> competition level problems and <math id="S3.I1.i3.p1.2.m2.2" class="ltx_Math" alttext="1,\!000" display="inline"><semantics id="S3.I1.i3.p1.2.m2.2a"><mrow id="S3.I1.i3.p1.2.m2.2.3.2" xref="S3.I1.i3.p1.2.m2.2.3.1.cmml"><mn id="S3.I1.i3.p1.2.m2.1.1" xref="S3.I1.i3.p1.2.m2.1.1.cmml">1</mn><mpadded width="0.275em"><mo id="S3.I1.i3.p1.2.m2.2.3.2.1" xref="S3.I1.i3.p1.2.m2.2.3.1.cmml">,</mo></mpadded><mn id="S3.I1.i3.p1.2.m2.2.2" xref="S3.I1.i3.p1.2.m2.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i3.p1.2.m2.2b"><list id="S3.I1.i3.p1.2.m2.2.3.1.cmml" xref="S3.I1.i3.p1.2.m2.2.3.2"><cn type="integer" id="S3.I1.i3.p1.2.m2.1.1.cmml" xref="S3.I1.i3.p1.2.m2.1.1">1</cn><cn type="integer" id="S3.I1.i3.p1.2.m2.2.2.cmml" xref="S3.I1.i3.p1.2.m2.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i3.p1.2.m2.2c">1,\!000</annotation></semantics></math> in the test set.</p>
</div>
</li>
</ol>
</div>
<div id="S3.p7" class="ltx_para ltx_noindent">
<p id="S3.p7.1" class="ltx_p"><span id="S3.p7.1.1" class="ltx_text ltx_font_bold">Problem Formats.</span> To accommodate a broad range of problem sources, problems in APPS come in two formats.</p>
<ul id="S3.I2" class="ltx_itemize">
<li id="S3.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i1.p1" class="ltx_para">
<p id="S3.I2.i1.p1.1" class="ltx_p"><span id="S3.I2.i1.p1.1.1" class="ltx_text ltx_font_bold">Call-Based Format</span> problems generally provide initial starter code, usually in the form of a function header, and ask for the solution to be provided as the function’s return value.</p>
</div>
</li>
<li id="S3.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i2.p1" class="ltx_para ltx_noindent">
<p id="S3.I2.i2.p1.1" class="ltx_p"><span id="S3.I2.i2.p1.1.1" class="ltx_text ltx_font_bold">Standard Input Format</span> problems generally lack starter code. Instead, the model is only provided with the problem and must output its answers to the STDOUT stream, such as by using print statements.</p>
</div>
</li>
</ul>
</div>
<div id="S3.p8" class="ltx_para ltx_noindent">
<p id="S3.p8.1" class="ltx_p">For the call-based format problems, we prompt models using the following inputs:</p>
</div>
<div id="S3.p9" class="ltx_para ltx_noindent">
<span id="S3.p9.1" class="ltx_ERROR undefined">{spverbatim}</span>
<p id="S3.p9.2" class="ltx_p">"<span id="S3.p9.2.1" class="ltx_ERROR undefined">\nQUESTION</span>:<span id="S3.p9.2.2" class="ltx_ERROR undefined">\n</span>" + q_str + "<span id="S3.p9.2.3" class="ltx_ERROR undefined">\n</span>" + starter_code_str + "<span id="S3.p9.2.4" class="ltx_ERROR undefined">\n</span>" + "<span id="S3.p9.2.5" class="ltx_ERROR undefined">\nUse</span>Call-Based Format<span id="S3.p9.2.6" class="ltx_ERROR undefined">\n</span><span id="S3.p9.2.7" class="ltx_ERROR undefined">\nANSWER</span>:<span id="S3.p9.2.8" class="ltx_ERROR undefined">\n</span>"</p>
</div>
<div id="S3.p10" class="ltx_para ltx_noindent">
<p id="S3.p10.1" class="ltx_p">For the above prompt, the variable <code id="S3.p10.1.1" class="ltx_verbatim ltx_font_typewriter">q_str</code> represents the raw text of the problem statement. The variable <code id="S3.p10.1.2" class="ltx_verbatim ltx_font_typewriter">starter_code_str</code> represents the starter code given in the problem definition, or the empty string if no starter code was provided. For the standard input format problems, we prompt the model with the input string as before, but we replace “Call-Based Format” with “Standard Input Format.” Note that if starter code is given, it is only part of the input. This means that to use the starter code, a model must learn to copy the starter code at the beginning of its outputted answer in order to get the question correct. We find that fine-tuned models are able to do this without difficulty.</p>
</div>
<figure id="S3.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S3.F2.23" class="ltx_p ltx_figure_panel"><span id="S3.F2.23.1" class="ltx_text ltx_font_bold">Problem</span> 
<br class="ltx_break">You are given a string <math id="S3.F2.1.m1.1" class="ltx_Math" alttext="s=s_{1}s_{2}\dots s_{n}" display="inline"><semantics id="S3.F2.1.m1.1a"><mrow id="S3.F2.1.m1.1.1" xref="S3.F2.1.m1.1.1.cmml"><mi id="S3.F2.1.m1.1.1.2" xref="S3.F2.1.m1.1.1.2.cmml">s</mi><mo id="S3.F2.1.m1.1.1.1" xref="S3.F2.1.m1.1.1.1.cmml">=</mo><mrow id="S3.F2.1.m1.1.1.3" xref="S3.F2.1.m1.1.1.3.cmml"><msub id="S3.F2.1.m1.1.1.3.2" xref="S3.F2.1.m1.1.1.3.2.cmml"><mi id="S3.F2.1.m1.1.1.3.2.2" xref="S3.F2.1.m1.1.1.3.2.2.cmml">s</mi><mn id="S3.F2.1.m1.1.1.3.2.3" xref="S3.F2.1.m1.1.1.3.2.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S3.F2.1.m1.1.1.3.1" xref="S3.F2.1.m1.1.1.3.1.cmml">​</mo><msub id="S3.F2.1.m1.1.1.3.3" xref="S3.F2.1.m1.1.1.3.3.cmml"><mi id="S3.F2.1.m1.1.1.3.3.2" xref="S3.F2.1.m1.1.1.3.3.2.cmml">s</mi><mn id="S3.F2.1.m1.1.1.3.3.3" xref="S3.F2.1.m1.1.1.3.3.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="S3.F2.1.m1.1.1.3.1a" xref="S3.F2.1.m1.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="S3.F2.1.m1.1.1.3.4" xref="S3.F2.1.m1.1.1.3.4.cmml">…</mi><mo lspace="0em" rspace="0em" id="S3.F2.1.m1.1.1.3.1b" xref="S3.F2.1.m1.1.1.3.1.cmml">​</mo><msub id="S3.F2.1.m1.1.1.3.5" xref="S3.F2.1.m1.1.1.3.5.cmml"><mi id="S3.F2.1.m1.1.1.3.5.2" xref="S3.F2.1.m1.1.1.3.5.2.cmml">s</mi><mi id="S3.F2.1.m1.1.1.3.5.3" xref="S3.F2.1.m1.1.1.3.5.3.cmml">n</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.F2.1.m1.1b"><apply id="S3.F2.1.m1.1.1.cmml" xref="S3.F2.1.m1.1.1"><eq id="S3.F2.1.m1.1.1.1.cmml" xref="S3.F2.1.m1.1.1.1"></eq><ci id="S3.F2.1.m1.1.1.2.cmml" xref="S3.F2.1.m1.1.1.2">𝑠</ci><apply id="S3.F2.1.m1.1.1.3.cmml" xref="S3.F2.1.m1.1.1.3"><times id="S3.F2.1.m1.1.1.3.1.cmml" xref="S3.F2.1.m1.1.1.3.1"></times><apply id="S3.F2.1.m1.1.1.3.2.cmml" xref="S3.F2.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="S3.F2.1.m1.1.1.3.2.1.cmml" xref="S3.F2.1.m1.1.1.3.2">subscript</csymbol><ci id="S3.F2.1.m1.1.1.3.2.2.cmml" xref="S3.F2.1.m1.1.1.3.2.2">𝑠</ci><cn type="integer" id="S3.F2.1.m1.1.1.3.2.3.cmml" xref="S3.F2.1.m1.1.1.3.2.3">1</cn></apply><apply id="S3.F2.1.m1.1.1.3.3.cmml" xref="S3.F2.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="S3.F2.1.m1.1.1.3.3.1.cmml" xref="S3.F2.1.m1.1.1.3.3">subscript</csymbol><ci id="S3.F2.1.m1.1.1.3.3.2.cmml" xref="S3.F2.1.m1.1.1.3.3.2">𝑠</ci><cn type="integer" id="S3.F2.1.m1.1.1.3.3.3.cmml" xref="S3.F2.1.m1.1.1.3.3.3">2</cn></apply><ci id="S3.F2.1.m1.1.1.3.4.cmml" xref="S3.F2.1.m1.1.1.3.4">…</ci><apply id="S3.F2.1.m1.1.1.3.5.cmml" xref="S3.F2.1.m1.1.1.3.5"><csymbol cd="ambiguous" id="S3.F2.1.m1.1.1.3.5.1.cmml" xref="S3.F2.1.m1.1.1.3.5">subscript</csymbol><ci id="S3.F2.1.m1.1.1.3.5.2.cmml" xref="S3.F2.1.m1.1.1.3.5.2">𝑠</ci><ci id="S3.F2.1.m1.1.1.3.5.3.cmml" xref="S3.F2.1.m1.1.1.3.5.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.1.m1.1c">s=s_{1}s_{2}\dots s_{n}</annotation></semantics></math> of length <math id="S3.F2.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.F2.2.m2.1a"><mi id="S3.F2.2.m2.1.1" xref="S3.F2.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.F2.2.m2.1b"><ci id="S3.F2.2.m2.1.1.cmml" xref="S3.F2.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.2.m2.1c">n</annotation></semantics></math>, which only contains digits <math id="S3.F2.3.m3.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.F2.3.m3.1a"><mn id="S3.F2.3.m3.1.1" xref="S3.F2.3.m3.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.F2.3.m3.1b"><cn type="integer" id="S3.F2.3.m3.1.1.cmml" xref="S3.F2.3.m3.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.3.m3.1c">1</annotation></semantics></math>, <math id="S3.F2.4.m4.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S3.F2.4.m4.1a"><mn id="S3.F2.4.m4.1.1" xref="S3.F2.4.m4.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S3.F2.4.m4.1b"><cn type="integer" id="S3.F2.4.m4.1.1.cmml" xref="S3.F2.4.m4.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.4.m4.1c">2</annotation></semantics></math>,…, <math id="S3.F2.5.m5.1" class="ltx_Math" alttext="9" display="inline"><semantics id="S3.F2.5.m5.1a"><mn id="S3.F2.5.m5.1.1" xref="S3.F2.5.m5.1.1.cmml">9</mn><annotation-xml encoding="MathML-Content" id="S3.F2.5.m5.1b"><cn type="integer" id="S3.F2.5.m5.1.1.cmml" xref="S3.F2.5.m5.1.1">9</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.5.m5.1c">9</annotation></semantics></math>. A substring <math id="S3.F2.6.m6.1" class="ltx_Math" alttext="s[l\dots r]" display="inline"><semantics id="S3.F2.6.m6.1a"><mrow id="S3.F2.6.m6.1.1" xref="S3.F2.6.m6.1.1.cmml"><mi id="S3.F2.6.m6.1.1.3" xref="S3.F2.6.m6.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.F2.6.m6.1.1.2" xref="S3.F2.6.m6.1.1.2.cmml">​</mo><mrow id="S3.F2.6.m6.1.1.1.1" xref="S3.F2.6.m6.1.1.1.2.cmml"><mo stretchy="false" id="S3.F2.6.m6.1.1.1.1.2" xref="S3.F2.6.m6.1.1.1.2.1.cmml">[</mo><mrow id="S3.F2.6.m6.1.1.1.1.1" xref="S3.F2.6.m6.1.1.1.1.1.cmml"><mi id="S3.F2.6.m6.1.1.1.1.1.2" xref="S3.F2.6.m6.1.1.1.1.1.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.F2.6.m6.1.1.1.1.1.1" xref="S3.F2.6.m6.1.1.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="S3.F2.6.m6.1.1.1.1.1.3" xref="S3.F2.6.m6.1.1.1.1.1.3.cmml">…</mi><mo lspace="0em" rspace="0em" id="S3.F2.6.m6.1.1.1.1.1.1a" xref="S3.F2.6.m6.1.1.1.1.1.1.cmml">​</mo><mi id="S3.F2.6.m6.1.1.1.1.1.4" xref="S3.F2.6.m6.1.1.1.1.1.4.cmml">r</mi></mrow><mo stretchy="false" id="S3.F2.6.m6.1.1.1.1.3" xref="S3.F2.6.m6.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.F2.6.m6.1b"><apply id="S3.F2.6.m6.1.1.cmml" xref="S3.F2.6.m6.1.1"><times id="S3.F2.6.m6.1.1.2.cmml" xref="S3.F2.6.m6.1.1.2"></times><ci id="S3.F2.6.m6.1.1.3.cmml" xref="S3.F2.6.m6.1.1.3">𝑠</ci><apply id="S3.F2.6.m6.1.1.1.2.cmml" xref="S3.F2.6.m6.1.1.1.1"><csymbol cd="latexml" id="S3.F2.6.m6.1.1.1.2.1.cmml" xref="S3.F2.6.m6.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.F2.6.m6.1.1.1.1.1.cmml" xref="S3.F2.6.m6.1.1.1.1.1"><times id="S3.F2.6.m6.1.1.1.1.1.1.cmml" xref="S3.F2.6.m6.1.1.1.1.1.1"></times><ci id="S3.F2.6.m6.1.1.1.1.1.2.cmml" xref="S3.F2.6.m6.1.1.1.1.1.2">𝑙</ci><ci id="S3.F2.6.m6.1.1.1.1.1.3.cmml" xref="S3.F2.6.m6.1.1.1.1.1.3">…</ci><ci id="S3.F2.6.m6.1.1.1.1.1.4.cmml" xref="S3.F2.6.m6.1.1.1.1.1.4">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.6.m6.1c">s[l\dots r]</annotation></semantics></math> of <math id="S3.F2.7.m7.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.F2.7.m7.1a"><mi id="S3.F2.7.m7.1.1" xref="S3.F2.7.m7.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.F2.7.m7.1b"><ci id="S3.F2.7.m7.1.1.cmml" xref="S3.F2.7.m7.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.7.m7.1c">s</annotation></semantics></math> is a string <math id="S3.F2.8.m8.1" class="ltx_Math" alttext="s_{l}s_{l+1}s_{l+2}\ldots s_{r}" display="inline"><semantics id="S3.F2.8.m8.1a"><mrow id="S3.F2.8.m8.1.1" xref="S3.F2.8.m8.1.1.cmml"><msub id="S3.F2.8.m8.1.1.2" xref="S3.F2.8.m8.1.1.2.cmml"><mi id="S3.F2.8.m8.1.1.2.2" xref="S3.F2.8.m8.1.1.2.2.cmml">s</mi><mi id="S3.F2.8.m8.1.1.2.3" xref="S3.F2.8.m8.1.1.2.3.cmml">l</mi></msub><mo lspace="0em" rspace="0em" id="S3.F2.8.m8.1.1.1" xref="S3.F2.8.m8.1.1.1.cmml">​</mo><msub id="S3.F2.8.m8.1.1.3" xref="S3.F2.8.m8.1.1.3.cmml"><mi id="S3.F2.8.m8.1.1.3.2" xref="S3.F2.8.m8.1.1.3.2.cmml">s</mi><mrow id="S3.F2.8.m8.1.1.3.3" xref="S3.F2.8.m8.1.1.3.3.cmml"><mi id="S3.F2.8.m8.1.1.3.3.2" xref="S3.F2.8.m8.1.1.3.3.2.cmml">l</mi><mo id="S3.F2.8.m8.1.1.3.3.1" xref="S3.F2.8.m8.1.1.3.3.1.cmml">+</mo><mn id="S3.F2.8.m8.1.1.3.3.3" xref="S3.F2.8.m8.1.1.3.3.3.cmml">1</mn></mrow></msub><mo lspace="0em" rspace="0em" id="S3.F2.8.m8.1.1.1a" xref="S3.F2.8.m8.1.1.1.cmml">​</mo><msub id="S3.F2.8.m8.1.1.4" xref="S3.F2.8.m8.1.1.4.cmml"><mi id="S3.F2.8.m8.1.1.4.2" xref="S3.F2.8.m8.1.1.4.2.cmml">s</mi><mrow id="S3.F2.8.m8.1.1.4.3" xref="S3.F2.8.m8.1.1.4.3.cmml"><mi id="S3.F2.8.m8.1.1.4.3.2" xref="S3.F2.8.m8.1.1.4.3.2.cmml">l</mi><mo id="S3.F2.8.m8.1.1.4.3.1" xref="S3.F2.8.m8.1.1.4.3.1.cmml">+</mo><mn id="S3.F2.8.m8.1.1.4.3.3" xref="S3.F2.8.m8.1.1.4.3.3.cmml">2</mn></mrow></msub><mo lspace="0em" rspace="0em" id="S3.F2.8.m8.1.1.1b" xref="S3.F2.8.m8.1.1.1.cmml">​</mo><mi mathvariant="normal" id="S3.F2.8.m8.1.1.5" xref="S3.F2.8.m8.1.1.5.cmml">…</mi><mo lspace="0em" rspace="0em" id="S3.F2.8.m8.1.1.1c" xref="S3.F2.8.m8.1.1.1.cmml">​</mo><msub id="S3.F2.8.m8.1.1.6" xref="S3.F2.8.m8.1.1.6.cmml"><mi id="S3.F2.8.m8.1.1.6.2" xref="S3.F2.8.m8.1.1.6.2.cmml">s</mi><mi id="S3.F2.8.m8.1.1.6.3" xref="S3.F2.8.m8.1.1.6.3.cmml">r</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.F2.8.m8.1b"><apply id="S3.F2.8.m8.1.1.cmml" xref="S3.F2.8.m8.1.1"><times id="S3.F2.8.m8.1.1.1.cmml" xref="S3.F2.8.m8.1.1.1"></times><apply id="S3.F2.8.m8.1.1.2.cmml" xref="S3.F2.8.m8.1.1.2"><csymbol cd="ambiguous" id="S3.F2.8.m8.1.1.2.1.cmml" xref="S3.F2.8.m8.1.1.2">subscript</csymbol><ci id="S3.F2.8.m8.1.1.2.2.cmml" xref="S3.F2.8.m8.1.1.2.2">𝑠</ci><ci id="S3.F2.8.m8.1.1.2.3.cmml" xref="S3.F2.8.m8.1.1.2.3">𝑙</ci></apply><apply id="S3.F2.8.m8.1.1.3.cmml" xref="S3.F2.8.m8.1.1.3"><csymbol cd="ambiguous" id="S3.F2.8.m8.1.1.3.1.cmml" xref="S3.F2.8.m8.1.1.3">subscript</csymbol><ci id="S3.F2.8.m8.1.1.3.2.cmml" xref="S3.F2.8.m8.1.1.3.2">𝑠</ci><apply id="S3.F2.8.m8.1.1.3.3.cmml" xref="S3.F2.8.m8.1.1.3.3"><plus id="S3.F2.8.m8.1.1.3.3.1.cmml" xref="S3.F2.8.m8.1.1.3.3.1"></plus><ci id="S3.F2.8.m8.1.1.3.3.2.cmml" xref="S3.F2.8.m8.1.1.3.3.2">𝑙</ci><cn type="integer" id="S3.F2.8.m8.1.1.3.3.3.cmml" xref="S3.F2.8.m8.1.1.3.3.3">1</cn></apply></apply><apply id="S3.F2.8.m8.1.1.4.cmml" xref="S3.F2.8.m8.1.1.4"><csymbol cd="ambiguous" id="S3.F2.8.m8.1.1.4.1.cmml" xref="S3.F2.8.m8.1.1.4">subscript</csymbol><ci id="S3.F2.8.m8.1.1.4.2.cmml" xref="S3.F2.8.m8.1.1.4.2">𝑠</ci><apply id="S3.F2.8.m8.1.1.4.3.cmml" xref="S3.F2.8.m8.1.1.4.3"><plus id="S3.F2.8.m8.1.1.4.3.1.cmml" xref="S3.F2.8.m8.1.1.4.3.1"></plus><ci id="S3.F2.8.m8.1.1.4.3.2.cmml" xref="S3.F2.8.m8.1.1.4.3.2">𝑙</ci><cn type="integer" id="S3.F2.8.m8.1.1.4.3.3.cmml" xref="S3.F2.8.m8.1.1.4.3.3">2</cn></apply></apply><ci id="S3.F2.8.m8.1.1.5.cmml" xref="S3.F2.8.m8.1.1.5">…</ci><apply id="S3.F2.8.m8.1.1.6.cmml" xref="S3.F2.8.m8.1.1.6"><csymbol cd="ambiguous" id="S3.F2.8.m8.1.1.6.1.cmml" xref="S3.F2.8.m8.1.1.6">subscript</csymbol><ci id="S3.F2.8.m8.1.1.6.2.cmml" xref="S3.F2.8.m8.1.1.6.2">𝑠</ci><ci id="S3.F2.8.m8.1.1.6.3.cmml" xref="S3.F2.8.m8.1.1.6.3">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.8.m8.1c">s_{l}s_{l+1}s_{l+2}\ldots s_{r}</annotation></semantics></math>. A substring <math id="S3.F2.9.m9.1" class="ltx_Math" alttext="s[l\dots r]" display="inline"><semantics id="S3.F2.9.m9.1a"><mrow id="S3.F2.9.m9.1.1" xref="S3.F2.9.m9.1.1.cmml"><mi id="S3.F2.9.m9.1.1.3" xref="S3.F2.9.m9.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.F2.9.m9.1.1.2" xref="S3.F2.9.m9.1.1.2.cmml">​</mo><mrow id="S3.F2.9.m9.1.1.1.1" xref="S3.F2.9.m9.1.1.1.2.cmml"><mo stretchy="false" id="S3.F2.9.m9.1.1.1.1.2" xref="S3.F2.9.m9.1.1.1.2.1.cmml">[</mo><mrow id="S3.F2.9.m9.1.1.1.1.1" xref="S3.F2.9.m9.1.1.1.1.1.cmml"><mi id="S3.F2.9.m9.1.1.1.1.1.2" xref="S3.F2.9.m9.1.1.1.1.1.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.F2.9.m9.1.1.1.1.1.1" xref="S3.F2.9.m9.1.1.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="S3.F2.9.m9.1.1.1.1.1.3" xref="S3.F2.9.m9.1.1.1.1.1.3.cmml">…</mi><mo lspace="0em" rspace="0em" id="S3.F2.9.m9.1.1.1.1.1.1a" xref="S3.F2.9.m9.1.1.1.1.1.1.cmml">​</mo><mi id="S3.F2.9.m9.1.1.1.1.1.4" xref="S3.F2.9.m9.1.1.1.1.1.4.cmml">r</mi></mrow><mo stretchy="false" id="S3.F2.9.m9.1.1.1.1.3" xref="S3.F2.9.m9.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.F2.9.m9.1b"><apply id="S3.F2.9.m9.1.1.cmml" xref="S3.F2.9.m9.1.1"><times id="S3.F2.9.m9.1.1.2.cmml" xref="S3.F2.9.m9.1.1.2"></times><ci id="S3.F2.9.m9.1.1.3.cmml" xref="S3.F2.9.m9.1.1.3">𝑠</ci><apply id="S3.F2.9.m9.1.1.1.2.cmml" xref="S3.F2.9.m9.1.1.1.1"><csymbol cd="latexml" id="S3.F2.9.m9.1.1.1.2.1.cmml" xref="S3.F2.9.m9.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.F2.9.m9.1.1.1.1.1.cmml" xref="S3.F2.9.m9.1.1.1.1.1"><times id="S3.F2.9.m9.1.1.1.1.1.1.cmml" xref="S3.F2.9.m9.1.1.1.1.1.1"></times><ci id="S3.F2.9.m9.1.1.1.1.1.2.cmml" xref="S3.F2.9.m9.1.1.1.1.1.2">𝑙</ci><ci id="S3.F2.9.m9.1.1.1.1.1.3.cmml" xref="S3.F2.9.m9.1.1.1.1.1.3">…</ci><ci id="S3.F2.9.m9.1.1.1.1.1.4.cmml" xref="S3.F2.9.m9.1.1.1.1.1.4">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.9.m9.1c">s[l\dots r]</annotation></semantics></math> of <math id="S3.F2.10.m10.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.F2.10.m10.1a"><mi id="S3.F2.10.m10.1.1" xref="S3.F2.10.m10.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.F2.10.m10.1b"><ci id="S3.F2.10.m10.1.1.cmml" xref="S3.F2.10.m10.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.10.m10.1c">s</annotation></semantics></math> is called even if the number represented by it is even. Find the number of even substrings of <math id="S3.F2.11.m11.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.F2.11.m11.1a"><mi id="S3.F2.11.m11.1.1" xref="S3.F2.11.m11.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.F2.11.m11.1b"><ci id="S3.F2.11.m11.1.1.cmml" xref="S3.F2.11.m11.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.11.m11.1c">s</annotation></semantics></math>. Note, that even if some substrings are equal as strings, but have different <math id="S3.F2.12.m12.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S3.F2.12.m12.1a"><mi id="S3.F2.12.m12.1.1" xref="S3.F2.12.m12.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S3.F2.12.m12.1b"><ci id="S3.F2.12.m12.1.1.cmml" xref="S3.F2.12.m12.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.12.m12.1c">l</annotation></semantics></math> and <math id="S3.F2.13.m13.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.F2.13.m13.1a"><mi id="S3.F2.13.m13.1.1" xref="S3.F2.13.m13.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.F2.13.m13.1b"><ci id="S3.F2.13.m13.1.1.cmml" xref="S3.F2.13.m13.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.13.m13.1c">r</annotation></semantics></math>, they are counted as different substrings. The first line contains an integer <math id="S3.F2.14.m14.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.F2.14.m14.1a"><mi id="S3.F2.14.m14.1.1" xref="S3.F2.14.m14.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.F2.14.m14.1b"><ci id="S3.F2.14.m14.1.1.cmml" xref="S3.F2.14.m14.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.14.m14.1c">n</annotation></semantics></math> (<math id="S3.F2.15.m15.1" class="ltx_Math" alttext="1\leq n\leq 65000" display="inline"><semantics id="S3.F2.15.m15.1a"><mrow id="S3.F2.15.m15.1.1" xref="S3.F2.15.m15.1.1.cmml"><mn id="S3.F2.15.m15.1.1.2" xref="S3.F2.15.m15.1.1.2.cmml">1</mn><mo id="S3.F2.15.m15.1.1.3" xref="S3.F2.15.m15.1.1.3.cmml">≤</mo><mi id="S3.F2.15.m15.1.1.4" xref="S3.F2.15.m15.1.1.4.cmml">n</mi><mo id="S3.F2.15.m15.1.1.5" xref="S3.F2.15.m15.1.1.5.cmml">≤</mo><mn id="S3.F2.15.m15.1.1.6" xref="S3.F2.15.m15.1.1.6.cmml">65000</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F2.15.m15.1b"><apply id="S3.F2.15.m15.1.1.cmml" xref="S3.F2.15.m15.1.1"><and id="S3.F2.15.m15.1.1a.cmml" xref="S3.F2.15.m15.1.1"></and><apply id="S3.F2.15.m15.1.1b.cmml" xref="S3.F2.15.m15.1.1"><leq id="S3.F2.15.m15.1.1.3.cmml" xref="S3.F2.15.m15.1.1.3"></leq><cn type="integer" id="S3.F2.15.m15.1.1.2.cmml" xref="S3.F2.15.m15.1.1.2">1</cn><ci id="S3.F2.15.m15.1.1.4.cmml" xref="S3.F2.15.m15.1.1.4">𝑛</ci></apply><apply id="S3.F2.15.m15.1.1c.cmml" xref="S3.F2.15.m15.1.1"><leq id="S3.F2.15.m15.1.1.5.cmml" xref="S3.F2.15.m15.1.1.5"></leq><share href="#S3.F2.15.m15.1.1.4.cmml" id="S3.F2.15.m15.1.1d.cmml" xref="S3.F2.15.m15.1.1"></share><cn type="integer" id="S3.F2.15.m15.1.1.6.cmml" xref="S3.F2.15.m15.1.1.6">65000</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.15.m15.1c">1\leq n\leq 65000</annotation></semantics></math>) — the length of the string <math id="S3.F2.16.m16.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.F2.16.m16.1a"><mi id="S3.F2.16.m16.1.1" xref="S3.F2.16.m16.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.F2.16.m16.1b"><ci id="S3.F2.16.m16.1.1.cmml" xref="S3.F2.16.m16.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.16.m16.1c">s</annotation></semantics></math>. The second line contains a string <math id="S3.F2.17.m17.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.F2.17.m17.1a"><mi id="S3.F2.17.m17.1.1" xref="S3.F2.17.m17.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.F2.17.m17.1b"><ci id="S3.F2.17.m17.1.1.cmml" xref="S3.F2.17.m17.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.17.m17.1c">s</annotation></semantics></math> of length <math id="S3.F2.18.m18.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.F2.18.m18.1a"><mi id="S3.F2.18.m18.1.1" xref="S3.F2.18.m18.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.F2.18.m18.1b"><ci id="S3.F2.18.m18.1.1.cmml" xref="S3.F2.18.m18.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.18.m18.1c">n</annotation></semantics></math>. The string <math id="S3.F2.19.m19.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.F2.19.m19.1a"><mi id="S3.F2.19.m19.1.1" xref="S3.F2.19.m19.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.F2.19.m19.1b"><ci id="S3.F2.19.m19.1.1.cmml" xref="S3.F2.19.m19.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.19.m19.1c">s</annotation></semantics></math> consists only of digits <math id="S3.F2.20.m20.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.F2.20.m20.1a"><mn id="S3.F2.20.m20.1.1" xref="S3.F2.20.m20.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.F2.20.m20.1b"><cn type="integer" id="S3.F2.20.m20.1.1.cmml" xref="S3.F2.20.m20.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.20.m20.1c">1</annotation></semantics></math>, <math id="S3.F2.21.m21.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S3.F2.21.m21.1a"><mn id="S3.F2.21.m21.1.1" xref="S3.F2.21.m21.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S3.F2.21.m21.1b"><cn type="integer" id="S3.F2.21.m21.1.1.cmml" xref="S3.F2.21.m21.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.21.m21.1c">2</annotation></semantics></math>,…, <math id="S3.F2.22.m22.1" class="ltx_Math" alttext="9" display="inline"><semantics id="S3.F2.22.m22.1a"><mn id="S3.F2.22.m22.1.1" xref="S3.F2.22.m22.1.1.cmml">9</mn><annotation-xml encoding="MathML-Content" id="S3.F2.22.m22.1b"><cn type="integer" id="S3.F2.22.m22.1.1.cmml" xref="S3.F2.22.m22.1.1">9</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.22.m22.1c">9</annotation></semantics></math>. Print the number of even substrings of <math id="S3.F2.23.m23.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.F2.23.m23.1a"><mi id="S3.F2.23.m23.1.1" xref="S3.F2.23.m23.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.F2.23.m23.1b"><ci id="S3.F2.23.m23.1.1.cmml" xref="S3.F2.23.m23.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.23.m23.1c">s</annotation></semantics></math>.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S3.F2.24" class="ltx_p ltx_figure_panel"><span id="S3.F2.24.1" class="ltx_text ltx_font_bold">Model Output</span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S3.F2.25" class="ltx_listing ltx_figure_panel ltx_lst_numbers_left ltx_lstlisting ltx_listing" style="background-color:#FFFFFF;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,Cm4gPSBpbnQoaW5wdXQoKSkKcyA9IGlucHV0KCkKYW5zID0gMApmb3IgaSBpbiByYW5nZShuKToKZm9yIGogaW4gcmFuZ2UoaSwgbik6CmlmIGludChzW2k6aisxXSkgYW5zICs9IDEKcHJpbnQoYW5zKQo=" download="">⬇</a></div>
<div id="lstnumberx1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>
</div>
<div id="lstnumberx2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="lstnumberx2.1" class="ltx_text ltx_lst_identifier">n</span><span id="lstnumberx2.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx2.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.4" class="ltx_text ltx_lst_identifier">int</span>(<span id="lstnumberx2.5" class="ltx_text ltx_lst_identifier">input</span>())
</div>
<div id="lstnumberx3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span><span id="lstnumberx3.1" class="ltx_text ltx_lst_identifier">s</span><span id="lstnumberx3.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx3.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx3.4" class="ltx_text ltx_lst_identifier">input</span>()
</div>
<div id="lstnumberx4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span><span id="lstnumberx4.1" class="ltx_text ltx_lst_identifier">ans</span><span id="lstnumberx4.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx4.3" class="ltx_text ltx_lst_space"> </span>0
</div>
<div id="lstnumberx5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span><span id="lstnumberx5.1" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx5.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx5.3" class="ltx_text ltx_lst_identifier">i</span><span id="lstnumberx5.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx5.5" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx5.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx5.7" class="ltx_text ltx_lst_identifier">range</span>(<span id="lstnumberx5.8" class="ltx_text ltx_lst_identifier">n</span>):
</div>
<div id="lstnumberx6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6</span><span id="lstnumberx6.1" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx6.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.3" class="ltx_text ltx_lst_identifier">j</span><span id="lstnumberx6.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.5" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx6.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.7" class="ltx_text ltx_lst_identifier">range</span>(<span id="lstnumberx6.8" class="ltx_text ltx_lst_identifier">i</span>,<span id="lstnumberx6.9" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.10" class="ltx_text ltx_lst_identifier">n</span>):
</div>
<div id="lstnumberx7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7</span><span id="lstnumberx7.1" class="ltx_text ltx_lst_identifier">if</span><span id="lstnumberx7.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx7.3" class="ltx_text ltx_lst_identifier">int</span>(<span id="lstnumberx7.4" class="ltx_text ltx_lst_identifier">s</span>[<span id="lstnumberx7.5" class="ltx_text ltx_lst_identifier">i</span>:<span id="lstnumberx7.6" class="ltx_text ltx_lst_identifier">j</span>+1])<span id="lstnumberx7.7" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx7.8" class="ltx_text ltx_lst_identifier">ans</span><span id="lstnumberx7.9" class="ltx_text ltx_lst_space"> </span>+=<span id="lstnumberx7.10" class="ltx_text ltx_lst_space"> </span>1
</div>
<div id="lstnumberx8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8</span><span id="lstnumberx8.1" class="ltx_text ltx_lst_identifier">print</span>(<span id="lstnumberx8.2" class="ltx_text ltx_lst_identifier">ans</span>)
</div>
</div>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Model-generated code from GPT-2 1.5B. 18/18 test cases pass for the problem. Note that models were given in problems exactly as they they are formatted here, which even meant parsing <span id="S3.F2.27.1" class="ltx_text ltx_LaTeX_logo" style="letter-spacing:-0.2em; margin-right:0.1em;">L<span id="S3.F2.27.1.1" class="ltx_text" style="position:relative; bottom:0.4ex;font-variant:small-caps;;">a</span>T<span id="S3.F2.27.1.2" class="ltx_text" style="position:relative; bottom:-0.2ex;font-variant:small-caps;font-size:120%;">e</span>X</span>. For brevity, we exclude formatting instructions from this depicted problem.</figcaption>
</figure>
<figure id="S3.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S3.F3.20" class="ltx_p ltx_figure_panel"><span id="S3.F3.20.1" class="ltx_text ltx_font_bold">Problem</span> 
<br class="ltx_break">You are given two integers <math id="S3.F3.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.F3.1.m1.1a"><mi id="S3.F3.1.m1.1.1" xref="S3.F3.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.F3.1.m1.1b"><ci id="S3.F3.1.m1.1.1.cmml" xref="S3.F3.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.1.m1.1c">n</annotation></semantics></math> and <math id="S3.F3.2.m2.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S3.F3.2.m2.1a"><mi id="S3.F3.2.m2.1.1" xref="S3.F3.2.m2.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S3.F3.2.m2.1b"><ci id="S3.F3.2.m2.1.1.cmml" xref="S3.F3.2.m2.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.2.m2.1c">m</annotation></semantics></math>. Calculate the number of pairs of arrays <math id="S3.F3.3.m3.2" class="ltx_Math" alttext="(a,b)" display="inline"><semantics id="S3.F3.3.m3.2a"><mrow id="S3.F3.3.m3.2.3.2" xref="S3.F3.3.m3.2.3.1.cmml"><mo stretchy="false" id="S3.F3.3.m3.2.3.2.1" xref="S3.F3.3.m3.2.3.1.cmml">(</mo><mi id="S3.F3.3.m3.1.1" xref="S3.F3.3.m3.1.1.cmml">a</mi><mo id="S3.F3.3.m3.2.3.2.2" xref="S3.F3.3.m3.2.3.1.cmml">,</mo><mi id="S3.F3.3.m3.2.2" xref="S3.F3.3.m3.2.2.cmml">b</mi><mo stretchy="false" id="S3.F3.3.m3.2.3.2.3" xref="S3.F3.3.m3.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.3.m3.2b"><interval closure="open" id="S3.F3.3.m3.2.3.1.cmml" xref="S3.F3.3.m3.2.3.2"><ci id="S3.F3.3.m3.1.1.cmml" xref="S3.F3.3.m3.1.1">𝑎</ci><ci id="S3.F3.3.m3.2.2.cmml" xref="S3.F3.3.m3.2.2">𝑏</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.3.m3.2c">(a,b)</annotation></semantics></math> such that: the length of both arrays is equal to <math id="S3.F3.4.m4.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S3.F3.4.m4.1a"><mi id="S3.F3.4.m4.1.1" xref="S3.F3.4.m4.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S3.F3.4.m4.1b"><ci id="S3.F3.4.m4.1.1.cmml" xref="S3.F3.4.m4.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.4.m4.1c">m</annotation></semantics></math>; each element of each array is an integer between <math id="S3.F3.5.m5.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.F3.5.m5.1a"><mn id="S3.F3.5.m5.1.1" xref="S3.F3.5.m5.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.F3.5.m5.1b"><cn type="integer" id="S3.F3.5.m5.1.1.cmml" xref="S3.F3.5.m5.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.5.m5.1c">1</annotation></semantics></math> and <math id="S3.F3.6.m6.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.F3.6.m6.1a"><mi id="S3.F3.6.m6.1.1" xref="S3.F3.6.m6.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.F3.6.m6.1b"><ci id="S3.F3.6.m6.1.1.cmml" xref="S3.F3.6.m6.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.6.m6.1c">n</annotation></semantics></math> (inclusive); <math id="S3.F3.7.m7.1" class="ltx_Math" alttext="a_{i}\leq b_{i}" display="inline"><semantics id="S3.F3.7.m7.1a"><mrow id="S3.F3.7.m7.1.1" xref="S3.F3.7.m7.1.1.cmml"><msub id="S3.F3.7.m7.1.1.2" xref="S3.F3.7.m7.1.1.2.cmml"><mi id="S3.F3.7.m7.1.1.2.2" xref="S3.F3.7.m7.1.1.2.2.cmml">a</mi><mi id="S3.F3.7.m7.1.1.2.3" xref="S3.F3.7.m7.1.1.2.3.cmml">i</mi></msub><mo id="S3.F3.7.m7.1.1.1" xref="S3.F3.7.m7.1.1.1.cmml">≤</mo><msub id="S3.F3.7.m7.1.1.3" xref="S3.F3.7.m7.1.1.3.cmml"><mi id="S3.F3.7.m7.1.1.3.2" xref="S3.F3.7.m7.1.1.3.2.cmml">b</mi><mi id="S3.F3.7.m7.1.1.3.3" xref="S3.F3.7.m7.1.1.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.7.m7.1b"><apply id="S3.F3.7.m7.1.1.cmml" xref="S3.F3.7.m7.1.1"><leq id="S3.F3.7.m7.1.1.1.cmml" xref="S3.F3.7.m7.1.1.1"></leq><apply id="S3.F3.7.m7.1.1.2.cmml" xref="S3.F3.7.m7.1.1.2"><csymbol cd="ambiguous" id="S3.F3.7.m7.1.1.2.1.cmml" xref="S3.F3.7.m7.1.1.2">subscript</csymbol><ci id="S3.F3.7.m7.1.1.2.2.cmml" xref="S3.F3.7.m7.1.1.2.2">𝑎</ci><ci id="S3.F3.7.m7.1.1.2.3.cmml" xref="S3.F3.7.m7.1.1.2.3">𝑖</ci></apply><apply id="S3.F3.7.m7.1.1.3.cmml" xref="S3.F3.7.m7.1.1.3"><csymbol cd="ambiguous" id="S3.F3.7.m7.1.1.3.1.cmml" xref="S3.F3.7.m7.1.1.3">subscript</csymbol><ci id="S3.F3.7.m7.1.1.3.2.cmml" xref="S3.F3.7.m7.1.1.3.2">𝑏</ci><ci id="S3.F3.7.m7.1.1.3.3.cmml" xref="S3.F3.7.m7.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.7.m7.1c">a_{i}\leq b_{i}</annotation></semantics></math> for any index <math id="S3.F3.8.m8.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.F3.8.m8.1a"><mi id="S3.F3.8.m8.1.1" xref="S3.F3.8.m8.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.F3.8.m8.1b"><ci id="S3.F3.8.m8.1.1.cmml" xref="S3.F3.8.m8.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.8.m8.1c">i</annotation></semantics></math> from <math id="S3.F3.9.m9.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.F3.9.m9.1a"><mn id="S3.F3.9.m9.1.1" xref="S3.F3.9.m9.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.F3.9.m9.1b"><cn type="integer" id="S3.F3.9.m9.1.1.cmml" xref="S3.F3.9.m9.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.9.m9.1c">1</annotation></semantics></math> to <math id="S3.F3.10.m10.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S3.F3.10.m10.1a"><mi id="S3.F3.10.m10.1.1" xref="S3.F3.10.m10.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S3.F3.10.m10.1b"><ci id="S3.F3.10.m10.1.1.cmml" xref="S3.F3.10.m10.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.10.m10.1c">m</annotation></semantics></math>; array <math id="S3.F3.11.m11.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S3.F3.11.m11.1a"><mi id="S3.F3.11.m11.1.1" xref="S3.F3.11.m11.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S3.F3.11.m11.1b"><ci id="S3.F3.11.m11.1.1.cmml" xref="S3.F3.11.m11.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.11.m11.1c">a</annotation></semantics></math> is sorted in non-descending order; array <math id="S3.F3.12.m12.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S3.F3.12.m12.1a"><mi id="S3.F3.12.m12.1.1" xref="S3.F3.12.m12.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S3.F3.12.m12.1b"><ci id="S3.F3.12.m12.1.1.cmml" xref="S3.F3.12.m12.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.12.m12.1c">b</annotation></semantics></math> is sorted in non-ascending order. As the result can be very large, you should print it modulo <math id="S3.F3.13.m13.1" class="ltx_Math" alttext="10^{9}+7" display="inline"><semantics id="S3.F3.13.m13.1a"><mrow id="S3.F3.13.m13.1.1" xref="S3.F3.13.m13.1.1.cmml"><msup id="S3.F3.13.m13.1.1.2" xref="S3.F3.13.m13.1.1.2.cmml"><mn id="S3.F3.13.m13.1.1.2.2" xref="S3.F3.13.m13.1.1.2.2.cmml">10</mn><mn id="S3.F3.13.m13.1.1.2.3" xref="S3.F3.13.m13.1.1.2.3.cmml">9</mn></msup><mo id="S3.F3.13.m13.1.1.1" xref="S3.F3.13.m13.1.1.1.cmml">+</mo><mn id="S3.F3.13.m13.1.1.3" xref="S3.F3.13.m13.1.1.3.cmml">7</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.13.m13.1b"><apply id="S3.F3.13.m13.1.1.cmml" xref="S3.F3.13.m13.1.1"><plus id="S3.F3.13.m13.1.1.1.cmml" xref="S3.F3.13.m13.1.1.1"></plus><apply id="S3.F3.13.m13.1.1.2.cmml" xref="S3.F3.13.m13.1.1.2"><csymbol cd="ambiguous" id="S3.F3.13.m13.1.1.2.1.cmml" xref="S3.F3.13.m13.1.1.2">superscript</csymbol><cn type="integer" id="S3.F3.13.m13.1.1.2.2.cmml" xref="S3.F3.13.m13.1.1.2.2">10</cn><cn type="integer" id="S3.F3.13.m13.1.1.2.3.cmml" xref="S3.F3.13.m13.1.1.2.3">9</cn></apply><cn type="integer" id="S3.F3.13.m13.1.1.3.cmml" xref="S3.F3.13.m13.1.1.3">7</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.13.m13.1c">10^{9}+7</annotation></semantics></math>. Input: The only line contains two integers <math id="S3.F3.14.m14.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.F3.14.m14.1a"><mi id="S3.F3.14.m14.1.1" xref="S3.F3.14.m14.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.F3.14.m14.1b"><ci id="S3.F3.14.m14.1.1.cmml" xref="S3.F3.14.m14.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.14.m14.1c">n</annotation></semantics></math> and <math id="S3.F3.15.m15.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S3.F3.15.m15.1a"><mi id="S3.F3.15.m15.1.1" xref="S3.F3.15.m15.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S3.F3.15.m15.1b"><ci id="S3.F3.15.m15.1.1.cmml" xref="S3.F3.15.m15.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.15.m15.1c">m</annotation></semantics></math> (<math id="S3.F3.16.m16.1" class="ltx_Math" alttext="1\leq n\leq 1000" display="inline"><semantics id="S3.F3.16.m16.1a"><mrow id="S3.F3.16.m16.1.1" xref="S3.F3.16.m16.1.1.cmml"><mn id="S3.F3.16.m16.1.1.2" xref="S3.F3.16.m16.1.1.2.cmml">1</mn><mo id="S3.F3.16.m16.1.1.3" xref="S3.F3.16.m16.1.1.3.cmml">≤</mo><mi id="S3.F3.16.m16.1.1.4" xref="S3.F3.16.m16.1.1.4.cmml">n</mi><mo id="S3.F3.16.m16.1.1.5" xref="S3.F3.16.m16.1.1.5.cmml">≤</mo><mn id="S3.F3.16.m16.1.1.6" xref="S3.F3.16.m16.1.1.6.cmml">1000</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.16.m16.1b"><apply id="S3.F3.16.m16.1.1.cmml" xref="S3.F3.16.m16.1.1"><and id="S3.F3.16.m16.1.1a.cmml" xref="S3.F3.16.m16.1.1"></and><apply id="S3.F3.16.m16.1.1b.cmml" xref="S3.F3.16.m16.1.1"><leq id="S3.F3.16.m16.1.1.3.cmml" xref="S3.F3.16.m16.1.1.3"></leq><cn type="integer" id="S3.F3.16.m16.1.1.2.cmml" xref="S3.F3.16.m16.1.1.2">1</cn><ci id="S3.F3.16.m16.1.1.4.cmml" xref="S3.F3.16.m16.1.1.4">𝑛</ci></apply><apply id="S3.F3.16.m16.1.1c.cmml" xref="S3.F3.16.m16.1.1"><leq id="S3.F3.16.m16.1.1.5.cmml" xref="S3.F3.16.m16.1.1.5"></leq><share href="#S3.F3.16.m16.1.1.4.cmml" id="S3.F3.16.m16.1.1d.cmml" xref="S3.F3.16.m16.1.1"></share><cn type="integer" id="S3.F3.16.m16.1.1.6.cmml" xref="S3.F3.16.m16.1.1.6">1000</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.16.m16.1c">1\leq n\leq 1000</annotation></semantics></math>, <math id="S3.F3.17.m17.1" class="ltx_Math" alttext="1\leq m\leq 10" display="inline"><semantics id="S3.F3.17.m17.1a"><mrow id="S3.F3.17.m17.1.1" xref="S3.F3.17.m17.1.1.cmml"><mn id="S3.F3.17.m17.1.1.2" xref="S3.F3.17.m17.1.1.2.cmml">1</mn><mo id="S3.F3.17.m17.1.1.3" xref="S3.F3.17.m17.1.1.3.cmml">≤</mo><mi id="S3.F3.17.m17.1.1.4" xref="S3.F3.17.m17.1.1.4.cmml">m</mi><mo id="S3.F3.17.m17.1.1.5" xref="S3.F3.17.m17.1.1.5.cmml">≤</mo><mn id="S3.F3.17.m17.1.1.6" xref="S3.F3.17.m17.1.1.6.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.17.m17.1b"><apply id="S3.F3.17.m17.1.1.cmml" xref="S3.F3.17.m17.1.1"><and id="S3.F3.17.m17.1.1a.cmml" xref="S3.F3.17.m17.1.1"></and><apply id="S3.F3.17.m17.1.1b.cmml" xref="S3.F3.17.m17.1.1"><leq id="S3.F3.17.m17.1.1.3.cmml" xref="S3.F3.17.m17.1.1.3"></leq><cn type="integer" id="S3.F3.17.m17.1.1.2.cmml" xref="S3.F3.17.m17.1.1.2">1</cn><ci id="S3.F3.17.m17.1.1.4.cmml" xref="S3.F3.17.m17.1.1.4">𝑚</ci></apply><apply id="S3.F3.17.m17.1.1c.cmml" xref="S3.F3.17.m17.1.1"><leq id="S3.F3.17.m17.1.1.5.cmml" xref="S3.F3.17.m17.1.1.5"></leq><share href="#S3.F3.17.m17.1.1.4.cmml" id="S3.F3.17.m17.1.1d.cmml" xref="S3.F3.17.m17.1.1"></share><cn type="integer" id="S3.F3.17.m17.1.1.6.cmml" xref="S3.F3.17.m17.1.1.6">10</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.17.m17.1c">1\leq m\leq 10</annotation></semantics></math>). Output: Print one integer – the number of arrays <math id="S3.F3.18.m18.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S3.F3.18.m18.1a"><mi id="S3.F3.18.m18.1.1" xref="S3.F3.18.m18.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S3.F3.18.m18.1b"><ci id="S3.F3.18.m18.1.1.cmml" xref="S3.F3.18.m18.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.18.m18.1c">a</annotation></semantics></math> and <math id="S3.F3.19.m19.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S3.F3.19.m19.1a"><mi id="S3.F3.19.m19.1.1" xref="S3.F3.19.m19.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S3.F3.19.m19.1b"><ci id="S3.F3.19.m19.1.1.cmml" xref="S3.F3.19.m19.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.19.m19.1c">b</annotation></semantics></math> satisfying the conditions described above modulo <math id="S3.F3.20.m20.1" class="ltx_Math" alttext="10^{9}+7" display="inline"><semantics id="S3.F3.20.m20.1a"><mrow id="S3.F3.20.m20.1.1" xref="S3.F3.20.m20.1.1.cmml"><msup id="S3.F3.20.m20.1.1.2" xref="S3.F3.20.m20.1.1.2.cmml"><mn id="S3.F3.20.m20.1.1.2.2" xref="S3.F3.20.m20.1.1.2.2.cmml">10</mn><mn id="S3.F3.20.m20.1.1.2.3" xref="S3.F3.20.m20.1.1.2.3.cmml">9</mn></msup><mo id="S3.F3.20.m20.1.1.1" xref="S3.F3.20.m20.1.1.1.cmml">+</mo><mn id="S3.F3.20.m20.1.1.3" xref="S3.F3.20.m20.1.1.3.cmml">7</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.20.m20.1b"><apply id="S3.F3.20.m20.1.1.cmml" xref="S3.F3.20.m20.1.1"><plus id="S3.F3.20.m20.1.1.1.cmml" xref="S3.F3.20.m20.1.1.1"></plus><apply id="S3.F3.20.m20.1.1.2.cmml" xref="S3.F3.20.m20.1.1.2"><csymbol cd="ambiguous" id="S3.F3.20.m20.1.1.2.1.cmml" xref="S3.F3.20.m20.1.1.2">superscript</csymbol><cn type="integer" id="S3.F3.20.m20.1.1.2.2.cmml" xref="S3.F3.20.m20.1.1.2.2">10</cn><cn type="integer" id="S3.F3.20.m20.1.1.2.3.cmml" xref="S3.F3.20.m20.1.1.2.3">9</cn></apply><cn type="integer" id="S3.F3.20.m20.1.1.3.cmml" xref="S3.F3.20.m20.1.1.3">7</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.20.m20.1c">10^{9}+7</annotation></semantics></math>.

<br class="ltx_break">
<br class="ltx_break"><span id="S3.F3.20.2" class="ltx_text ltx_font_bold">Model Output</span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S3.F3.21" class="ltx_listing ltx_figure_panel ltx_lst_numbers_left ltx_lstlisting ltx_listing" style="background-color:#FFFFFF;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,Cm4sIG0gPSBtYXAoaW50LCBpbnB1dCgpLnNwbGl0KCkpCm1vZCA9IDEwKio5KzcKXHBhcmZhY3QgPSBbMSwgMV0KZmFjdGludiA9IFsxLCAxXQppbnYgPSBbMCwgMV0KXHBhcmZvciBpIGluIHJhbmdlKDIsIG4rMSk6CmZhY3QuYXBwZW5kKChmYWN0Wy0xXSAqIGkpIGludi5hcHBlbmQoKC1pbnZbbW9kIGZhY3RpbnYuYXBwZW5kKChmYWN0aW52Wy0xXSAqIGludlstMV0pIFxwYXJkZWYgY21iKG4sIHIsIG1vZCk6CmlmIChyIDwgMCkgb3IgKG4gPCByKToKcmV0dXJuIDAKciA9IG1pbihyLCBuIC0gcikKcmV0dXJuIGZhY3Rbbl0gKiBmYWN0aW52W3JdICogZmFjdGludltuLXJdIFxwYXJwcmludCgoY21iKG4sIG0sIG1vZCkgKiBjbWIobS0xLCBuLTEsIG1vZCkpIA==" download="">⬇</a></div>
<div id="lstnumberx9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>
</div>
<div id="lstnumberx10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="lstnumberx10.1" class="ltx_text ltx_lst_identifier">n</span>,<span id="lstnumberx10.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx10.3" class="ltx_text ltx_lst_identifier">m</span><span id="lstnumberx10.4" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx10.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx10.6" class="ltx_text ltx_lst_identifier">map</span>(<span id="lstnumberx10.7" class="ltx_text ltx_lst_identifier">int</span>,<span id="lstnumberx10.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx10.9" class="ltx_text ltx_lst_identifier">input</span>().<span id="lstnumberx10.10" class="ltx_text ltx_lst_identifier">split</span>())
</div>
<div id="lstnumberx11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span><span id="lstnumberx11.1" class="ltx_text ltx_lst_identifier">mod</span><span id="lstnumberx11.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx11.3" class="ltx_text ltx_lst_space"> </span>10**9+7
</div>
<div id="lstnumberx12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span>\<span id="lstnumberx12.1" class="ltx_text ltx_lst_identifier">parfact</span><span id="lstnumberx12.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx12.3" class="ltx_text ltx_lst_space"> </span>[1,<span id="lstnumberx12.4" class="ltx_text ltx_lst_space"> </span>1]
</div>
<div id="lstnumberx13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span><span id="lstnumberx13.1" class="ltx_text ltx_lst_identifier">factinv</span><span id="lstnumberx13.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx13.3" class="ltx_text ltx_lst_space"> </span>[1,<span id="lstnumberx13.4" class="ltx_text ltx_lst_space"> </span>1]
</div>
<div id="lstnumberx14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6</span><span id="lstnumberx14.1" class="ltx_text ltx_lst_identifier">inv</span><span id="lstnumberx14.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx14.3" class="ltx_text ltx_lst_space"> </span>[0,<span id="lstnumberx14.4" class="ltx_text ltx_lst_space"> </span>1]
</div>
<div id="lstnumberx15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7</span>\<span id="lstnumberx15.1" class="ltx_text ltx_lst_identifier">parfor</span><span id="lstnumberx15.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx15.3" class="ltx_text ltx_lst_identifier">i</span><span id="lstnumberx15.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx15.5" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx15.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx15.7" class="ltx_text ltx_lst_identifier">range</span>(2,<span id="lstnumberx15.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx15.9" class="ltx_text ltx_lst_identifier">n</span>+1):
</div>
<div id="lstnumberx16" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8</span><span id="lstnumberx16.1" class="ltx_text ltx_lst_identifier">fact</span>.<span id="lstnumberx16.2" class="ltx_text ltx_lst_identifier">append</span>((<span id="lstnumberx16.3" class="ltx_text ltx_lst_identifier">fact</span>[-1]<span id="lstnumberx16.4" class="ltx_text ltx_lst_space"> </span>*<span id="lstnumberx16.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx16.6" class="ltx_text ltx_lst_identifier">i</span>)<span id="lstnumberx16.7" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx16.8" class="ltx_text ltx_lst_identifier">inv</span>.<span id="lstnumberx16.9" class="ltx_text ltx_lst_identifier">append</span>((-<span id="lstnumberx16.10" class="ltx_text ltx_lst_identifier">inv</span>[<span id="lstnumberx16.11" class="ltx_text ltx_lst_identifier">mod</span><span id="lstnumberx16.12" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx16.13" class="ltx_text ltx_lst_identifier">factinv</span>.<span id="lstnumberx16.14" class="ltx_text ltx_lst_identifier">append</span>((<span id="lstnumberx16.15" class="ltx_text ltx_lst_identifier">factinv</span>[-1]<span id="lstnumberx16.16" class="ltx_text ltx_lst_space"> </span>*<span id="lstnumberx16.17" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx16.18" class="ltx_text ltx_lst_identifier">inv</span>[-1])<span id="lstnumberx16.19" class="ltx_text ltx_lst_space"> </span>\<span id="lstnumberx16.20" class="ltx_text ltx_lst_identifier">pardef</span><span id="lstnumberx16.21" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx16.22" class="ltx_text ltx_lst_identifier">cmb</span>(<span id="lstnumberx16.23" class="ltx_text ltx_lst_identifier">n</span>,<span id="lstnumberx16.24" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx16.25" class="ltx_text ltx_lst_identifier">r</span>,<span id="lstnumberx16.26" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx16.27" class="ltx_text ltx_lst_identifier">mod</span>):
</div>
<div id="lstnumberx17" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9</span><span id="lstnumberx17.1" class="ltx_text ltx_lst_identifier">if</span><span id="lstnumberx17.2" class="ltx_text ltx_lst_space"> </span>(<span id="lstnumberx17.3" class="ltx_text ltx_lst_identifier">r</span><span id="lstnumberx17.4" class="ltx_text ltx_lst_space"> </span>&lt;<span id="lstnumberx17.5" class="ltx_text ltx_lst_space"> </span>0)<span id="lstnumberx17.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx17.7" class="ltx_text ltx_lst_identifier">or</span><span id="lstnumberx17.8" class="ltx_text ltx_lst_space"> </span>(<span id="lstnumberx17.9" class="ltx_text ltx_lst_identifier">n</span><span id="lstnumberx17.10" class="ltx_text ltx_lst_space"> </span>&lt;<span id="lstnumberx17.11" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx17.12" class="ltx_text ltx_lst_identifier">r</span>):
</div>
<div id="lstnumberx18" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10</span><span id="lstnumberx18.1" class="ltx_text ltx_lst_identifier">return</span><span id="lstnumberx18.2" class="ltx_text ltx_lst_space"> </span>0
</div>
<div id="lstnumberx19" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11</span><span id="lstnumberx19.1" class="ltx_text ltx_lst_identifier">r</span><span id="lstnumberx19.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx19.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx19.4" class="ltx_text ltx_lst_identifier">min</span>(<span id="lstnumberx19.5" class="ltx_text ltx_lst_identifier">r</span>,<span id="lstnumberx19.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx19.7" class="ltx_text ltx_lst_identifier">n</span><span id="lstnumberx19.8" class="ltx_text ltx_lst_space"> </span>-<span id="lstnumberx19.9" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx19.10" class="ltx_text ltx_lst_identifier">r</span>)
</div>
<div id="lstnumberx20" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12</span><span id="lstnumberx20.1" class="ltx_text ltx_lst_identifier">return</span><span id="lstnumberx20.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx20.3" class="ltx_text ltx_lst_identifier">fact</span>[<span id="lstnumberx20.4" class="ltx_text ltx_lst_identifier">n</span>]<span id="lstnumberx20.5" class="ltx_text ltx_lst_space"> </span>*<span id="lstnumberx20.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx20.7" class="ltx_text ltx_lst_identifier">factinv</span>[<span id="lstnumberx20.8" class="ltx_text ltx_lst_identifier">r</span>]<span id="lstnumberx20.9" class="ltx_text ltx_lst_space"> </span>*<span id="lstnumberx20.10" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx20.11" class="ltx_text ltx_lst_identifier">factinv</span>[<span id="lstnumberx20.12" class="ltx_text ltx_lst_identifier">n</span>-<span id="lstnumberx20.13" class="ltx_text ltx_lst_identifier">r</span>]<span id="lstnumberx20.14" class="ltx_text ltx_lst_space"> </span>\<span id="lstnumberx20.15" class="ltx_text ltx_lst_identifier">parprint</span>((<span id="lstnumberx20.16" class="ltx_text ltx_lst_identifier">cmb</span>(<span id="lstnumberx20.17" class="ltx_text ltx_lst_identifier">n</span>,<span id="lstnumberx20.18" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx20.19" class="ltx_text ltx_lst_identifier">m</span>,<span id="lstnumberx20.20" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx20.21" class="ltx_text ltx_lst_identifier">mod</span>)<span id="lstnumberx20.22" class="ltx_text ltx_lst_space"> </span>*<span id="lstnumberx20.23" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx20.24" class="ltx_text ltx_lst_identifier">cmb</span>(<span id="lstnumberx20.25" class="ltx_text ltx_lst_identifier">m</span>-1,<span id="lstnumberx20.26" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx20.27" class="ltx_text ltx_lst_identifier">n</span>-1,<span id="lstnumberx20.28" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx20.29" class="ltx_text ltx_lst_identifier">mod</span>))
</div>
</div>
</div>
<div class="ltx_flex_break"></div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>An example from GPT-2 1.5B. Although the code generated passes 0 test cases, it looks plausible at first glance.</figcaption>
</figure>
<section id="S3.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Test Case Quality.</h4>

<div id="S3.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS0.SSS0.Px1.p1.2" class="ltx_p">In the APPS test split, the average number of test cases is <math id="S3.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="21.2" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.1.m1.1a"><mn id="S3.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.cmml">21.2</mn><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.1.m1.1b"><cn type="float" id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1">21.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.1.m1.1c">21.2</annotation></semantics></math>, but some problems only have two test cases. These problems mainly come from Kattis and were chosen for the test split due to limited numbers of competition problems. A potential concern is that these problems could result in false positives if models happen to guess both test cases correctly. This is very unlikely in problems with large output spaces, but some problems have small output spaces, such as <math id="S3.SS0.SSS0.Px1.p1.2.m2.2" class="ltx_Math" alttext="\{\text{``YES''},\text{``NO''}\}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.2.m2.2a"><mrow id="S3.SS0.SSS0.Px1.p1.2.m2.2.3.2" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.3.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.2.m2.2.3.2.1" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.3.1.cmml">{</mo><mtext id="S3.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1a.cmml">“YES”</mtext><mo id="S3.SS0.SSS0.Px1.p1.2.m2.2.3.2.2" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.3.1.cmml">,</mo><mtext id="S3.SS0.SSS0.Px1.p1.2.m2.2.2" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2a.cmml">“NO”</mtext><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.2.m2.2.3.2.3" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.3.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.2.m2.2b"><set id="S3.SS0.SSS0.Px1.p1.2.m2.2.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.3.2"><ci id="S3.SS0.SSS0.Px1.p1.2.m2.1.1a.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1"><mtext id="S3.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1">“YES”</mtext></ci><ci id="S3.SS0.SSS0.Px1.p1.2.m2.2.2a.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2"><mtext id="S3.SS0.SSS0.Px1.p1.2.m2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2">“NO”</mtext></ci></set></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.2.m2.2c">\{\text{``YES''},\text{``NO''}\}</annotation></semantics></math>. Although the model must narrow down these two options from the space of all possible strings, we find that this is not difficult for current models to accomplish, and this may cause false positives in our evaluation.</p>
</div>
<div id="S3.SS0.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS0.SSS0.Px1.p2.2" class="ltx_p">To quantify the impact of these problems, we took all 2- and 3-test-case problems that GPT-Neo 2.7B passed and manually checked for false positives. Out of 12 problems that the model got correct with 2 test cases, 8 were false positives. Out of 6 problems with 3 test cases, only 1 was a false positive. Since the test cases are from coding challenge websites with many users, we can assume that false negatives are absent. Thus, the false positive rate with 2 test cases is <math id="S3.SS0.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="8/(890-4)=0.009&lt;1\%" display="inline"><semantics id="S3.SS0.SSS0.Px1.p2.1.m1.1a"><mrow id="S3.SS0.SSS0.Px1.p2.1.m1.1.1" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.cmml"><mrow id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.cmml"><mn id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.3.cmml">8</mn><mo id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.2" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.2.cmml">/</mo><mrow id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.2" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.cmml"><mn id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.2" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.2.cmml">890</mn><mo id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.1.cmml">−</mo><mn id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.3.cmml">4</mn></mrow><mo stretchy="false" id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.3" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.3.cmml">=</mo><mn id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.4" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.4.cmml">0.009</mn><mo id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.5" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.5.cmml">&lt;</mo><mrow id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6.cmml"><mn id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6.2" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6.2.cmml">1</mn><mo id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6.1" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p2.1.m1.1b"><apply id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1"><and id="S3.SS0.SSS0.Px1.p2.1.m1.1.1a.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1"></and><apply id="S3.SS0.SSS0.Px1.p2.1.m1.1.1b.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1"><eq id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.3"></eq><apply id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1"><divide id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.2"></divide><cn type="integer" id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.3">8</cn><apply id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1"><minus id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.1"></minus><cn type="integer" id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.2">890</cn><cn type="integer" id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.1.1.1.1.3">4</cn></apply></apply><cn type="float" id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.4.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.4">0.009</cn></apply><apply id="S3.SS0.SSS0.Px1.p2.1.m1.1.1c.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1"><lt id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.5.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.5"></lt><share href="#S3.SS0.SSS0.Px1.p2.1.m1.1.1.4.cmml" id="S3.SS0.SSS0.Px1.p2.1.m1.1.1d.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1"></share><apply id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6"><csymbol cd="latexml" id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6.1.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6.1">percent</csymbol><cn type="integer" id="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6.2.cmml" xref="S3.SS0.SSS0.Px1.p2.1.m1.1.1.6.2">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p2.1.m1.1c">8/(890-4)=0.009&lt;1\%</annotation></semantics></math> and <math id="S3.SS0.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="1/(765-5)=0.001&lt;1\%" display="inline"><semantics id="S3.SS0.SSS0.Px1.p2.2.m2.1a"><mrow id="S3.SS0.SSS0.Px1.p2.2.m2.1.1" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.cmml"><mrow id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.cmml"><mn id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.3" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.3.cmml">1</mn><mo id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.2" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.2.cmml">/</mo><mrow id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.2" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.cmml"><mn id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.2" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.2.cmml">765</mn><mo id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.1.cmml">−</mo><mn id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.3.cmml">5</mn></mrow><mo stretchy="false" id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.3" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.3.cmml">=</mo><mn id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.4" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.4.cmml">0.001</mn><mo id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.5" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.5.cmml">&lt;</mo><mrow id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6.cmml"><mn id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6.2" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6.2.cmml">1</mn><mo id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6.1" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p2.2.m2.1b"><apply id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1"><and id="S3.SS0.SSS0.Px1.p2.2.m2.1.1a.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1"></and><apply id="S3.SS0.SSS0.Px1.p2.2.m2.1.1b.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1"><eq id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.3"></eq><apply id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1"><divide id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.2"></divide><cn type="integer" id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.3">1</cn><apply id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1"><minus id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.1"></minus><cn type="integer" id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.2">765</cn><cn type="integer" id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.1.1.1.1.3">5</cn></apply></apply><cn type="float" id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.4.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.4">0.001</cn></apply><apply id="S3.SS0.SSS0.Px1.p2.2.m2.1.1c.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1"><lt id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.5.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.5"></lt><share href="#S3.SS0.SSS0.Px1.p2.2.m2.1.1.4.cmml" id="S3.SS0.SSS0.Px1.p2.2.m2.1.1d.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1"></share><apply id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6"><csymbol cd="latexml" id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6.1.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6.1">percent</csymbol><cn type="integer" id="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6.2.cmml" xref="S3.SS0.SSS0.Px1.p2.2.m2.1.1.6.2">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p2.2.m2.1c">1/(765-5)=0.001&lt;1\%</annotation></semantics></math> with 3 test cases. The upshot is that the potential for noise is significantly less than that of most naturally annotated datasets.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Experimental Setup</h3>

<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Models.</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p">We use GPT-2 <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite>, GPT-3 <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib5" title="" class="ltx_ref">2020</a>)</cite>, and GPT-Neo <cite class="ltx_cite ltx_citemacro_citep">(Black et al., <a href="#bib.bib4" title="" class="ltx_ref">2021</a>)</cite> models. The GPT architecture is especially appropriate for text generation because it is autoregressive. However, GPT-2 was not pretrained on code, so we pretrain it on GitHub as described in the next paragraph. Anecdotal evidence indicates that GPT-3 can generate code. To determine the extent of its code generation ability, we use the ‘davinci’ (Instruct series) model, the largest publicly available model speculated to have 175 billion parameters. Finally, GPT-Neo has an architecture similar to GPT-3, and it was pretrained on the Pile <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib9" title="" class="ltx_ref">2020</a>)</cite> which includes GitHub. Unlike GPT-3, GPT-Neo’s weights are publicly available, hence we are able to fine-tune it with APPS.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">GPT-2 Pretraining.</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">Since GPT-2 was trained on natural language and not code, we collected GitHub code to further pretrain GPT-2. GitHub repositories with fewer than one star were filtered out.
While Neo’s GitHub pretraining data did <em id="S4.SS1.SSS0.Px2.p1.1.1" class="ltx_emph ltx_font_italic">not</em> undergo an APPS data decontamination process, our GPT-2 models are trained on decontaminated data.
Specifically, all repositories matching certain keywords that would suggest overlap with common programming exercises were removed. We provide the list of keywords in the Supplementary Materials. We also discard any GitHub code that contains functions with the same signatures as functions in the starter code in many of our APPS problems. This leaves us with 30 GB of Python code. To improve the efficiency of pretraining, we process all Python code in the pretraining dataset by converting from spaces to tabs, which saves the character conversion when running model tokenizers.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Fine-tuning.</h4>

<div id="S4.SS1.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p1.1" class="ltx_p">During fine-tuning with APPS, the objective is to predict the entire code solution, given both the English text problem statement and the problem format (call-based format or standard input format). For problems with starter code, we exclude the starter code from the training loss.</p>
</div>
<div id="S4.SS1.SSS0.Px3.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p2.4" class="ltx_p">Across pretraining and fine-tuning, we use the AdamW optimizer <cite class="ltx_cite ltx_citemacro_citep">(Loshchilov and Hutter, <a href="#bib.bib22" title="" class="ltx_ref">2019</a>)</cite>, a batch size of <math id="S4.SS1.SSS0.Px3.p2.1.m1.1" class="ltx_Math" alttext="256" display="inline"><semantics id="S4.SS1.SSS0.Px3.p2.1.m1.1a"><mn id="S4.SS1.SSS0.Px3.p2.1.m1.1.1" xref="S4.SS1.SSS0.Px3.p2.1.m1.1.1.cmml">256</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p2.1.m1.1b"><cn type="integer" id="S4.SS1.SSS0.Px3.p2.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px3.p2.1.m1.1.1">256</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p2.1.m1.1c">256</annotation></semantics></math>, and a weight decay of <math id="S4.SS1.SSS0.Px3.p2.2.m2.1" class="ltx_Math" alttext="0.05" display="inline"><semantics id="S4.SS1.SSS0.Px3.p2.2.m2.1a"><mn id="S4.SS1.SSS0.Px3.p2.2.m2.1.1" xref="S4.SS1.SSS0.Px3.p2.2.m2.1.1.cmml">0.05</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p2.2.m2.1b"><cn type="float" id="S4.SS1.SSS0.Px3.p2.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px3.p2.2.m2.1.1">0.05</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p2.2.m2.1c">0.05</annotation></semantics></math>. We fine-tune for <math id="S4.SS1.SSS0.Px3.p2.3.m3.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S4.SS1.SSS0.Px3.p2.3.m3.1a"><mn id="S4.SS1.SSS0.Px3.p2.3.m3.1.1" xref="S4.SS1.SSS0.Px3.p2.3.m3.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p2.3.m3.1b"><cn type="integer" id="S4.SS1.SSS0.Px3.p2.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px3.p2.3.m3.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p2.3.m3.1c">10</annotation></semantics></math> epochs. We use DeepSpeed and its implementation of the ZeRO optimizer to reduce memory consumption while training large models <cite class="ltx_cite ltx_citemacro_citep">(Rasley et al., <a href="#bib.bib29" title="" class="ltx_ref">2020</a>; Rajbhandari et al., <a href="#bib.bib28" title="" class="ltx_ref">2020</a>)</cite>. Unless otherwise specified, we use the default HuggingFace generation parameters, except that we use beam search with a beam size of <math id="S4.SS1.SSS0.Px3.p2.4.m4.1" class="ltx_Math" alttext="5" display="inline"><semantics id="S4.SS1.SSS0.Px3.p2.4.m4.1a"><mn id="S4.SS1.SSS0.Px3.p2.4.m4.1.1" xref="S4.SS1.SSS0.Px3.p2.4.m4.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p2.4.m4.1b"><cn type="integer" id="S4.SS1.SSS0.Px3.p2.4.m4.1.1.cmml" xref="S4.SS1.SSS0.Px3.p2.4.m4.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p2.4.m4.1c">5</annotation></semantics></math>. Models are fine-tuned on 8 A100 GPUs.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Metrics</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.1" class="ltx_p">To obtain a comprehensive evaluation of code generation ability, we use the large bank of test cases and ground-truth solutions provided with APPS. Test cases allow for <em id="S4.SS2.p1.1.1" class="ltx_emph ltx_font_italic">automatic</em> evaluation, even though the the space of possible programs can be combinatorially large. Therefore, unlike many other text generation tasks, manual analysis is not necessary. We aggregate the generated code’s performance on test cases with two metrics, “test case average” and “strict accuracy.”</p>
</div>
<div id="S4.SS2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.p2.6" class="ltx_p"><span id="S4.SS2.p2.6.1" class="ltx_text ltx_font_bold">Test Case Average.</span> We compute the average fraction of test cases passed.
Concretely, let the number of problems in the test set be <math id="S4.SS2.p2.1.m1.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S4.SS2.p2.1.m1.1a"><mi id="S4.SS2.p2.1.m1.1.1" xref="S4.SS2.p2.1.m1.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.1.m1.1b"><ci id="S4.SS2.p2.1.m1.1.1.cmml" xref="S4.SS2.p2.1.m1.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.1.m1.1c">P</annotation></semantics></math>. For a given problem <math id="S4.SS2.p2.2.m2.1" class="ltx_Math" alttext="p" display="inline"><semantics id="S4.SS2.p2.2.m2.1a"><mi id="S4.SS2.p2.2.m2.1.1" xref="S4.SS2.p2.2.m2.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.2.m2.1b"><ci id="S4.SS2.p2.2.m2.1.1.cmml" xref="S4.SS2.p2.2.m2.1.1">𝑝</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.2.m2.1c">p</annotation></semantics></math>, let the code generated to solve problem <math id="S4.SS2.p2.3.m3.1" class="ltx_Math" alttext="p" display="inline"><semantics id="S4.SS2.p2.3.m3.1a"><mi id="S4.SS2.p2.3.m3.1.1" xref="S4.SS2.p2.3.m3.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.3.m3.1b"><ci id="S4.SS2.p2.3.m3.1.1.cmml" xref="S4.SS2.p2.3.m3.1.1">𝑝</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.3.m3.1c">p</annotation></semantics></math> be denoted <math id="S4.SS2.p2.4.m4.1" class="ltx_Math" alttext="\langle\texttt{code}_{p}\rangle" display="inline"><semantics id="S4.SS2.p2.4.m4.1a"><mrow id="S4.SS2.p2.4.m4.1.1.1" xref="S4.SS2.p2.4.m4.1.1.2.cmml"><mo stretchy="false" id="S4.SS2.p2.4.m4.1.1.1.2" xref="S4.SS2.p2.4.m4.1.1.2.1.cmml">⟨</mo><msub id="S4.SS2.p2.4.m4.1.1.1.1" xref="S4.SS2.p2.4.m4.1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.SS2.p2.4.m4.1.1.1.1.2" xref="S4.SS2.p2.4.m4.1.1.1.1.2a.cmml">code</mtext><mi id="S4.SS2.p2.4.m4.1.1.1.1.3" xref="S4.SS2.p2.4.m4.1.1.1.1.3.cmml">p</mi></msub><mo stretchy="false" id="S4.SS2.p2.4.m4.1.1.1.3" xref="S4.SS2.p2.4.m4.1.1.2.1.cmml">⟩</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.4.m4.1b"><apply id="S4.SS2.p2.4.m4.1.1.2.cmml" xref="S4.SS2.p2.4.m4.1.1.1"><csymbol cd="latexml" id="S4.SS2.p2.4.m4.1.1.2.1.cmml" xref="S4.SS2.p2.4.m4.1.1.1.2">delimited-⟨⟩</csymbol><apply id="S4.SS2.p2.4.m4.1.1.1.1.cmml" xref="S4.SS2.p2.4.m4.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p2.4.m4.1.1.1.1.1.cmml" xref="S4.SS2.p2.4.m4.1.1.1.1">subscript</csymbol><ci id="S4.SS2.p2.4.m4.1.1.1.1.2a.cmml" xref="S4.SS2.p2.4.m4.1.1.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S4.SS2.p2.4.m4.1.1.1.1.2.cmml" xref="S4.SS2.p2.4.m4.1.1.1.1.2">code</mtext></ci><ci id="S4.SS2.p2.4.m4.1.1.1.1.3.cmml" xref="S4.SS2.p2.4.m4.1.1.1.1.3">𝑝</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.4.m4.1c">\langle\texttt{code}_{p}\rangle</annotation></semantics></math>, and set of test cases for problem <math id="S4.SS2.p2.5.m5.1" class="ltx_Math" alttext="p" display="inline"><semantics id="S4.SS2.p2.5.m5.1a"><mi id="S4.SS2.p2.5.m5.1.1" xref="S4.SS2.p2.5.m5.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.5.m5.1b"><ci id="S4.SS2.p2.5.m5.1.1.cmml" xref="S4.SS2.p2.5.m5.1.1">𝑝</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.5.m5.1c">p</annotation></semantics></math> be <math id="S4.SS2.p2.6.m6.5" class="ltx_Math" alttext="\{(x_{p,c},\,y_{p,c})\}_{c=1}^{C_{p}}" display="inline"><semantics id="S4.SS2.p2.6.m6.5a"><msubsup id="S4.SS2.p2.6.m6.5.5" xref="S4.SS2.p2.6.m6.5.5.cmml"><mrow id="S4.SS2.p2.6.m6.5.5.1.1.1" xref="S4.SS2.p2.6.m6.5.5.1.1.2.cmml"><mo stretchy="false" id="S4.SS2.p2.6.m6.5.5.1.1.1.2" xref="S4.SS2.p2.6.m6.5.5.1.1.2.cmml">{</mo><mrow id="S4.SS2.p2.6.m6.5.5.1.1.1.1.2" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.3.cmml"><mo stretchy="false" id="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.3" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.3.cmml">(</mo><msub id="S4.SS2.p2.6.m6.5.5.1.1.1.1.1.1" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.1.1.cmml"><mi id="S4.SS2.p2.6.m6.5.5.1.1.1.1.1.1.2" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.1.1.2.cmml">x</mi><mrow id="S4.SS2.p2.6.m6.2.2.2.4" xref="S4.SS2.p2.6.m6.2.2.2.3.cmml"><mi id="S4.SS2.p2.6.m6.1.1.1.1" xref="S4.SS2.p2.6.m6.1.1.1.1.cmml">p</mi><mo id="S4.SS2.p2.6.m6.2.2.2.4.1" xref="S4.SS2.p2.6.m6.2.2.2.3.cmml">,</mo><mi id="S4.SS2.p2.6.m6.2.2.2.2" xref="S4.SS2.p2.6.m6.2.2.2.2.cmml">c</mi></mrow></msub><mo rspace="0.337em" id="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.4" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.3.cmml">,</mo><msub id="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.2" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.2.cmml"><mi id="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.2.2" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.2.2.cmml">y</mi><mrow id="S4.SS2.p2.6.m6.4.4.2.4" xref="S4.SS2.p2.6.m6.4.4.2.3.cmml"><mi id="S4.SS2.p2.6.m6.3.3.1.1" xref="S4.SS2.p2.6.m6.3.3.1.1.cmml">p</mi><mo id="S4.SS2.p2.6.m6.4.4.2.4.1" xref="S4.SS2.p2.6.m6.4.4.2.3.cmml">,</mo><mi id="S4.SS2.p2.6.m6.4.4.2.2" xref="S4.SS2.p2.6.m6.4.4.2.2.cmml">c</mi></mrow></msub><mo stretchy="false" id="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.5" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.3.cmml">)</mo></mrow><mo stretchy="false" id="S4.SS2.p2.6.m6.5.5.1.1.1.3" xref="S4.SS2.p2.6.m6.5.5.1.1.2.cmml">}</mo></mrow><mrow id="S4.SS2.p2.6.m6.5.5.1.3" xref="S4.SS2.p2.6.m6.5.5.1.3.cmml"><mi id="S4.SS2.p2.6.m6.5.5.1.3.2" xref="S4.SS2.p2.6.m6.5.5.1.3.2.cmml">c</mi><mo id="S4.SS2.p2.6.m6.5.5.1.3.1" xref="S4.SS2.p2.6.m6.5.5.1.3.1.cmml">=</mo><mn id="S4.SS2.p2.6.m6.5.5.1.3.3" xref="S4.SS2.p2.6.m6.5.5.1.3.3.cmml">1</mn></mrow><msub id="S4.SS2.p2.6.m6.5.5.3" xref="S4.SS2.p2.6.m6.5.5.3.cmml"><mi id="S4.SS2.p2.6.m6.5.5.3.2" xref="S4.SS2.p2.6.m6.5.5.3.2.cmml">C</mi><mi id="S4.SS2.p2.6.m6.5.5.3.3" xref="S4.SS2.p2.6.m6.5.5.3.3.cmml">p</mi></msub></msubsup><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.6.m6.5b"><apply id="S4.SS2.p2.6.m6.5.5.cmml" xref="S4.SS2.p2.6.m6.5.5"><csymbol cd="ambiguous" id="S4.SS2.p2.6.m6.5.5.2.cmml" xref="S4.SS2.p2.6.m6.5.5">superscript</csymbol><apply id="S4.SS2.p2.6.m6.5.5.1.cmml" xref="S4.SS2.p2.6.m6.5.5"><csymbol cd="ambiguous" id="S4.SS2.p2.6.m6.5.5.1.2.cmml" xref="S4.SS2.p2.6.m6.5.5">subscript</csymbol><set id="S4.SS2.p2.6.m6.5.5.1.1.2.cmml" xref="S4.SS2.p2.6.m6.5.5.1.1.1"><interval closure="open" id="S4.SS2.p2.6.m6.5.5.1.1.1.1.3.cmml" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.2"><apply id="S4.SS2.p2.6.m6.5.5.1.1.1.1.1.1.cmml" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p2.6.m6.5.5.1.1.1.1.1.1.1.cmml" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.1.1">subscript</csymbol><ci id="S4.SS2.p2.6.m6.5.5.1.1.1.1.1.1.2.cmml" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.1.1.2">𝑥</ci><list id="S4.SS2.p2.6.m6.2.2.2.3.cmml" xref="S4.SS2.p2.6.m6.2.2.2.4"><ci id="S4.SS2.p2.6.m6.1.1.1.1.cmml" xref="S4.SS2.p2.6.m6.1.1.1.1">𝑝</ci><ci id="S4.SS2.p2.6.m6.2.2.2.2.cmml" xref="S4.SS2.p2.6.m6.2.2.2.2">𝑐</ci></list></apply><apply id="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.2.cmml" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.2.1.cmml" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.2">subscript</csymbol><ci id="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.2.2.cmml" xref="S4.SS2.p2.6.m6.5.5.1.1.1.1.2.2.2">𝑦</ci><list id="S4.SS2.p2.6.m6.4.4.2.3.cmml" xref="S4.SS2.p2.6.m6.4.4.2.4"><ci id="S4.SS2.p2.6.m6.3.3.1.1.cmml" xref="S4.SS2.p2.6.m6.3.3.1.1">𝑝</ci><ci id="S4.SS2.p2.6.m6.4.4.2.2.cmml" xref="S4.SS2.p2.6.m6.4.4.2.2">𝑐</ci></list></apply></interval></set><apply id="S4.SS2.p2.6.m6.5.5.1.3.cmml" xref="S4.SS2.p2.6.m6.5.5.1.3"><eq id="S4.SS2.p2.6.m6.5.5.1.3.1.cmml" xref="S4.SS2.p2.6.m6.5.5.1.3.1"></eq><ci id="S4.SS2.p2.6.m6.5.5.1.3.2.cmml" xref="S4.SS2.p2.6.m6.5.5.1.3.2">𝑐</ci><cn type="integer" id="S4.SS2.p2.6.m6.5.5.1.3.3.cmml" xref="S4.SS2.p2.6.m6.5.5.1.3.3">1</cn></apply></apply><apply id="S4.SS2.p2.6.m6.5.5.3.cmml" xref="S4.SS2.p2.6.m6.5.5.3"><csymbol cd="ambiguous" id="S4.SS2.p2.6.m6.5.5.3.1.cmml" xref="S4.SS2.p2.6.m6.5.5.3">subscript</csymbol><ci id="S4.SS2.p2.6.m6.5.5.3.2.cmml" xref="S4.SS2.p2.6.m6.5.5.3.2">𝐶</ci><ci id="S4.SS2.p2.6.m6.5.5.3.3.cmml" xref="S4.SS2.p2.6.m6.5.5.3.3">𝑝</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.6.m6.5c">\{(x_{p,c},\,y_{p,c})\}_{c=1}^{C_{p}}</annotation></semantics></math>. Then the test case average is</p>
<table id="S4.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.Ex1.m1.5" class="ltx_Math" alttext="\frac{1}{P}\sum_{p=1}^{P}\frac{1}{C_{p}}\sum_{c=1}^{C_{p}}\mathds{1}\{\texttt{eval}(\langle\texttt{code}_{p}\rangle,x_{p,c})=y_{p,c}\}." display="block"><semantics id="S4.Ex1.m1.5a"><mrow id="S4.Ex1.m1.5.5.1" xref="S4.Ex1.m1.5.5.1.1.cmml"><mrow id="S4.Ex1.m1.5.5.1.1" xref="S4.Ex1.m1.5.5.1.1.cmml"><mfrac id="S4.Ex1.m1.5.5.1.1.3" xref="S4.Ex1.m1.5.5.1.1.3.cmml"><mn id="S4.Ex1.m1.5.5.1.1.3.2" xref="S4.Ex1.m1.5.5.1.1.3.2.cmml">1</mn><mi id="S4.Ex1.m1.5.5.1.1.3.3" xref="S4.Ex1.m1.5.5.1.1.3.3.cmml">P</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.Ex1.m1.5.5.1.1.2" xref="S4.Ex1.m1.5.5.1.1.2.cmml">​</mo><mrow id="S4.Ex1.m1.5.5.1.1.1" xref="S4.Ex1.m1.5.5.1.1.1.cmml"><munderover id="S4.Ex1.m1.5.5.1.1.1.2" xref="S4.Ex1.m1.5.5.1.1.1.2.cmml"><mo movablelimits="false" id="S4.Ex1.m1.5.5.1.1.1.2.2.2" xref="S4.Ex1.m1.5.5.1.1.1.2.2.2.cmml">∑</mo><mrow id="S4.Ex1.m1.5.5.1.1.1.2.2.3" xref="S4.Ex1.m1.5.5.1.1.1.2.2.3.cmml"><mi id="S4.Ex1.m1.5.5.1.1.1.2.2.3.2" xref="S4.Ex1.m1.5.5.1.1.1.2.2.3.2.cmml">p</mi><mo id="S4.Ex1.m1.5.5.1.1.1.2.2.3.1" xref="S4.Ex1.m1.5.5.1.1.1.2.2.3.1.cmml">=</mo><mn id="S4.Ex1.m1.5.5.1.1.1.2.2.3.3" xref="S4.Ex1.m1.5.5.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S4.Ex1.m1.5.5.1.1.1.2.3" xref="S4.Ex1.m1.5.5.1.1.1.2.3.cmml">P</mi></munderover><mrow id="S4.Ex1.m1.5.5.1.1.1.1" xref="S4.Ex1.m1.5.5.1.1.1.1.cmml"><mfrac id="S4.Ex1.m1.5.5.1.1.1.1.3" xref="S4.Ex1.m1.5.5.1.1.1.1.3.cmml"><mn id="S4.Ex1.m1.5.5.1.1.1.1.3.2" xref="S4.Ex1.m1.5.5.1.1.1.1.3.2.cmml">1</mn><msub id="S4.Ex1.m1.5.5.1.1.1.1.3.3" xref="S4.Ex1.m1.5.5.1.1.1.1.3.3.cmml"><mi id="S4.Ex1.m1.5.5.1.1.1.1.3.3.2" xref="S4.Ex1.m1.5.5.1.1.1.1.3.3.2.cmml">C</mi><mi id="S4.Ex1.m1.5.5.1.1.1.1.3.3.3" xref="S4.Ex1.m1.5.5.1.1.1.1.3.3.3.cmml">p</mi></msub></mfrac><mo lspace="0em" rspace="0em" id="S4.Ex1.m1.5.5.1.1.1.1.2" xref="S4.Ex1.m1.5.5.1.1.1.1.2.cmml">​</mo><mrow id="S4.Ex1.m1.5.5.1.1.1.1.1" xref="S4.Ex1.m1.5.5.1.1.1.1.1.cmml"><munderover id="S4.Ex1.m1.5.5.1.1.1.1.1.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.cmml"><mi id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.2.cmml">c</mi><mo id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.1" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.1.cmml">=</mo><mn id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.3.cmml">1</mn></mrow><msub id="S4.Ex1.m1.5.5.1.1.1.1.1.2.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.cmml"><mi id="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.2.cmml">C</mi><mi id="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.3.cmml">p</mi></msub></munderover><mrow id="S4.Ex1.m1.5.5.1.1.1.1.1.1" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.cmml"><mn id="S4.Ex1.m1.5.5.1.1.1.1.1.1.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.3.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.2.cmml">{</mo><mrow id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.4" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.4a.cmml">eval</mtext><mo lspace="0em" rspace="0em" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.3.cmml">​</mo><mrow id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.3.cmml">(</mo><mrow id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">⟨</mo><msub id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2a.cmml">code</mtext><mi id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">p</mi></msub><mo stretchy="false" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">⟩</mo></mrow><mo id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.4" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.3.cmml">,</mo><msub id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml"><mi id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.2.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.2.2.cmml">x</mi><mrow id="S4.Ex1.m1.2.2.2.4" xref="S4.Ex1.m1.2.2.2.3.cmml"><mi id="S4.Ex1.m1.1.1.1.1" xref="S4.Ex1.m1.1.1.1.1.cmml">p</mi><mo id="S4.Ex1.m1.2.2.2.4.1" xref="S4.Ex1.m1.2.2.2.3.cmml">,</mo><mi id="S4.Ex1.m1.2.2.2.2" xref="S4.Ex1.m1.2.2.2.2.cmml">c</mi></mrow></msub><mo stretchy="false" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.5" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.3.cmml">=</mo><msub id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.4" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.4.cmml"><mi id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.4.2" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.4.2.cmml">y</mi><mrow id="S4.Ex1.m1.4.4.2.4" xref="S4.Ex1.m1.4.4.2.3.cmml"><mi id="S4.Ex1.m1.3.3.1.1" xref="S4.Ex1.m1.3.3.1.1.cmml">p</mi><mo id="S4.Ex1.m1.4.4.2.4.1" xref="S4.Ex1.m1.4.4.2.3.cmml">,</mo><mi id="S4.Ex1.m1.4.4.2.2" xref="S4.Ex1.m1.4.4.2.2.cmml">c</mi></mrow></msub></mrow><mo stretchy="false" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.3" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.2.cmml">}</mo></mrow></mrow></mrow></mrow></mrow></mrow><mo lspace="0em" id="S4.Ex1.m1.5.5.1.2" xref="S4.Ex1.m1.5.5.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.Ex1.m1.5b"><apply id="S4.Ex1.m1.5.5.1.1.cmml" xref="S4.Ex1.m1.5.5.1"><times id="S4.Ex1.m1.5.5.1.1.2.cmml" xref="S4.Ex1.m1.5.5.1.1.2"></times><apply id="S4.Ex1.m1.5.5.1.1.3.cmml" xref="S4.Ex1.m1.5.5.1.1.3"><divide id="S4.Ex1.m1.5.5.1.1.3.1.cmml" xref="S4.Ex1.m1.5.5.1.1.3"></divide><cn type="integer" id="S4.Ex1.m1.5.5.1.1.3.2.cmml" xref="S4.Ex1.m1.5.5.1.1.3.2">1</cn><ci id="S4.Ex1.m1.5.5.1.1.3.3.cmml" xref="S4.Ex1.m1.5.5.1.1.3.3">𝑃</ci></apply><apply id="S4.Ex1.m1.5.5.1.1.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1"><apply id="S4.Ex1.m1.5.5.1.1.1.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex1.m1.5.5.1.1.1.2.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.2">superscript</csymbol><apply id="S4.Ex1.m1.5.5.1.1.1.2.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex1.m1.5.5.1.1.1.2.2.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.2">subscript</csymbol><sum id="S4.Ex1.m1.5.5.1.1.1.2.2.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.2.2.2"></sum><apply id="S4.Ex1.m1.5.5.1.1.1.2.2.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.2.2.3"><eq id="S4.Ex1.m1.5.5.1.1.1.2.2.3.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.2.2.3.1"></eq><ci id="S4.Ex1.m1.5.5.1.1.1.2.2.3.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.2.2.3.2">𝑝</ci><cn type="integer" id="S4.Ex1.m1.5.5.1.1.1.2.2.3.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="S4.Ex1.m1.5.5.1.1.1.2.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.2.3">𝑃</ci></apply><apply id="S4.Ex1.m1.5.5.1.1.1.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1"><times id="S4.Ex1.m1.5.5.1.1.1.1.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.2"></times><apply id="S4.Ex1.m1.5.5.1.1.1.1.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.3"><divide id="S4.Ex1.m1.5.5.1.1.1.1.3.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.3"></divide><cn type="integer" id="S4.Ex1.m1.5.5.1.1.1.1.3.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.3.2">1</cn><apply id="S4.Ex1.m1.5.5.1.1.1.1.3.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S4.Ex1.m1.5.5.1.1.1.1.3.3.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.3.3">subscript</csymbol><ci id="S4.Ex1.m1.5.5.1.1.1.1.3.3.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.3.3.2">𝐶</ci><ci id="S4.Ex1.m1.5.5.1.1.1.1.3.3.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.3.3.3">𝑝</ci></apply></apply><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1"><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex1.m1.5.5.1.1.1.1.1.2.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2">superscript</csymbol><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2">subscript</csymbol><sum id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.2"></sum><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3"><eq id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.1"></eq><ci id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.2">𝑐</ci><cn type="integer" id="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.2.3.3">1</cn></apply></apply><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.3">subscript</csymbol><ci id="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.2">𝐶</ci><ci id="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.2.3.3">𝑝</ci></apply></apply><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1"><times id="S4.Ex1.m1.5.5.1.1.1.1.1.1.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.2"></times><cn type="integer" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.3">1</cn><set id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1"><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1"><eq id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.3"></eq><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2"><times id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.3"></times><ci id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.4a.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.4"><mtext class="ltx_mathvariant_monospace" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.4.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.4">eval</mtext></ci><interval closure="open" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2"><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">delimited-⟨⟩</csymbol><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2a.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">code</mtext></ci><ci id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑝</ci></apply></apply><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.2.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.2.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.2.2.2">𝑥</ci><list id="S4.Ex1.m1.2.2.2.3.cmml" xref="S4.Ex1.m1.2.2.2.4"><ci id="S4.Ex1.m1.1.1.1.1.cmml" xref="S4.Ex1.m1.1.1.1.1">𝑝</ci><ci id="S4.Ex1.m1.2.2.2.2.cmml" xref="S4.Ex1.m1.2.2.2.2">𝑐</ci></list></apply></interval></apply><apply id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.4.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.4.1.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.4">subscript</csymbol><ci id="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.4.2.cmml" xref="S4.Ex1.m1.5.5.1.1.1.1.1.1.1.1.1.4.2">𝑦</ci><list id="S4.Ex1.m1.4.4.2.3.cmml" xref="S4.Ex1.m1.4.4.2.4"><ci id="S4.Ex1.m1.3.3.1.1.cmml" xref="S4.Ex1.m1.3.3.1.1">𝑝</ci><ci id="S4.Ex1.m1.4.4.2.2.cmml" xref="S4.Ex1.m1.4.4.2.2">𝑐</ci></list></apply></apply></set></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Ex1.m1.5c">\frac{1}{P}\sum_{p=1}^{P}\frac{1}{C_{p}}\sum_{c=1}^{C_{p}}\mathds{1}\{\texttt{eval}(\langle\texttt{code}_{p}\rangle,x_{p,c})=y_{p,c}\}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S4.SS2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS2.p3.1" class="ltx_p">Oftentimes, solutions can successfully pass a subset of the test cases but not cover every corner case. This allows for less stringent model evaluation, as strict accuracy may currently obscure model improvements.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<table id="S4.T2.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T2.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="S4.T2.1.1.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;" colspan="4">Test Case Average</td>
<td id="S4.T2.1.1.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;" colspan="4">Strict Accuracy</td>
</tr>
<tr id="S4.T2.1.2" class="ltx_tr">
<td id="S4.T2.1.2.1" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">Model</td>
<td id="S4.T2.1.2.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Introductory</td>
<td id="S4.T2.1.2.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Interview</td>
<td id="S4.T2.1.2.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Competitive</td>
<td id="S4.T2.1.2.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:2.0pt;padding-right:2.0pt;">Average</td>
<td id="S4.T2.1.2.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Introductory</td>
<td id="S4.T2.1.2.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Interview</td>
<td id="S4.T2.1.2.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Competition</td>
<td id="S4.T2.1.2.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Average</td>
</tr>
<tr id="S4.T2.1.3" class="ltx_tr">
<td id="S4.T2.1.3.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">GPT-2 0.1B</td>
<td id="S4.T2.1.3.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">5.64</td>
<td id="S4.T2.1.3.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">6.93</td>
<td id="S4.T2.1.3.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">4.37</td>
<td id="S4.T2.1.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">6.16</td>
<td id="S4.T2.1.3.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">1.00</td>
<td id="S4.T2.1.3.7" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">0.33</td>
<td id="S4.T2.1.3.8" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">0.00</td>
<td id="S4.T2.1.3.9" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">0.40</td>
</tr>
<tr id="S4.T2.1.4" class="ltx_tr">
<td id="S4.T2.1.4.1" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">GPT-2 1.5B</td>
<td id="S4.T2.1.4.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">7.40</td>
<td id="S4.T2.1.4.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">9.11</td>
<td id="S4.T2.1.4.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">5.05</td>
<td id="S4.T2.1.4.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:2.0pt;padding-right:2.0pt;">7.96</td>
<td id="S4.T2.1.4.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">1.30</td>
<td id="S4.T2.1.4.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.70</td>
<td id="S4.T2.1.4.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.00</td>
<td id="S4.T2.1.4.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.68</td>
</tr>
<tr id="S4.T2.1.5" class="ltx_tr">
<td id="S4.T2.1.5.1" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">GPT-Neo 2.7B</td>
<td id="S4.T2.1.5.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">14.68</td>
<td id="S4.T2.1.5.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">9.85</td>
<td id="S4.T2.1.5.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">6.54</td>
<td id="S4.T2.1.5.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:2.0pt;padding-right:2.0pt;">10.15</td>
<td id="S4.T2.1.5.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">3.90</td>
<td id="S4.T2.1.5.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.57</td>
<td id="S4.T2.1.5.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.00</td>
<td id="S4.T2.1.5.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">1.12</td>
</tr>
<tr id="S4.T2.1.6" class="ltx_tr">
<td id="S4.T2.1.6.1" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">GPT-3 175B</td>
<td id="S4.T2.1.6.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.57</td>
<td id="S4.T2.1.6.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.65</td>
<td id="S4.T2.1.6.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.21</td>
<td id="S4.T2.1.6.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:2.0pt;padding-right:2.0pt;">0.55</td>
<td id="S4.T2.1.6.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.20</td>
<td id="S4.T2.1.6.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.03</td>
<td id="S4.T2.1.6.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.00</td>
<td id="S4.T2.1.6.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.06</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Average percentage of test cases passed and strict accuracy for each model and difficulty level. All values are percentages. Note ‘0.1B’ indicates the number of model parameters in billions. GPT-3 is a <em id="S4.T2.3.1" class="ltx_emph ltx_font_italic">few-shot</em> model and not fine-tuned, unlike the other models. GPT-Neo does best and attains approximately 4% strict accuracy on Introductory problems, and for these problems it passes approximately 15% of the test cases.</figcaption>
</figure>
<div id="S4.SS2.p4" class="ltx_para ltx_noindent">
<p id="S4.SS2.p4.1" class="ltx_p"><span id="S4.SS2.p4.1.1" class="ltx_text ltx_font_bold">Strict Accuracy.</span> Eventually, generated solutions should pass all test cases including corner cases.
To compute the strict accuracy which requires programs pass every test case, we run the code generated by the model on every test case of every problem. Strict accuracy is then computed by taking the number of solutions passing every test case divided by the total number of exercises. Using the notation from before, we can write the strict accuracy as <math id="S4.SS2.p4.1.m1.5" class="ltx_Math" alttext="\frac{1}{P}\sum_{p=1}^{P}\prod_{c=1}^{C_{p}}\mathds{1}\{\texttt{eval}(\langle\texttt{code}_{p}\rangle,x_{p,c})=y_{p,c}\}." display="inline"><semantics id="S4.SS2.p4.1.m1.5a"><mrow id="S4.SS2.p4.1.m1.5.5.1" xref="S4.SS2.p4.1.m1.5.5.1.1.cmml"><mrow id="S4.SS2.p4.1.m1.5.5.1.1" xref="S4.SS2.p4.1.m1.5.5.1.1.cmml"><mfrac id="S4.SS2.p4.1.m1.5.5.1.1.3" xref="S4.SS2.p4.1.m1.5.5.1.1.3.cmml"><mn id="S4.SS2.p4.1.m1.5.5.1.1.3.2" xref="S4.SS2.p4.1.m1.5.5.1.1.3.2.cmml">1</mn><mi id="S4.SS2.p4.1.m1.5.5.1.1.3.3" xref="S4.SS2.p4.1.m1.5.5.1.1.3.3.cmml">P</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.SS2.p4.1.m1.5.5.1.1.2" xref="S4.SS2.p4.1.m1.5.5.1.1.2.cmml">​</mo><mrow id="S4.SS2.p4.1.m1.5.5.1.1.1" xref="S4.SS2.p4.1.m1.5.5.1.1.1.cmml"><msubsup id="S4.SS2.p4.1.m1.5.5.1.1.1.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.cmml"><mo rspace="0em" id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.2.cmml">∑</mo><mrow id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.cmml"><mi id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.2.cmml">p</mi><mo id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.1" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.1.cmml">=</mo><mn id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S4.SS2.p4.1.m1.5.5.1.1.1.2.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.3.cmml">P</mi></msubsup><mrow id="S4.SS2.p4.1.m1.5.5.1.1.1.1" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.cmml"><msubsup id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.cmml"><mo id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.2.cmml">∏</mo><mrow id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.cmml"><mi id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.2.cmml">c</mi><mo id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.1" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.1.cmml">=</mo><mn id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.3.cmml">1</mn></mrow><msub id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.cmml"><mi id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.2.cmml">C</mi><mi id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.3.cmml">p</mi></msub></msubsup><mrow id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.cmml"><mn id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.3.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.2.cmml">​</mo><mrow id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.2.cmml">{</mo><mrow id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.cmml"><mrow id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.4" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.4a.cmml">eval</mtext><mo lspace="0em" rspace="0em" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.3.cmml">​</mo><mrow id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.3.cmml">(</mo><mrow id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">⟨</mo><msub id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2a.cmml">code</mtext><mi id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">p</mi></msub><mo stretchy="false" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">⟩</mo></mrow><mo id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.4" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.3.cmml">,</mo><msub id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.2.cmml"><mi id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.2.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.2.2.cmml">x</mi><mrow id="S4.SS2.p4.1.m1.2.2.2.4" xref="S4.SS2.p4.1.m1.2.2.2.3.cmml"><mi id="S4.SS2.p4.1.m1.1.1.1.1" xref="S4.SS2.p4.1.m1.1.1.1.1.cmml">p</mi><mo id="S4.SS2.p4.1.m1.2.2.2.4.1" xref="S4.SS2.p4.1.m1.2.2.2.3.cmml">,</mo><mi id="S4.SS2.p4.1.m1.2.2.2.2" xref="S4.SS2.p4.1.m1.2.2.2.2.cmml">c</mi></mrow></msub><mo stretchy="false" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.5" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.3.cmml">=</mo><msub id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.4" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.4.cmml"><mi id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.4.2" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.4.2.cmml">y</mi><mrow id="S4.SS2.p4.1.m1.4.4.2.4" xref="S4.SS2.p4.1.m1.4.4.2.3.cmml"><mi id="S4.SS2.p4.1.m1.3.3.1.1" xref="S4.SS2.p4.1.m1.3.3.1.1.cmml">p</mi><mo id="S4.SS2.p4.1.m1.4.4.2.4.1" xref="S4.SS2.p4.1.m1.4.4.2.3.cmml">,</mo><mi id="S4.SS2.p4.1.m1.4.4.2.2" xref="S4.SS2.p4.1.m1.4.4.2.2.cmml">c</mi></mrow></msub></mrow><mo stretchy="false" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.3" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.2.cmml">}</mo></mrow></mrow></mrow></mrow></mrow><mo lspace="0em" id="S4.SS2.p4.1.m1.5.5.1.2" xref="S4.SS2.p4.1.m1.5.5.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.1.m1.5b"><apply id="S4.SS2.p4.1.m1.5.5.1.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1"><times id="S4.SS2.p4.1.m1.5.5.1.1.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.2"></times><apply id="S4.SS2.p4.1.m1.5.5.1.1.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.3"><divide id="S4.SS2.p4.1.m1.5.5.1.1.3.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.3"></divide><cn type="integer" id="S4.SS2.p4.1.m1.5.5.1.1.3.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.3.2">1</cn><ci id="S4.SS2.p4.1.m1.5.5.1.1.3.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.3.3">𝑃</ci></apply><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1"><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.5.5.1.1.1.2.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2">superscript</csymbol><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2">subscript</csymbol><sum id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.2"></sum><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3"><eq id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.1"></eq><ci id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.2">𝑝</ci><cn type="integer" id="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="S4.SS2.p4.1.m1.5.5.1.1.1.2.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.2.3">𝑃</ci></apply><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1"><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2">superscript</csymbol><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2">subscript</csymbol><csymbol cd="latexml" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.2">product</csymbol><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3"><eq id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.1"></eq><ci id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.2">𝑐</ci><cn type="integer" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.2.3.3">1</cn></apply></apply><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3">subscript</csymbol><ci id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.2">𝐶</ci><ci id="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.2.3.3">𝑝</ci></apply></apply><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1"><times id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.2"></times><cn type="integer" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.3">1</cn><set id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1"><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1"><eq id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.3"></eq><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2"><times id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.3"></times><ci id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.4a.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.4"><mtext class="ltx_mathvariant_monospace" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.4.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.4">eval</mtext></ci><interval closure="open" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2"><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2">delimited-⟨⟩</csymbol><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2a.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">code</mtext></ci><ci id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑝</ci></apply></apply><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.2.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.2.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.2.2.2.2.2">𝑥</ci><list id="S4.SS2.p4.1.m1.2.2.2.3.cmml" xref="S4.SS2.p4.1.m1.2.2.2.4"><ci id="S4.SS2.p4.1.m1.1.1.1.1.cmml" xref="S4.SS2.p4.1.m1.1.1.1.1">𝑝</ci><ci id="S4.SS2.p4.1.m1.2.2.2.2.cmml" xref="S4.SS2.p4.1.m1.2.2.2.2">𝑐</ci></list></apply></interval></apply><apply id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.4.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.4.1.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.4">subscript</csymbol><ci id="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.4.2.cmml" xref="S4.SS2.p4.1.m1.5.5.1.1.1.1.1.1.1.1.4.2">𝑦</ci><list id="S4.SS2.p4.1.m1.4.4.2.3.cmml" xref="S4.SS2.p4.1.m1.4.4.2.4"><ci id="S4.SS2.p4.1.m1.3.3.1.1.cmml" xref="S4.SS2.p4.1.m1.3.3.1.1">𝑝</ci><ci id="S4.SS2.p4.1.m1.4.4.2.2.cmml" xref="S4.SS2.p4.1.m1.4.4.2.2">𝑐</ci></list></apply></apply></set></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.1.m1.5c">\frac{1}{P}\sum_{p=1}^{P}\prod_{c=1}^{C_{p}}\mathds{1}\{\texttt{eval}(\langle\texttt{code}_{p}\rangle,x_{p,c})=y_{p,c}\}.</annotation></semantics></math> Future research may only use strict accuracy when models become sufficiently capable.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Model Performance Analysis</h3>

<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Qualitative Output Analysis.</h4>

<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px1.p1.1" class="ltx_p">Models can sometimes generate correct or superficially plausible code. <a href="#S3.F2" title="In 3 The APPS Dataset ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">2</span></a> shows code generated by GPT-2 1.5B that passes all test cases. When models do not pass the test cases, sometimes their generated code still appears plausible at first glance. For example, in <a href="#S3.F3" title="In 3 The APPS Dataset ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>, we see that the 1.5B parameter model generates code that is related to the problem statement and makes a plausible attempt to solve it.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Test Case Evaluation.</h4>

<div id="S4.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px2.p1.1" class="ltx_p">We show the main results in <a href="#S4.T2" title="In 4.2 Metrics ‣ 4 Experiments ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a>. We observe that models are able to generate code that passed some test cases, implying many generated programs are free of syntax errors and can successfully process inputs test cases to produce correct answers. Note that for Introductory questions, GPT-Neo passes approximately <math id="S4.SS3.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="15\%" display="inline"><semantics id="S4.SS3.SSS0.Px2.p1.1.m1.1a"><mrow id="S4.SS3.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.cmml"><mn id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.2" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.2.cmml">15</mn><mo id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.1" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p1.1.m1.1b"><apply id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1"><csymbol cd="latexml" id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.2">15</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p1.1.m1.1c">15\%</annotation></semantics></math> of the test cases. We visualize Test Case Average results in <a href="#S4.F5" title="In Syntax Errors. ‣ 4.3 Model Performance Analysis ‣ 4 Experiments ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">5</span></a>. This demonstrates models are showing marked improvements on code generation and now starting to have traction on code generation.</p>
</div>
<figure id="S4.T3" class="ltx_table ltx_align_floatright">
<table id="S4.T3.4.4" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T3.4.4.5" class="ltx_tr">
<td id="S4.T3.4.4.5.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T3.4.4.5.2" class="ltx_td ltx_align_center">Top-1</td>
<td id="S4.T3.4.4.5.3" class="ltx_td ltx_align_center">Top-5</td>
</tr>
<tr id="S4.T3.2.2.2" class="ltx_tr">
<td id="S4.T3.2.2.2.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Test Case Average</td>
<td id="S4.T3.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t"><math id="S4.T3.1.1.1.1.m1.1" class="ltx_Math" alttext="14.7\%" display="inline"><semantics id="S4.T3.1.1.1.1.m1.1a"><mrow id="S4.T3.1.1.1.1.m1.1.1" xref="S4.T3.1.1.1.1.m1.1.1.cmml"><mn id="S4.T3.1.1.1.1.m1.1.1.2" xref="S4.T3.1.1.1.1.m1.1.1.2.cmml">14.7</mn><mo id="S4.T3.1.1.1.1.m1.1.1.1" xref="S4.T3.1.1.1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.1.1.1.1.m1.1b"><apply id="S4.T3.1.1.1.1.m1.1.1.cmml" xref="S4.T3.1.1.1.1.m1.1.1"><csymbol cd="latexml" id="S4.T3.1.1.1.1.m1.1.1.1.cmml" xref="S4.T3.1.1.1.1.m1.1.1.1">percent</csymbol><cn type="float" id="S4.T3.1.1.1.1.m1.1.1.2.cmml" xref="S4.T3.1.1.1.1.m1.1.1.2">14.7</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.1.1.1.1.m1.1c">14.7\%</annotation></semantics></math></td>
<td id="S4.T3.2.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><math id="S4.T3.2.2.2.2.m1.1" class="ltx_Math" alttext="19.9\%" display="inline"><semantics id="S4.T3.2.2.2.2.m1.1a"><mrow id="S4.T3.2.2.2.2.m1.1.1" xref="S4.T3.2.2.2.2.m1.1.1.cmml"><mn id="S4.T3.2.2.2.2.m1.1.1.2" xref="S4.T3.2.2.2.2.m1.1.1.2.cmml">19.9</mn><mo id="S4.T3.2.2.2.2.m1.1.1.1" xref="S4.T3.2.2.2.2.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.2.2.2.2.m1.1b"><apply id="S4.T3.2.2.2.2.m1.1.1.cmml" xref="S4.T3.2.2.2.2.m1.1.1"><csymbol cd="latexml" id="S4.T3.2.2.2.2.m1.1.1.1.cmml" xref="S4.T3.2.2.2.2.m1.1.1.1">percent</csymbol><cn type="float" id="S4.T3.2.2.2.2.m1.1.1.2.cmml" xref="S4.T3.2.2.2.2.m1.1.1.2">19.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.2.2.2.2.m1.1c">19.9\%</annotation></semantics></math></td>
</tr>
<tr id="S4.T3.4.4.4" class="ltx_tr">
<td id="S4.T3.4.4.4.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">Strict Accuracy</td>
<td id="S4.T3.3.3.3.1" class="ltx_td ltx_align_center ltx_border_bb"><math id="S4.T3.3.3.3.1.m1.1" class="ltx_Math" alttext="3.9\%" display="inline"><semantics id="S4.T3.3.3.3.1.m1.1a"><mrow id="S4.T3.3.3.3.1.m1.1.1" xref="S4.T3.3.3.3.1.m1.1.1.cmml"><mn id="S4.T3.3.3.3.1.m1.1.1.2" xref="S4.T3.3.3.3.1.m1.1.1.2.cmml">3.9</mn><mo id="S4.T3.3.3.3.1.m1.1.1.1" xref="S4.T3.3.3.3.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.3.3.3.1.m1.1b"><apply id="S4.T3.3.3.3.1.m1.1.1.cmml" xref="S4.T3.3.3.3.1.m1.1.1"><csymbol cd="latexml" id="S4.T3.3.3.3.1.m1.1.1.1.cmml" xref="S4.T3.3.3.3.1.m1.1.1.1">percent</csymbol><cn type="float" id="S4.T3.3.3.3.1.m1.1.1.2.cmml" xref="S4.T3.3.3.3.1.m1.1.1.2">3.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.3.3.3.1.m1.1c">3.9\%</annotation></semantics></math></td>
<td id="S4.T3.4.4.4.2" class="ltx_td ltx_align_center ltx_border_bb"><math id="S4.T3.4.4.4.2.m1.1" class="ltx_Math" alttext="5.5\%" display="inline"><semantics id="S4.T3.4.4.4.2.m1.1a"><mrow id="S4.T3.4.4.4.2.m1.1.1" xref="S4.T3.4.4.4.2.m1.1.1.cmml"><mn id="S4.T3.4.4.4.2.m1.1.1.2" xref="S4.T3.4.4.4.2.m1.1.1.2.cmml">5.5</mn><mo id="S4.T3.4.4.4.2.m1.1.1.1" xref="S4.T3.4.4.4.2.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.4.4.4.2.m1.1b"><apply id="S4.T3.4.4.4.2.m1.1.1.cmml" xref="S4.T3.4.4.4.2.m1.1.1"><csymbol cd="latexml" id="S4.T3.4.4.4.2.m1.1.1.1.cmml" xref="S4.T3.4.4.4.2.m1.1.1.1">percent</csymbol><cn type="float" id="S4.T3.4.4.4.2.m1.1.1.2.cmml" xref="S4.T3.4.4.4.2.m1.1.1.2">5.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.4.4.4.2.m1.1c">5.5\%</annotation></semantics></math></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>GPT-Neo 2.7B performance on introductory problems using one generated program (Top-1) and the best of five generated programs (Top-5). Full results are in the Supplementary Materials. </figcaption>
</figure>
<div id="S4.SS3.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px2.p2.3" class="ltx_p">Performance can be further improved by sampling multiple solutions and selecting the best. Here, we perform beam search with beam width <math id="S4.SS3.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="5" display="inline"><semantics id="S4.SS3.SSS0.Px2.p2.1.m1.1a"><mn id="S4.SS3.SSS0.Px2.p2.1.m1.1.1" xref="S4.SS3.SSS0.Px2.p2.1.m1.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p2.1.m1.1b"><cn type="integer" id="S4.SS3.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px2.p2.1.m1.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p2.1.m1.1c">5</annotation></semantics></math> and evaluate its <math id="S4.SS3.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="5" display="inline"><semantics id="S4.SS3.SSS0.Px2.p2.2.m2.1a"><mn id="S4.SS3.SSS0.Px2.p2.2.m2.1.1" xref="S4.SS3.SSS0.Px2.p2.2.m2.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p2.2.m2.1b"><cn type="integer" id="S4.SS3.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px2.p2.2.m2.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p2.2.m2.1c">5</annotation></semantics></math> beams, so that each model has five attempts to get a problem correct rather than one. With this setup, GPT-Neo’s strict accuracy on Introductory problem then exceeds <math id="S4.SS3.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="5\%" display="inline"><semantics id="S4.SS3.SSS0.Px2.p2.3.m3.1a"><mrow id="S4.SS3.SSS0.Px2.p2.3.m3.1.1" xref="S4.SS3.SSS0.Px2.p2.3.m3.1.1.cmml"><mn id="S4.SS3.SSS0.Px2.p2.3.m3.1.1.2" xref="S4.SS3.SSS0.Px2.p2.3.m3.1.1.2.cmml">5</mn><mo id="S4.SS3.SSS0.Px2.p2.3.m3.1.1.1" xref="S4.SS3.SSS0.Px2.p2.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p2.3.m3.1b"><apply id="S4.SS3.SSS0.Px2.p2.3.m3.1.1.cmml" xref="S4.SS3.SSS0.Px2.p2.3.m3.1.1"><csymbol cd="latexml" id="S4.SS3.SSS0.Px2.p2.3.m3.1.1.1.cmml" xref="S4.SS3.SSS0.Px2.p2.3.m3.1.1.1">percent</csymbol><cn type="integer" id="S4.SS3.SSS0.Px2.p2.3.m3.1.1.2.cmml" xref="S4.SS3.SSS0.Px2.p2.3.m3.1.1.2">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p2.3.m3.1c">5\%</annotation></semantics></math>, as shown in <a href="#S4.T3" title="In Test Case Evaluation. ‣ 4.3 Model Performance Analysis ‣ 4 Experiments ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">3</span></a>. Our results in the Supplementary Materials show that the top-5 test case average GPT-2 0.1B is 10.75 while the top-1 test case average of GPT-2 1.5B is 7.96. This highlights that simply sampling multiple candidate solutions is a powerful way to markedly improve performance.</p>
</div>
<div id="S4.SS3.SSS0.Px2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px2.p3.1" class="ltx_p">Our results also provide us with information about the importance of model choice. Evidently existing few-shot GPT-3 models are not necessarily better at code generation than fine-tuned models that are smaller by two orders of magnitude. Additionally, performance improvement from GPT-2 1.5B to GPT-Neo 2.7B is larger than that from GPT-2 0.1B to GPT-2 1.5B. Potential causes of GPT-Neo’s better performance are that GPT-Neo is trained on more code from GitHub, it has more parameters, or its architecture hyperparameters were chosen better. Memorization explaining all performance is an implausible explanation as performance tracks problem difficulty; were models just memorizing, we would expect uniform performance across difficulties. Since models still have large room for improvement, solving the APPS benchmark without unreasonable amounts of computational resources may require architectural or algorithmic improvements.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Syntax Errors.</h4>

<div id="S4.SS3.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px3.p1.2" class="ltx_p">We now assess the frequency of syntax errors, errors that prevent the program from being interpreted including inconsistent spacing, unbalanced brackets, missing colons, and so on. Syntax errors are identified in our testing framework based on the heuristic of whether pyext is able to load the generated code as a Python module. For our purposes, this almost exclusively occurs for syntax errors. We visualize the prevalence of syntax errors in <a href="#S4.F5" title="In Syntax Errors. ‣ 4.3 Model Performance Analysis ‣ 4 Experiments ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">5</span></a>. While approximately <math id="S4.SS3.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="59\%" display="inline"><semantics id="S4.SS3.SSS0.Px3.p1.1.m1.1a"><mrow id="S4.SS3.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1.cmml"><mn id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.2" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1.2.cmml">59</mn><mo id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.1" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px3.p1.1.m1.1b"><apply id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1"><csymbol cd="latexml" id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1.2">59</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px3.p1.1.m1.1c">59\%</annotation></semantics></math> of GPT-3’s generated solutions for introductory problems have syntax errors, GPT-Neo syntax error frequency is approximately <math id="S4.SS3.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="3\%" display="inline"><semantics id="S4.SS3.SSS0.Px3.p1.2.m2.1a"><mrow id="S4.SS3.SSS0.Px3.p1.2.m2.1.1" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1.cmml"><mn id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.2" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1.2.cmml">3</mn><mo id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.1" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px3.p1.2.m2.1b"><apply id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1"><csymbol cd="latexml" id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1.2">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px3.p1.2.m2.1c">3\%</annotation></semantics></math>. Note that recent work such as <cite class="ltx_cite ltx_citemacro_cite">Yasunaga and Liang (<a href="#bib.bib38" title="" class="ltx_ref">2020</a>)</cite> create a separate model to repair source code to fix compilation issues, but our results suggest that such efforts may be unnecessary in the future as syntax error frequency is sharply decreasing automatically.</p>
</div>
<figure id="S4.F5" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F5.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_middle" style="width:212.5pt;"><img src="/html/2105.09938/assets/x2.png" id="S4.F5.1.g1" class="ltx_graphics ltx_img_square" width="460" height="402" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>The average percentage of test cases passed increases with larger fine-tuned models.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F5.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_middle" style="width:212.5pt;"><img src="/html/2105.09938/assets/x3.png" id="S4.F5.2.g1" class="ltx_graphics ltx_img_square" width="461" height="418" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Syntax errors decrease exponentially with fine-tuning and increased model sizes. GPT-Neo 2.7B has very few syntax errors.</figcaption>
</figure>
</div>
</div>
</figure>
</section>
<section id="S4.SS3.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">BLEU.</h4>

<div id="S4.SS3.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px4.p1.6" class="ltx_p">We find that assessing model performance with BLEU is a poor substitute for evaluating with test cases. To evaluate BLEU, we take the generated solution and compute its BLEU with each human-written solution for a given problem; we then record the highest BLEU score. Observe in <a href="#S4.F6" title="In BLEU. ‣ 4.3 Model Performance Analysis ‣ 4 Experiments ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">6</span></a> that BLEU increases as problem sources become more difficult, even though models actually perform worse on harder problems. Moreover, worse models can have similar or higher BLEU scores. For example, GPT-2 0.1B has <math id="S4.SS3.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="26.8" display="inline"><semantics id="S4.SS3.SSS0.Px4.p1.1.m1.1a"><mn id="S4.SS3.SSS0.Px4.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px4.p1.1.m1.1.1.cmml">26.8</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px4.p1.1.m1.1b"><cn type="float" id="S4.SS3.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px4.p1.1.m1.1.1">26.8</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px4.p1.1.m1.1c">26.8</annotation></semantics></math>, <math id="S4.SS3.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="29.7" display="inline"><semantics id="S4.SS3.SSS0.Px4.p1.2.m2.1a"><mn id="S4.SS3.SSS0.Px4.p1.2.m2.1.1" xref="S4.SS3.SSS0.Px4.p1.2.m2.1.1.cmml">29.7</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px4.p1.2.m2.1b"><cn type="float" id="S4.SS3.SSS0.Px4.p1.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px4.p1.2.m2.1.1">29.7</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px4.p1.2.m2.1c">29.7</annotation></semantics></math>, and <math id="S4.SS3.SSS0.Px4.p1.3.m3.1" class="ltx_Math" alttext="30.2" display="inline"><semantics id="S4.SS3.SSS0.Px4.p1.3.m3.1a"><mn id="S4.SS3.SSS0.Px4.p1.3.m3.1.1" xref="S4.SS3.SSS0.Px4.p1.3.m3.1.1.cmml">30.2</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px4.p1.3.m3.1b"><cn type="float" id="S4.SS3.SSS0.Px4.p1.3.m3.1.1.cmml" xref="S4.SS3.SSS0.Px4.p1.3.m3.1.1">30.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px4.p1.3.m3.1c">30.2</annotation></semantics></math> as BLEU scores for introductory, interview, and competition problems, respectively. Meanwhile GPT-Neo 2.7B has <math id="S4.SS3.SSS0.Px4.p1.4.m4.1" class="ltx_Math" alttext="27.1" display="inline"><semantics id="S4.SS3.SSS0.Px4.p1.4.m4.1a"><mn id="S4.SS3.SSS0.Px4.p1.4.m4.1.1" xref="S4.SS3.SSS0.Px4.p1.4.m4.1.1.cmml">27.1</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px4.p1.4.m4.1b"><cn type="float" id="S4.SS3.SSS0.Px4.p1.4.m4.1.1.cmml" xref="S4.SS3.SSS0.Px4.p1.4.m4.1.1">27.1</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px4.p1.4.m4.1c">27.1</annotation></semantics></math>, <math id="S4.SS3.SSS0.Px4.p1.5.m5.1" class="ltx_Math" alttext="29.1" display="inline"><semantics id="S4.SS3.SSS0.Px4.p1.5.m5.1a"><mn id="S4.SS3.SSS0.Px4.p1.5.m5.1.1" xref="S4.SS3.SSS0.Px4.p1.5.m5.1.1.cmml">29.1</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px4.p1.5.m5.1b"><cn type="float" id="S4.SS3.SSS0.Px4.p1.5.m5.1.1.cmml" xref="S4.SS3.SSS0.Px4.p1.5.m5.1.1">29.1</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px4.p1.5.m5.1c">29.1</annotation></semantics></math>, and <math id="S4.SS3.SSS0.Px4.p1.6.m6.1" class="ltx_Math" alttext="29.3" display="inline"><semantics id="S4.SS3.SSS0.Px4.p1.6.m6.1a"><mn id="S4.SS3.SSS0.Px4.p1.6.m6.1.1" xref="S4.SS3.SSS0.Px4.p1.6.m6.1.1.cmml">29.3</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px4.p1.6.m6.1b"><cn type="float" id="S4.SS3.SSS0.Px4.p1.6.m6.1.1.cmml" xref="S4.SS3.SSS0.Px4.p1.6.m6.1.1">29.3</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px4.p1.6.m6.1c">29.3</annotation></semantics></math> as its BLEU scores, respectively. Hence BLEU wrongly suggests GPT-Neo is a worse model.</p>
</div>
<figure id="S4.F6" class="ltx_figure ltx_align_floatright"><img src="/html/2105.09938/assets/x4.png" id="S4.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="230" height="178" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>BLEU scores for GPT-Neo 2.7B increase with difficulty level and are anticorrelated with a gold-standard accuracy metric.</figcaption>
</figure>
</section>
<section id="S4.SS3.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluating GPT-3.</h4>

<div id="S4.SS3.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px5.p1.2" class="ltx_p">We evaluate GPT-3 175B on APPS in a few-shot setting. A separate prompt is used for standard input and call-based questions, and each prompt includes instruction text along with two example questions and solutions from the corresponding question type. We find that GPT-3 only solves <math id="S4.SS3.SSS0.Px5.p1.1.m1.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S4.SS3.SSS0.Px5.p1.1.m1.1a"><mn id="S4.SS3.SSS0.Px5.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px5.p1.1.m1.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px5.p1.1.m1.1b"><cn type="integer" id="S4.SS3.SSS0.Px5.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px5.p1.1.m1.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px5.p1.1.m1.1c">3</annotation></semantics></math> problems out of <math id="S4.SS3.SSS0.Px5.p1.2.m2.2" class="ltx_Math" alttext="5,\!000" display="inline"><semantics id="S4.SS3.SSS0.Px5.p1.2.m2.2a"><mrow id="S4.SS3.SSS0.Px5.p1.2.m2.2.3.2" xref="S4.SS3.SSS0.Px5.p1.2.m2.2.3.1.cmml"><mn id="S4.SS3.SSS0.Px5.p1.2.m2.1.1" xref="S4.SS3.SSS0.Px5.p1.2.m2.1.1.cmml">5</mn><mpadded width="0.275em"><mo id="S4.SS3.SSS0.Px5.p1.2.m2.2.3.2.1" xref="S4.SS3.SSS0.Px5.p1.2.m2.2.3.1.cmml">,</mo></mpadded><mn id="S4.SS3.SSS0.Px5.p1.2.m2.2.2" xref="S4.SS3.SSS0.Px5.p1.2.m2.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px5.p1.2.m2.2b"><list id="S4.SS3.SSS0.Px5.p1.2.m2.2.3.1.cmml" xref="S4.SS3.SSS0.Px5.p1.2.m2.2.3.2"><cn type="integer" id="S4.SS3.SSS0.Px5.p1.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px5.p1.2.m2.1.1">5</cn><cn type="integer" id="S4.SS3.SSS0.Px5.p1.2.m2.2.2.cmml" xref="S4.SS3.SSS0.Px5.p1.2.m2.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px5.p1.2.m2.2c">5,\!000</annotation></semantics></math>: two introductory problems and one interview problem. The two introductory problems are simple interpretation tasks, such as implementing a specified algebraic expression. The interview problem requires higher-level thinking that suggests nontrivial reasoning. However, it is possible that GPT-3 memorized the solution during pretraining, or that it took a lucky guess based on heuristics in the question. One potential factor in GPT-3’s poor performance is that it handles syntax poorly. Namely, we observed cases where improper formatting of otherwise functioning code causes a syntax error. For specific examples and more details, see the Supplementary Materials.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px6" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluations on Larger Models.</h4>

<div id="S4.SS3.SSS0.Px6.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px6.p1.1" class="ltx_p">Since the public release of APPS, several others have trained even larger models on APPS than we evaluate here. OpenAI Codex is a 12B parameter Transformer language model pre-trained on large quantities of public code and comments. <cite class="ltx_cite ltx_citemacro_citet">Chen et al. (<a href="#bib.bib7" title="" class="ltx_ref">2021</a>)</cite> evaluate Codex on APPS under various configurations and achieve top-1 and top-5 accuracy on introductory problems of 4.14% and 9.65% respectively, close to double the top-5 accuracy of GPT-Neo 2.7B. Furthermore, by scaling up to a top-1000 evaluation they obtain 25% accuracy. This demonstrates that larger models trained specifically for code generation can improve APPS performance even further, but are still far from solving the task.</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.2" class="ltx_p">We introduced APPS, a benchmark of <math id="S5.p1.1.m1.2" class="ltx_Math" alttext="10,\!000" display="inline"><semantics id="S5.p1.1.m1.2a"><mrow id="S5.p1.1.m1.2.3.2" xref="S5.p1.1.m1.2.3.1.cmml"><mn id="S5.p1.1.m1.1.1" xref="S5.p1.1.m1.1.1.cmml">10</mn><mpadded width="0.275em"><mo id="S5.p1.1.m1.2.3.2.1" xref="S5.p1.1.m1.2.3.1.cmml">,</mo></mpadded><mn id="S5.p1.1.m1.2.2" xref="S5.p1.1.m1.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.p1.1.m1.2b"><list id="S5.p1.1.m1.2.3.1.cmml" xref="S5.p1.1.m1.2.3.2"><cn type="integer" id="S5.p1.1.m1.1.1.cmml" xref="S5.p1.1.m1.1.1">10</cn><cn type="integer" id="S5.p1.1.m1.2.2.cmml" xref="S5.p1.1.m1.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S5.p1.1.m1.2c">10,\!000</annotation></semantics></math> Python programming problems. Unlike prior work that focused on pseudocode to code generation or translation between programming languages, our benchmark measures how well language models can generate python code given natural language specifications. By performing extensive quality assurance and including hundreds of thousands of test cases and ground-truth solutions across different difficulty levels, we created a comprehensive and rigorous testbed for evaluating models. We assessed state-of-the-art generative models on our benchmark and found that overall performance was low. However, the prevalence of syntax errors decreased exponentially as models improved, and recent models such as GPT-Neo solved over <math id="S5.p1.2.m2.1" class="ltx_Math" alttext="5\%" display="inline"><semantics id="S5.p1.2.m2.1a"><mrow id="S5.p1.2.m2.1.1" xref="S5.p1.2.m2.1.1.cmml"><mn id="S5.p1.2.m2.1.1.2" xref="S5.p1.2.m2.1.1.2.cmml">5</mn><mo id="S5.p1.2.m2.1.1.1" xref="S5.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.p1.2.m2.1b"><apply id="S5.p1.2.m2.1.1.cmml" xref="S5.p1.2.m2.1.1"><csymbol cd="latexml" id="S5.p1.2.m2.1.1.1.cmml" xref="S5.p1.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S5.p1.2.m2.1.1.2.cmml" xref="S5.p1.2.m2.1.1.2">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p1.2.m2.1c">5\%</annotation></semantics></math> of our introductory problems. As models become more competent at code generation, it is important to have a proxy for tracking this capability which could one day result in automation or malicious code generation. The APPS benchmark can provide an important measure for tracking upstream program synthesis advancements.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Allamanis and Sutton (2013)</span>
<span class="ltx_bibblock">
Miltiadis Allamanis and Charles Sutton.

</span>
<span class="ltx_bibblock">Mining source code repositories at massive scale using language
modeling.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">2013 10th Working Conference on Mining Software Repositories
(MSR)</em>, pages 207–216. IEEE, 2013.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Alur et al. (2018)</span>
<span class="ltx_bibblock">
Rajeev Alur, Dana Fisman, Saswat Padhi, Rishabh Singh, and Abhishek Udupa.

</span>
<span class="ltx_bibblock">Sygus-comp 2018: Results and analysis.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">SYNT</em>, 2018.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bisk et al. (2019)</span>
<span class="ltx_bibblock">
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.

</span>
<span class="ltx_bibblock">Piqa: Reasoning about physical commonsense in natural language, 2019.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Black et al. (2021)</span>
<span class="ltx_bibblock">
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman.

</span>
<span class="ltx_bibblock">GPT-Neo: Large Scale Autoregressive Language Modeling with
Mesh-Tensorflow, March 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.5281/zenodo.5297715" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.5281/zenodo.5297715</a>.

</span>
<span class="ltx_bibblock">If you use this software, please cite it using these metadata.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
T. Brown, B. Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini
Agarwal, Ariel Herbert-Voss, G. Krüger, T. Henighan, R. Child, Aditya
Ramesh, D. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen,
E. Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, J. Clark, Christopher
Berner, Sam McCandlish, A. Radford, Ilya Sutskever, and Dario Amodei.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2005.14165, 2020.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cai et al. (2017)</span>
<span class="ltx_bibblock">
Jonathon Cai, Richard Shin, and D. Song.

</span>
<span class="ltx_bibblock">Making neural programming architectures generalize via recursion.

</span>
<span class="ltx_bibblock">2017.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2021)</span>
<span class="ltx_bibblock">
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan,
Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al.

</span>
<span class="ltx_bibblock">Evaluating large language models trained on code.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2107.03374</em>, 2021.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Desai et al. (2016)</span>
<span class="ltx_bibblock">
Aditya Desai, Sumit Gulwani, Vineet Hingorani, Nidhi Jain, Amey Karkare, Mark
Marron, and Subhajit Roy.

</span>
<span class="ltx_bibblock">Program synthesis using natural language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 38th International Conference on Software
Engineering</em>, pages 345–356, 2016.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2020)</span>
<span class="ltx_bibblock">
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles
Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.

</span>
<span class="ltx_bibblock">The pile: An 800gb dataset of diverse text for language modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2101.00027</em>, 2020.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gebru et al. (2018)</span>
<span class="ltx_bibblock">
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan,
Hanna Wallach, Hal Daumeé III, and Kate Crawford.

</span>
<span class="ltx_bibblock">Datasheets for datasets.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1803.09010</em>, 2018.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gulwani et al. (2017)</span>
<span class="ltx_bibblock">
Sumit Gulwani, Oleksandr Polozov, and R. Singh.

</span>
<span class="ltx_bibblock">Program synthesis.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Found. Trends Program. Lang.</em>, 4:1–119, 2017.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021a)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song,
and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Aligning AI with shared human values.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Proceedings of the International Conference on Learning
Representations (ICLR)</em>, 2021a.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021b)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn
Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Proceedings of the International Conference on Learning
Representations (ICLR)</em>, 2021b.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021c)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric
Tang, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring mathematical problem solving with the math dataset.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2103.03874</em>, 2021c.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2019)</span>
<span class="ltx_bibblock">
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.

</span>
<span class="ltx_bibblock">Cosmos qa: Machine reading comprehension with contextual commonsense
reasoning, 2019.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Iyer et al. (2018)</span>
<span class="ltx_bibblock">
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Mapping language to code in programmatic context.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing</em>, Brussels, Belgium, October-November 2018.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kulal et al. (2019)</span>
<span class="ltx_bibblock">
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex
Aiken, and Percy S Liang.

</span>
<span class="ltx_bibblock">Spoc: Search-based pseudocode to code.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
volume 32, 2019.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lachaux et al. (2020)</span>
<span class="ltx_bibblock">
Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, and Guillaume Lample.

</span>
<span class="ltx_bibblock">Unsupervised translation of programming languages.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2006.03511</em>, 2020.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ling et al. (2016)</span>
<span class="ltx_bibblock">
W. Ling, P. Blunsom, Edward Grefenstette, K. Hermann, Tomás Kociský,
Fumin Wang, and A. Senior.

</span>
<span class="ltx_bibblock">Latent predictor networks for code generation.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1603.06744, 2016.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2020a)</span>
<span class="ltx_bibblock">
Hui Liu, Mingzhu Shen, Jiaqi Zhu, Nan Niu, Ge Li, and Lu Zhang.

</span>
<span class="ltx_bibblock">Deep learning based program generation from requirements text: Are we
there yet?

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Software Engineering</em>, 2020a.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2020b)</span>
<span class="ltx_bibblock">
J. Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang.

</span>
<span class="ltx_bibblock">LogiQA: A challenge dataset for machine reading comprehension with
logical reasoning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">IJCAI</em>, 2020b.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Loshchilov and Hutter (2019)</span>
<span class="ltx_bibblock">
I. Loshchilov and F. Hutter.

</span>
<span class="ltx_bibblock">Decoupled weight decay regularization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2019.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. (2021)</span>
<span class="ltx_bibblock">
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, A. Blanco,
C. Clément, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, L. Zhou, Linjun
Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, N. Duan,
N. Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu.

</span>
<span class="ltx_bibblock">Codexglue: A machine learning benchmark dataset for code
understanding and generation.

</span>
<span class="ltx_bibblock">2021.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Oda et al. (2015)</span>
<span class="ltx_bibblock">
Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti,
Tomoki Toda, and Satoshi Nakamura.

</span>
<span class="ltx_bibblock">Learning to generate pseudo-code from source code using statistical
machine translation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">International Conference on Automated Software Engineering
(ASE)</em>, 2015.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Papineni et al. (2002)</span>
<span class="ltx_bibblock">
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.

</span>
<span class="ltx_bibblock">Bleu: a method for automatic evaluation of machine translation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 40th annual meeting of the Association
for Computational Linguistics</em>, pages 311–318, 2002.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Polu and Sutskever (2020)</span>
<span class="ltx_bibblock">
Stanislas Polu and Ilya Sutskever.

</span>
<span class="ltx_bibblock">Generative language modeling for automated theorem proving.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2009.03393, 2020.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2019)</span>
<span class="ltx_bibblock">
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">OpenAI blog</em>, 1(8):9, 2019.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rajbhandari et al. (2020)</span>
<span class="ltx_bibblock">
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.

</span>
<span class="ltx_bibblock">Zero: Memory optimizations toward training trillion parameter models,
2020.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rasley et al. (2020)</span>
<span class="ltx_bibblock">
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.

</span>
<span class="ltx_bibblock">Deepspeed: System optimizations enable training deep learning models
with over 100 billion parameters.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 26th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining</em>, 2020.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raychev et al. (2016)</span>
<span class="ltx_bibblock">
Veselin Raychev, Pavol Bielik, and Martin T. Vechev.

</span>
<span class="ltx_bibblock">Probabilistic model for code with decision trees.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2016 ACM SIGPLAN International Conference on
Object-Oriented Programming, Systems, Languages, and Applications</em>, 2016.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raza et al. (2015)</span>
<span class="ltx_bibblock">
Mohammad Raza, Sumit Gulwani, and Natasa Milic-Frayling.

</span>
<span class="ltx_bibblock">Compositional program synthesis from natural language and examples.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">IJCAI</em>, 2015.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ren et al. (2020)</span>
<span class="ltx_bibblock">
Shuo Ren, Daya Guo, Shuai Lu, L. Zhou, Shujie Liu, Duyu Tang, M. Zhou,
A. Blanco, and S. Ma.

</span>
<span class="ltx_bibblock">Codebleu: a method for automatic evaluation of code synthesis.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2009.10297, 2020.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tang and Mooney (2001)</span>
<span class="ltx_bibblock">
L. Tang and R. Mooney.

</span>
<span class="ltx_bibblock">Using multiple clause constructors in inductive logic programming for
semantic parsing.

</span>
<span class="ltx_bibblock">In <em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">ECML</em>, 2001.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. (2017)</span>
<span class="ltx_bibblock">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N. Gomez, L. Kaiser, and Illia Polosukhin.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1706.03762, 2017.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2019a)</span>
<span class="ltx_bibblock">
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael,
Felix Hill, Omer Levy, and Samuel Bowman.

</span>
<span class="ltx_bibblock">Superglue: A stickier benchmark for general-purpose language
understanding systems.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2019a.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2019b)</span>
<span class="ltx_bibblock">
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael,
Felix Hill, Omer Levy, and Samuel R. Bowman.

</span>
<span class="ltx_bibblock">Superglue: A stickier benchmark for general-purpose language
understanding systems.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2019b.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2019c)</span>
<span class="ltx_bibblock">
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew
Richardson.

</span>
<span class="ltx_bibblock">Rat-sql: Relation-aware schema encoding and linking for text-to-sql
parsers.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1911.04942</em>, 2019c.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yasunaga and Liang (2020)</span>
<span class="ltx_bibblock">
Michihiro Yasunaga and Percy Liang.

</span>
<span class="ltx_bibblock">Graph-based, self-supervised program repair from diagnostic feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2005.10636, 2020.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. (2018)</span>
<span class="ltx_bibblock">
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James
Ma, Irene Li, Qingning Yao, Shanelle Roman, et al.

</span>
<span class="ltx_bibblock">Spider: A large-scale human-labeled dataset for complex and
cross-domain semantic parsing and text-to-sql task.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1809.08887</em>, 2018.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zavershynskyi et al. (2018)</span>
<span class="ltx_bibblock">
Maksym Zavershynskyi, A. Skidanov, and Illia Polosukhin.

</span>
<span class="ltx_bibblock">Naps: Natural program synthesis dataset.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">2nd Workshop on Neural Abstract Machines and Program
Induction</em>, 2018.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zelle and Mooney (1996)</span>
<span class="ltx_bibblock">
J. Zelle and R. Mooney.

</span>
<span class="ltx_bibblock">Learning to parse database queries using inductive logic programming.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">AAAI/IAAI, Vol. 2</em>, 1996.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zellers et al. (2019)</span>
<span class="ltx_bibblock">
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.

</span>
<span class="ltx_bibblock">Hellaswag: Can a machine really finish your sentence?, 2019.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A0.T4" class="ltx_table">
<table id="A0.T4.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A0.T4.2.3" class="ltx_tr">
<td id="A0.T4.2.3.1" class="ltx_td" style="padding-left:9.0pt;padding-right:9.0pt;"></td>
<td id="A0.T4.2.3.2" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">Hearthstone</td>
<td id="A0.T4.2.3.3" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">Django</td>
<td id="A0.T4.2.3.4" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">NAPS</td>
<td id="A0.T4.2.3.5" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">APPS</td>
</tr>
<tr id="A0.T4.2.4" class="ltx_tr">
<td id="A0.T4.2.4.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:9.0pt;padding-right:9.0pt;">Programming Language</td>
<td id="A0.T4.2.4.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:9.0pt;padding-right:9.0pt;">Python</td>
<td id="A0.T4.2.4.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:9.0pt;padding-right:9.0pt;">Python</td>
<td id="A0.T4.2.4.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:9.0pt;padding-right:9.0pt;">UAST</td>
<td id="A0.T4.2.4.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:9.0pt;padding-right:9.0pt;">Python</td>
</tr>
<tr id="A0.T4.2.2" class="ltx_tr">
<td id="A0.T4.2.2.3" class="ltx_td ltx_align_left" style="padding-left:9.0pt;padding-right:9.0pt;">Test Cases</td>
<td id="A0.T4.2.2.4" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;"><span id="A0.T4.2.2.4.1" class="ltx_text" style="color:#FF0000;">✕</span></td>
<td id="A0.T4.2.2.5" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;"><span id="A0.T4.2.2.5.1" class="ltx_text" style="color:#FF0000;">✕</span></td>
<td id="A0.T4.1.1.1" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;"><span id="A0.T4.1.1.1.1" class="ltx_text" style="color:#009A18;"><svg id="A0.T4.1.1.1.1.pic1" class="ltx_picture" height="11.02" overflow="visible" version="1.1" width="15.75"><g transform="translate(0,11.02) matrix(1 0 0 -1 0 0)" fill="#009A18" stroke="#009A18" stroke-width="0.4pt"><path d="M 0 5.51 L 3.94 0 L 15.75 11.02 L 3.94 2.36 Z" style="stroke:none"></path></g></svg></span></td>
<td id="A0.T4.2.2.2" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;"><span id="A0.T4.2.2.2.1" class="ltx_text" style="color:#009A18;"><svg id="A0.T4.2.2.2.1.pic1" class="ltx_picture" height="11.02" overflow="visible" version="1.1" width="15.75"><g transform="translate(0,11.02) matrix(1 0 0 -1 0 0)" fill="#009A18" stroke="#009A18" stroke-width="0.4pt"><path d="M 0 5.51 L 3.94 0 L 15.75 11.02 L 3.94 2.36 Z" style="stroke:none"></path></g></svg></span></td>
</tr>
<tr id="A0.T4.2.5" class="ltx_tr">
<td id="A0.T4.2.5.1" class="ltx_td ltx_align_left" style="padding-left:9.0pt;padding-right:9.0pt;">Number of Programs</td>
<td id="A0.T4.2.5.2" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">665</td>
<td id="A0.T4.2.5.3" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">18,805</td>
<td id="A0.T4.2.5.4" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">17,477</td>
<td id="A0.T4.2.5.5" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">232,421</td>
</tr>
<tr id="A0.T4.2.6" class="ltx_tr">
<td id="A0.T4.2.6.1" class="ltx_td ltx_align_left" style="padding-left:9.0pt;padding-right:9.0pt;">Lines per Program (Avg.)</td>
<td id="A0.T4.2.6.2" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">7.7</td>
<td id="A0.T4.2.6.3" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">1</td>
<td id="A0.T4.2.6.4" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">21.7</td>
<td id="A0.T4.2.6.5" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">18.0</td>
</tr>
<tr id="A0.T4.2.7" class="ltx_tr">
<td id="A0.T4.2.7.1" class="ltx_td ltx_align_left" style="padding-left:9.0pt;padding-right:9.0pt;">Number of Exercises</td>
<td id="A0.T4.2.7.2" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">665</td>
<td id="A0.T4.2.7.3" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">18,805</td>
<td id="A0.T4.2.7.4" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">2,231</td>
<td id="A0.T4.2.7.5" class="ltx_td ltx_align_center" style="padding-left:9.0pt;padding-right:9.0pt;">10,000</td>
</tr>
<tr id="A0.T4.2.8" class="ltx_tr">
<td id="A0.T4.2.8.1" class="ltx_td ltx_align_left ltx_border_b" style="padding-left:9.0pt;padding-right:9.0pt;">Text Input</td>
<td id="A0.T4.2.8.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:9.0pt;padding-right:9.0pt;">Card Text</td>
<td id="A0.T4.2.8.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:9.0pt;padding-right:9.0pt;">Comment</td>
<td id="A0.T4.2.8.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:9.0pt;padding-right:9.0pt;">Pseudocode</td>
<td id="A0.T4.2.8.5" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:9.0pt;padding-right:9.0pt;">Problem Descriptions</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Further comparisons of APPS with previous datasets.</figcaption>
</figure>
<figure id="A0.T5" class="ltx_table">
<table id="A0.T5.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A0.T5.1.1" class="ltx_tr">
<td id="A0.T5.1.1.1" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="A0.T5.1.1.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;" colspan="4">Top-5 Test Case Average</td>
<td id="A0.T5.1.1.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;" colspan="4">Top-5 Strict Accuracy</td>
</tr>
<tr id="A0.T5.1.2" class="ltx_tr">
<td id="A0.T5.1.2.1" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">Model</td>
<td id="A0.T5.1.2.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Introductory</td>
<td id="A0.T5.1.2.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Interview</td>
<td id="A0.T5.1.2.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Competitive</td>
<td id="A0.T5.1.2.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:2.0pt;padding-right:2.0pt;">Average</td>
<td id="A0.T5.1.2.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Introductory</td>
<td id="A0.T5.1.2.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Interview</td>
<td id="A0.T5.1.2.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Competition</td>
<td id="A0.T5.1.2.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">Average</td>
</tr>
<tr id="A0.T5.1.3" class="ltx_tr">
<td id="A0.T5.1.3.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">GPT-2 0.1B</td>
<td id="A0.T5.1.3.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">13.81</td>
<td id="A0.T5.1.3.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">10.97</td>
<td id="A0.T5.1.3.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">7.03</td>
<td id="A0.T5.1.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">10.75</td>
<td id="A0.T5.1.3.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">2.70</td>
<td id="A0.T5.1.3.7" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">0.73</td>
<td id="A0.T5.1.3.8" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">0.00</td>
<td id="A0.T5.1.3.9" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">1.02</td>
</tr>
<tr id="A0.T5.1.4" class="ltx_tr">
<td id="A0.T5.1.4.1" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">GPT-2 1.5B</td>
<td id="A0.T5.1.4.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">16.86</td>
<td id="A0.T5.1.4.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">13.84</td>
<td id="A0.T5.1.4.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">9.01</td>
<td id="A0.T5.1.4.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:2.0pt;padding-right:2.0pt;">13.48</td>
<td id="A0.T5.1.4.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">3.60</td>
<td id="A0.T5.1.4.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">1.03</td>
<td id="A0.T5.1.4.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.00</td>
<td id="A0.T5.1.4.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">1.34</td>
</tr>
<tr id="A0.T5.1.5" class="ltx_tr">
<td id="A0.T5.1.5.1" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">GPT-Neo 2.7B</td>
<td id="A0.T5.1.5.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">19.89</td>
<td id="A0.T5.1.5.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">13.19</td>
<td id="A0.T5.1.5.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">9.90</td>
<td id="A0.T5.1.5.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:2.0pt;padding-right:2.0pt;">13.87</td>
<td id="A0.T5.1.5.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">5.50</td>
<td id="A0.T5.1.5.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.80</td>
<td id="A0.T5.1.5.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">0.00</td>
<td id="A0.T5.1.5.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">1.58</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Top-5 performance of GPT-2 models and GPT-Neo. Taking the best of five candidate solutions markedly improves performance.</figcaption>
</figure>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Auxiliary Dataset Information</h2>

<section id="A1.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Legal Compliance.</h4>

<div id="A1.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS0.SSS0.Px1.p1.1" class="ltx_p">In APPS, we scrape question text, ground-truth solutions, and test cases from various coding challenge websites. These websites are AtCoder, CodeChef, Codeforces, Codewars, HackerRank, Kattis, and LeetCode. In all cases, we only scrape public-facing data. For instance, we avoid scraping data from paywalled portions of sites. In the case of Kattis, all problems we scrape are under the CC BY-SA 3.0 license (<a target="_blank" href="https://creativecommons.org/licenses/by-sa/3.0/" title="" class="ltx_ref ltx_href">https://creativecommons.org/licenses/by-sa/3.0/</a>). For other websites, some content may be copyrighted. In these cases, we abide by Fair Use §107: “the fair use of a copyrighted work, including such use by … scholarship, or research, is not an infringement of copyright”, where fair use is determined by “the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes”, “the amount and substantiality of the portion used in relation to the copyrighted work as a whole”, and “the effect of the use upon the potential market for or value of the copyrighted work.” The APPS dataset is noncommercial and is likely to have no effect on the value of the original problems. Moreover, for all problem sources, we only scrape a fraction of the available problems and ground-truth solutions.</p>
</div>
<div id="A1.SS0.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="A1.SS0.SSS0.Px1.p2.1" class="ltx_p">Regarding international copyright laws, the websites that we scrape from are based in the United States, Japan, India, and Russia, all of which are contracting parties to the WIPO Copyright Treaty. In the United States, the WIPO Copyright Treaty is implemented by the Digital Millenium Copyright Act (DMCA). Since APPS was made in the United States, the DMCA is the relevant legislation that we must comply with. Notably, DMCA §1201 states, “No person shall circumvent a technological measure that effectively controls access to a work protected under this title.” We do not circumvent access controls when creating APPS and hence abide by §1201. Fair Use extends to content protected by the DMCA, for which we refer readers to the previous paragraph.</p>
</div>
<div id="A1.SS0.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="A1.SS0.SSS0.Px1.p3.1" class="ltx_p">Although GDPR only applies in the European Union, some of the ground-truth solutions in APPS may have been written by EU citizens. GDPR is chiefly concerned with the protection of personal data gathered by entities engaging in economic activity. The only personally linked information in APPS is the problem solutions written by individuals and published under aliases to public websites. In some cases, these solutions contain identifying information in comments, which we remove to preserve privacy. We comply with GDPR, because our processed solutions remove identifiers, and we are compliant because we collect the data for academic research purposes.</p>
</div>
</section>
<section id="A1.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Author Statement and License.</h4>

<div id="A1.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A1.SS0.SSS0.Px2.p1.1" class="ltx_p">We bear all responsibility in case of violation of rights. The APPS data is licensed under CC BY-SA 3.0 in accordance with the Kattis problem licenses and the ShareAlike terms. Our code is open sourced under the MIT license.</p>
</div>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Datasheets</h2>

<div id="A2.p1" class="ltx_para ltx_noindent">
<p id="A2.p1.1" class="ltx_p">We follow the recommendations of <cite class="ltx_cite ltx_citemacro_citet">Gebru et al. (<a href="#bib.bib10" title="" class="ltx_ref">2018</a>)</cite> and provide a datasheet for the ETHICS dataset in this section.</p>
</div>
<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>Motivation</h3>

<section id="A2.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">For what purpose was the dataset created? Was there a specific task
in mind? Was there a specific gap that needed to be filled? Please provide
a description.</h4>

<div id="A2.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS1.SSS0.Px1.p1.1" class="ltx_p">The APPS dataset was created to track the progress of code generation models on the task of generating arbitrary Python code from complex natural language specifications, a challenging setting that had no rigorous benchmark before our work.</p>
</div>
</section>
<section id="A2.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Who created the dataset (e.g., which team, research group) and on
behalf of which entity (e.g., company, institution, organization)?</h4>

<div id="A2.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A2.SS1.SSS0.Px2.p1.1" class="ltx_p">Refer to the main document.</p>
</div>
</section>
<section id="A2.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Who funded the creation of the dataset? If there is an associated
grant, please provide the name of the grantor and the grant name and
number.</h4>

<div id="A2.SS1.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="A2.SS1.SSS0.Px3.p1.1" class="ltx_p">There is no associated grant.</p>
</div>
</section>
<section id="A2.SS1.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Any other comments?</h4>

<div id="A2.SS1.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="A2.SS1.SSS0.Px4.p1.1" class="ltx_p">No.</p>
</div>
</section>
</section>
<section id="A2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.2 </span>Composition</h3>

<section id="A2.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">What do the instances that comprise the dataset represent (e.g.,
documents, photos, people, countries)? Are there multiple types of
instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.</h4>

<div id="A2.SS2.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px1.p1.1" class="ltx_p">The instances are coding challenge problems posed in natural language, each of which consists of question text, ground-truth solutions, and test cases. Please refer to the main document for more detail.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">How many instances are there in total (of each type, if appropriate)?</h4>

<div id="A2.SS2.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px2.p1.1" class="ltx_p">APPS contains 10,000 problems, 232,421 ground-truth solutions, and 131,777 test cases.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Does the dataset contain all possible instances or is it a sample
(not necessarily random) of instances from a larger set? If the
dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how
this representativeness was validated/verified. If it is not representative
of the larger set, please describe why not (e.g., to cover a more diverse
range of instances, because instances were withheld or unavailable).</h4>

<div id="A2.SS2.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px3.p1.1" class="ltx_p">APPS contains a subset of all possible test cases for its problems. These test cases are written by problem designers to cover important functionality.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features? In either case, please provide a description.</h4>

<div id="A2.SS2.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px4.p1.1" class="ltx_p">Each instance consists of text and numerical data.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Is there a label or target associated with each instance? If so, please
provide a description.</h4>

<div id="A2.SS2.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px5.p1.1" class="ltx_p">Each instance is associated with test cases, which provide a ground-truth signal for functional correctness.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px6" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Is any information missing from individual instances? If so, please
provide a description, explaining why this information is missing (e.g.,
because it was unavailable). This does not include intentionally removed
information, but might include, e.g., redacted text.</h4>

<div id="A2.SS2.SSS0.Px6.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px6.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px7" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Are relationships between individual instances made explicit
(e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit.</h4>

<div id="A2.SS2.SSS0.Px7.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px7.p1.1" class="ltx_p">We remove duplicate or near-duplicate problems from APPS.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px8" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Are there recommended data splits (e.g., training, development/validation,
testing)? If so, please provide a description of these splits, explaining
the rationale behind them.</h4>

<div id="A2.SS2.SSS0.Px8.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px8.p1.1" class="ltx_p">We provide a training and test split. The splits were optimized for increasing the number of test cases in the test split while maintaining a fixed number of problems from each difficulty.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px9" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Are there any errors, sources of noise, or redundancies in the
dataset? If so, please provide a description.</h4>

<div id="A2.SS2.SSS0.Px9.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px9.p1.1" class="ltx_p">See Section 3 in the main paper for a discussion of test case quality.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px10" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)?</h4>

<div id="A2.SS2.SSS0.Px10.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px10.p1.1" class="ltx_p">The dataset is self-contained.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px11" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals’ non-public communications)? If so, please provide a description.</h4>

<div id="A2.SS2.SSS0.Px11.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px11.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px12" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.</h4>

<div id="A2.SS2.SSS0.Px12.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px12.p1.1" class="ltx_p">Unknown.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px13" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Does the dataset relate to people? If not, you may skip the remaining
questions in this section.</h4>

<div id="A2.SS2.SSS0.Px13.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px13.p1.1" class="ltx_p">Yes.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px14" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and
provide a description of their respective distributions within the dataset.</h4>

<div id="A2.SS2.SSS0.Px14.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px14.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px15" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other
data) from the dataset? If so, please describe how</h4>

<div id="A2.SS2.SSS0.Px15.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px15.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px16" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Does the dataset contain data that might be considered sensitive
in any way (e.g., data that reveals racial or ethnic origins, sexual
orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.</h4>

<div id="A2.SS2.SSS0.Px16.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px16.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS2.SSS0.Px17" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Any other comments?</h4>

<div id="A2.SS2.SSS0.Px17.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.SSS0.Px17.p1.1" class="ltx_p">No.</p>
</div>
</section>
</section>
<section id="A2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.3 </span>Collection Process</h3>

<section id="A2.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">How was the data associated with each instance acquired? Was
the data directly observable (e.g., raw text, movie ratings), reported by
subjects (e.g., survey responses), or indirectly inferred/derived from other
data (e.g., part-of-speech tags, model-based guesses for age or language)?
If data was reported by subjects or indirectly inferred/derived from other
data, was the data validated/verified? If so, please describe how.</h4>

<div id="A2.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px1.p1.1" class="ltx_p">All data was collected by scraping problems from coding challenge websites, such as Codewars, AtCoder and Kattis.</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">What mechanisms or procedures were used to collect the data
(e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?</h4>

<div id="A2.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px2.p1.1" class="ltx_p">We used off-the-shelf and custom-built scrapers. We manually checked whether scraped data matched text on the websites.</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">If the dataset is a sample from a larger set, what was the sampling
strategy (e.g., deterministic, probabilistic with specific sampling
probabilities)?</h4>

<div id="A2.SS3.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px3.p1.1" class="ltx_p">Some problems we scraped were left out of APPS for various reasons, e.g. they required images to solve, they lacked ground-truth solutions and test cases, or they were duplicate problems.</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Who was involved in the data collection process (e.g., students,
crowdworkers, contractors) and how were they compensated (e.g.,
how much were crowdworkers paid)?</h4>

<div id="A2.SS3.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px4.p1.1" class="ltx_p">All data was collected by undergraduate and graduate student authors on the paper.</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Over what timeframe was the data collected? Does this timeframe
match the creation timeframe of the data associated with the instances
(e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.</h4>

<div id="A2.SS3.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px5.p1.1" class="ltx_p">Data was collected from late 2020 to early 2021 and refined for six months.</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px6" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review
processes, including the outcomes, as well as a link or other access point
to any supporting documentation</h4>

<div id="A2.SS3.SSS0.Px6.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px6.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px7" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Does the dataset relate to people? If not, you may skip the remainder
of the questions in this section.</h4>

<div id="A2.SS3.SSS0.Px7.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px7.p1.1" class="ltx_p">Yes.</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px8" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Did you collect the data from the individuals in question directly,
or obtain it via third parties or other sources (e.g., websites)?</h4>

<div id="A2.SS3.SSS0.Px8.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px8.p1.1" class="ltx_p">We scraped data via websites where individuals had publicly posted problem solutions.</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px9" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.</h4>

<div id="A2.SS3.SSS0.Px9.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px9.p1.1" class="ltx_p">Users who posted on the Internet were not notified of our collection, because their examples were posted publicly.</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px10" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Did the individuals in question consent to the collection and use
of their data? If so, please describe (or show with screenshots or other
information) how consent was requested and provided, and provide a
link or other access point to, or otherwise reproduce, the exact language
to which the individuals consented.</h4>

<div id="A2.SS3.SSS0.Px10.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px10.p1.1" class="ltx_p">N/A</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px11" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or
for certain uses? If so, please provide a description, as well as a link or
other access point to the mechanism (if appropriate).</h4>

<div id="A2.SS3.SSS0.Px11.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px11.p1.1" class="ltx_p">N/A</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px12" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Has an analysis of the potential impact of the dataset and its use
on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including
the outcomes, as well as a link or other access point to any supporting
documentation.</h4>

<div id="A2.SS3.SSS0.Px12.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px12.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS3.SSS0.Px13" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Any other comments?</h4>

<div id="A2.SS3.SSS0.Px13.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.SSS0.Px13.p1.1" class="ltx_p">No.</p>
</div>
</section>
</section>
<section id="A2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.4 </span>Preprocessing/Cleaning/Labeling</h3>

<section id="A2.SS4.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Was any preprocessing/cleaning/labeling of the data done (e.g.,
discretization or bucketing, tokenization, part-of-speech tagging,
SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the
remainder of the questions in this section.</h4>

<div id="A2.SS4.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS4.SSS0.Px1.p1.1" class="ltx_p">Yes, as described in Section 3 of the main paper.</p>
</div>
</section>
<section id="A2.SS4.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Was the “raw” data saved in addition to the preprocessed/cleaned/labeled
data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data.</h4>

<div id="A2.SS4.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A2.SS4.SSS0.Px2.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS4.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.</h4>

<div id="A2.SS4.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="A2.SS4.SSS0.Px3.p1.1" class="ltx_p">Not at this time.</p>
</div>
</section>
<section id="A2.SS4.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Any other comments?</h4>

<div id="A2.SS4.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="A2.SS4.SSS0.Px4.p1.1" class="ltx_p">No.</p>
</div>
</section>
</section>
<section id="A2.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.5 </span>Uses</h3>

<section id="A2.SS5.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Has the dataset been used for any tasks already? If so, please provide
a description.</h4>

<div id="A2.SS5.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS5.SSS0.Px1.p1.1" class="ltx_p">Yes, see the main paper.</p>
</div>
</section>
<section id="A2.SS5.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Is there a repository that links to any or all papers or systems that
use the dataset? If so, please provide a link or other access point.</h4>

<div id="A2.SS5.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A2.SS5.SSS0.Px2.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS5.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">What (other) tasks could the dataset be used for?</h4>

<div id="A2.SS5.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="A2.SS5.SSS0.Px3.p1.1" class="ltx_p">N/A</p>
</div>
</section>
<section id="A2.SS5.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Is there anything about the composition of the dataset or the way
it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user
might need to know to avoid uses that could result in unfair treatment
of individuals or groups (e.g., stereotyping, quality of service issues) or
other undesirable harms (e.g., financial harms, legal risks) If so, please
provide a description. Is there anything a future user could do to mitigate
these undesirable harms?</h4>

<div id="A2.SS5.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="A2.SS5.SSS0.Px4.p1.1" class="ltx_p">We describe how our data collection is legally compliant in Appendix A.</p>
</div>
</section>
<section id="A2.SS5.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Are there tasks for which the dataset should not be used? If so,
please provide a description.</h4>

<div id="A2.SS5.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="A2.SS5.SSS0.Px5.p1.1" class="ltx_p">N/A</p>
</div>
</section>
<section id="A2.SS5.SSS0.Px6" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Any other comments?</h4>

<div id="A2.SS5.SSS0.Px6.p1" class="ltx_para ltx_noindent">
<p id="A2.SS5.SSS0.Px6.p1.1" class="ltx_p">No.</p>
</div>
</section>
</section>
<section id="A2.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.6 </span>Distribution</h3>

<section id="A2.SS6.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which
the dataset was created? If so, please provide a description.</h4>

<div id="A2.SS6.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS6.SSS0.Px1.p1.1" class="ltx_p">Yes, the dataset will be publicly distributed.</p>
</div>
</section>
<section id="A2.SS6.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">How will the dataset will be distributed (e.g., tarball on website,
API, GitHub)? Does the dataset have a digital object identifier (DOI)?</h4>

<div id="A2.SS6.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A2.SS6.SSS0.Px2.p1.1" class="ltx_p">The dataset is available at <a target="_blank" href="https://github.com/hendrycks/apps" title="" class="ltx_ref ltx_href">https://github.com/hendrycks/apps</a>.</p>
</div>
</section>
<section id="A2.SS6.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">When will the dataset be distributed?</h4>

<div id="A2.SS6.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="A2.SS6.SSS0.Px3.p1.1" class="ltx_p">The dataset is currently available.</p>
</div>
</section>
<section id="A2.SS6.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use
(ToU)? If so, please describe this license and/or ToU, and provide a link
or other access point to, or otherwise reproduce, any relevant licensing
terms or ToU, as well as any fees associated with these restrictions.</h4>

<div id="A2.SS6.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="A2.SS6.SSS0.Px4.p1.1" class="ltx_p">The code for our experimental framework is distributed under an MIT license. Where applicable,</p>
</div>
</section>
<section id="A2.SS6.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Have any third parties imposed IP-based or other restrictions on
the data associated with the instances? If so, please describe these
restrictions, and provide a link or other access point to, or otherwise
reproduce, any relevant licensing terms, as well as any fees associated
with these restrictions.</h4>

<div id="A2.SS6.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="A2.SS6.SSS0.Px5.p1.1" class="ltx_p">In cases where websites that we scrape data from have copyright policies, we abide by Fair Use according to §107, and we comply with GDPR even though all our problem sources with ground-truth solutions are based in the US. See Appendix A for details.</p>
</div>
</section>
<section id="A2.SS6.SSS0.Px6" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Do any export controls or other regulatory restrictions apply to
the dataset or to individual instances? If so, please describe these
restrictions, and provide a link or other access point to, or otherwise
reproduce, any supporting documentation.</h4>

<div id="A2.SS6.SSS0.Px6.p1" class="ltx_para ltx_noindent">
<p id="A2.SS6.SSS0.Px6.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS6.SSS0.Px7" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Any other comments?</h4>

<div id="A2.SS6.SSS0.Px7.p1" class="ltx_para ltx_noindent">
<p id="A2.SS6.SSS0.Px7.p1.1" class="ltx_p">No.</p>
</div>
</section>
</section>
<section id="A2.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.7 </span>Maintenance</h3>

<section id="A2.SS7.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Who is supporting/hosting/maintaining the dataset?</h4>

<div id="A2.SS7.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS7.SSS0.Px1.p1.1" class="ltx_p">Refer to the main document.</p>
</div>
</section>
<section id="A2.SS7.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">How can the owner/curator/manager of the dataset be contacted
(e.g., email address)?</h4>

<div id="A2.SS7.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A2.SS7.SSS0.Px2.p1.1" class="ltx_p">Refer to the main document.</p>
</div>
</section>
<section id="A2.SS7.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Is there an erratum? If so, please provide a link or other access point.</h4>

<div id="A2.SS7.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="A2.SS7.SSS0.Px3.p1.1" class="ltx_p">Not at this time.</p>
</div>
</section>
<section id="A2.SS7.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Will the dataset be updated (e.g., to correct labeling errors, add
new instances, delete instances)? If so, please describe how often, by
whom, and how updates will be communicated to users (e.g., mailing list,
GitHub)?</h4>

<div id="A2.SS7.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="A2.SS7.SSS0.Px4.p1.1" class="ltx_p">We plan to update the dataset with an additional JSON of test cases present in the question text for each problem. This will be available through GitHub.</p>
</div>
</section>
<section id="A2.SS7.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">If the dataset relates to people, are there applicable limits on the
retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a
fixed period of time and then deleted)? If so, please describe these
limits and explain how they will be enforced</h4>

<div id="A2.SS7.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="A2.SS7.SSS0.Px5.p1.1" class="ltx_p">No.</p>
</div>
</section>
<section id="A2.SS7.SSS0.Px6" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Will older versions of the dataset continue to be supported/hosted/maintained?
If so, please describe how. If not, please describe how its obsolescence
will be communicated to users.</h4>

<div id="A2.SS7.SSS0.Px6.p1" class="ltx_para ltx_noindent">
<p id="A2.SS7.SSS0.Px6.p1.1" class="ltx_p">N/A</p>
</div>
</section>
<section id="A2.SS7.SSS0.Px7" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">If others want to extend/augment/build on/contribute to the
dataset, is there a mechanism for them to do so? If so, please
provide a description. Will these contributions be validated/verified?
If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please
provide a description.</h4>

<div id="A2.SS7.SSS0.Px7.p1" class="ltx_para ltx_noindent">
<p id="A2.SS7.SSS0.Px7.p1.1" class="ltx_p">Our dataset could be extended with additional problems that follow the formatting of existing problems.</p>
</div>
</section>
<section id="A2.SS7.SSS0.Px8" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Any other comments?</h4>

<div id="A2.SS7.SSS0.Px8.p1" class="ltx_para ltx_noindent">
<p id="A2.SS7.SSS0.Px8.p1.1" class="ltx_p">No.</p>
</div>
</section>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Additional Dataset Information</h2>

<section id="A3.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Expanded Dataset Comparisons.</h4>

<div id="A3.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A3.SS0.SSS0.Px1.p1.1" class="ltx_p">We compared to several datasets in the <cite class="ltx_cite ltx_citemacro_citep">(Kulal et al., <a href="#bib.bib17" title="" class="ltx_ref">2019</a>; Yu et al., <a href="#bib.bib39" title="" class="ltx_ref">2018</a>; Raychev et al., <a href="#bib.bib30" title="" class="ltx_ref">2016</a>; Iyer et al., <a href="#bib.bib16" title="" class="ltx_ref">2018</a>; Lu et al., <a href="#bib.bib23" title="" class="ltx_ref">2021</a>)</cite> main paper. We continue the comparisons below.
<cite class="ltx_cite ltx_citemacro_citet">Ling et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite> introduce datasets based on Hearthstone and Magic the Gathering card games for code generation. <cite class="ltx_cite ltx_citemacro_citet">Oda et al. (<a href="#bib.bib24" title="" class="ltx_ref">2015</a>)</cite> provide a language-to-code dataset using simple code comments. <cite class="ltx_cite ltx_citemacro_citet">Zavershynskyi et al. (<a href="#bib.bib40" title="" class="ltx_ref">2018</a>)</cite> introduce the NAPS dataset for converting pseudocode to code, obtained by crowdsourcing low-level descriptions of programming exercises, and apply machine translation techniques to the problem. Recent anecdotal posts on social media have demonstrated that modern Transformers can in some instances generate JSX code adhering to user requests, but our work provides precision to the discussion through quantitative evaluation. <cite class="ltx_cite ltx_citemacro_citet">Allamanis and Sutton (<a href="#bib.bib1" title="" class="ltx_ref">2013</a>)</cite> introduce the GitHub Java Corpus used for performing language modeling on Java code. <cite class="ltx_cite ltx_citemacro_citet">Liu et al. (<a href="#bib.bib20" title="" class="ltx_ref">2020a</a>)</cite> do a smaller-scale analysis of code generation but with their limited language-specific training data models “fail to pass even a single predefined test case” on their 300 test problems, while with our large training set and test set, trained models can pass tens of thousands of test cases. <cite class="ltx_cite ltx_citemacro_citet">Zelle and Mooney (<a href="#bib.bib41" title="" class="ltx_ref">1996</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Tang and Mooney (<a href="#bib.bib33" title="" class="ltx_ref">2001</a>)</cite> precedes <cite class="ltx_cite ltx_citemacro_citet">Yu et al. (<a href="#bib.bib39" title="" class="ltx_ref">2018</a>)</cite> by also facilitating the synthesis of database queries, though more recent program synthesis works such as <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib37" title="" class="ltx_ref">2019c</a>)</cite> use Spider from <cite class="ltx_cite ltx_citemacro_citet">Yu et al. (<a href="#bib.bib39" title="" class="ltx_ref">2018</a>)</cite>.</p>
</div>
<div id="A3.SS0.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="A3.SS0.SSS0.Px1.p2.1" class="ltx_p"><a href="#A0.T4" title="In Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">4</span></a> compares APPS to Hearthstone <cite class="ltx_cite ltx_citemacro_citep">(Ling et al., <a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite>, Django <cite class="ltx_cite ltx_citemacro_citep">(Oda et al., <a href="#bib.bib24" title="" class="ltx_ref">2015</a>)</cite>, and <cite class="ltx_cite ltx_citemacro_citet">Zavershynskyi et al. (<a href="#bib.bib40" title="" class="ltx_ref">2018</a>)</cite>. ‘Number of Programs’ refers to the number of human-written programs or functions in the dataset, and ‘Number of Exercises’ refers to the number of tasks that the network must solve. These numbers can differ in datasets such as APPS with multiple human-written solutions per exercise.</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Excluded Keywords.</h4>

<div id="A3.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A3.SS0.SSS0.Px2.p1.1" class="ltx_p">In creating the GitHub pretraining dataset, we exclude the following keywords to prevent overlap with coding challenge questions similar to those in APPS: ‘atcoder’, ‘coderbyte’, ‘leetcode’, ‘codeforces’, ‘codewars’, ‘hackerrank’, ‘topcoder’, ‘codechef’, ‘checkio’, ‘HackerEarth’, ‘Programmr’, ‘Exercism’, ‘Codier’, ‘PyBites’, ‘Tynker’, ‘CodinGame’, ‘CodeCombat’, ‘usaco’, ‘IOI’, ‘UVA’, ‘ICFP’, ‘EPIJudge’, ‘SPOJ’, ‘UVaOJ’, ‘judge’, ‘interview’, ‘solution’, ‘coding’, ‘code’, ‘problem’, ‘exercise’, ‘challenge’, ‘algo’, ‘practice’, ‘competitive’, ‘program’.</p>
</div>
</section>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Additional Results</h2>

<section id="A4.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Top-5 Performance.</h4>

<div id="A4.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A4.SS0.SSS0.Px1.p1.1" class="ltx_p">Rather than allowing models to generate just one potential solution, we let models generate five and we choose the best performing solution. Full top-5 performance results are in <a href="#A0.T5" title="In Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
</section>
<section id="A4.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">GPT-3.</h4>

<div id="A4.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A4.SS0.SSS0.Px2.p1.1" class="ltx_p">We evaluate GPT-3 175B on APPS in a few-shot setting. A separate prompt is used for standard input and call-based questions, and each prompt includes instruction text along with two example questions and solutions from the corresponding question type. The solutions we select for each prompt are the shortest solutions passing all test cases for the corresponding problems. In preliminary experiments, we found that the original GPT-3 175B did not perform well, and the instruct series was necessary for reasonable generations. Inference on the <math id="A4.SS0.SSS0.Px2.p1.1.m1.2" class="ltx_Math" alttext="5,\!000" display="inline"><semantics id="A4.SS0.SSS0.Px2.p1.1.m1.2a"><mrow id="A4.SS0.SSS0.Px2.p1.1.m1.2.3.2" xref="A4.SS0.SSS0.Px2.p1.1.m1.2.3.1.cmml"><mn id="A4.SS0.SSS0.Px2.p1.1.m1.1.1" xref="A4.SS0.SSS0.Px2.p1.1.m1.1.1.cmml">5</mn><mpadded width="0.275em"><mo id="A4.SS0.SSS0.Px2.p1.1.m1.2.3.2.1" xref="A4.SS0.SSS0.Px2.p1.1.m1.2.3.1.cmml">,</mo></mpadded><mn id="A4.SS0.SSS0.Px2.p1.1.m1.2.2" xref="A4.SS0.SSS0.Px2.p1.1.m1.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.SS0.SSS0.Px2.p1.1.m1.2b"><list id="A4.SS0.SSS0.Px2.p1.1.m1.2.3.1.cmml" xref="A4.SS0.SSS0.Px2.p1.1.m1.2.3.2"><cn type="integer" id="A4.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="A4.SS0.SSS0.Px2.p1.1.m1.1.1">5</cn><cn type="integer" id="A4.SS0.SSS0.Px2.p1.1.m1.2.2.cmml" xref="A4.SS0.SSS0.Px2.p1.1.m1.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A4.SS0.SSS0.Px2.p1.1.m1.2c">5,\!000</annotation></semantics></math> problems in the APPS test set is possible for approximately $500.</p>
</div>
<div id="A4.SS0.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="A4.SS0.SSS0.Px2.p2.2" class="ltx_p">We find that GPT-3 only solves <math id="A4.SS0.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="3" display="inline"><semantics id="A4.SS0.SSS0.Px2.p2.1.m1.1a"><mn id="A4.SS0.SSS0.Px2.p2.1.m1.1.1" xref="A4.SS0.SSS0.Px2.p2.1.m1.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="A4.SS0.SSS0.Px2.p2.1.m1.1b"><cn type="integer" id="A4.SS0.SSS0.Px2.p2.1.m1.1.1.cmml" xref="A4.SS0.SSS0.Px2.p2.1.m1.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="A4.SS0.SSS0.Px2.p2.1.m1.1c">3</annotation></semantics></math> problems out of <math id="A4.SS0.SSS0.Px2.p2.2.m2.2" class="ltx_Math" alttext="5,\!000" display="inline"><semantics id="A4.SS0.SSS0.Px2.p2.2.m2.2a"><mrow id="A4.SS0.SSS0.Px2.p2.2.m2.2.3.2" xref="A4.SS0.SSS0.Px2.p2.2.m2.2.3.1.cmml"><mn id="A4.SS0.SSS0.Px2.p2.2.m2.1.1" xref="A4.SS0.SSS0.Px2.p2.2.m2.1.1.cmml">5</mn><mpadded width="0.275em"><mo id="A4.SS0.SSS0.Px2.p2.2.m2.2.3.2.1" xref="A4.SS0.SSS0.Px2.p2.2.m2.2.3.1.cmml">,</mo></mpadded><mn id="A4.SS0.SSS0.Px2.p2.2.m2.2.2" xref="A4.SS0.SSS0.Px2.p2.2.m2.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.SS0.SSS0.Px2.p2.2.m2.2b"><list id="A4.SS0.SSS0.Px2.p2.2.m2.2.3.1.cmml" xref="A4.SS0.SSS0.Px2.p2.2.m2.2.3.2"><cn type="integer" id="A4.SS0.SSS0.Px2.p2.2.m2.1.1.cmml" xref="A4.SS0.SSS0.Px2.p2.2.m2.1.1">5</cn><cn type="integer" id="A4.SS0.SSS0.Px2.p2.2.m2.2.2.cmml" xref="A4.SS0.SSS0.Px2.p2.2.m2.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A4.SS0.SSS0.Px2.p2.2.m2.2c">5,\!000</annotation></semantics></math>: two introductory problems and one interview problem. The two introductory problems are simple interpretation tasks, such as implementing a specified algebraic expression. The interview problem requires higher-level thinking and is show in Figure <a href="#A4.F7" title="Figure 7 ‣ GPT-3. ‣ Appendix D Additional Results ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>. Legitimately solving this problem would suggest nontrivial reasoning. However, it is possible that GPT-3 memorized the solution during pretraining, or that it took a lucky guess based on heuristics in the question.</p>
</div>
<div id="A4.SS0.SSS0.Px2.p3" class="ltx_para ltx_noindent">
<p id="A4.SS0.SSS0.Px2.p3.1" class="ltx_p">We also find that GPT-3 tends to handle syntax poorly. In Figure <a href="#A4.F8" title="Figure 8 ‣ GPT-3. ‣ Appendix D Additional Results ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>, we see a generation with improper formatting of an if-else block, which causes a syntax error. By contrast, GPT-2 0.1B has fewer syntax errors with three orders of magnitude fewer parameters, and GPT-Neo 2.7B has almost no syntax errors. These two models are fine-tuned on GitHub and the APPS training set.</p>
</div>
<figure id="A4.F7" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F7.6" class="ltx_p ltx_figure_panel"><span id="A4.F7.6.1" class="ltx_text ltx_font_bold">Problem</span> 
<br class="ltx_break">Mahmoud and Ehab play a game called the even-odd game. Ehab chooses his favorite integer <math id="A4.F7.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A4.F7.1.m1.1a"><mi id="A4.F7.1.m1.1.1" xref="A4.F7.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A4.F7.1.m1.1b"><ci id="A4.F7.1.m1.1.1.cmml" xref="A4.F7.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.F7.1.m1.1c">n</annotation></semantics></math> and then they take turns, starting from Mahmoud. In each player’s turn, he has to choose an integer <math id="A4.F7.2.m2.1" class="ltx_Math" alttext="a" display="inline"><semantics id="A4.F7.2.m2.1a"><mi id="A4.F7.2.m2.1.1" xref="A4.F7.2.m2.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="A4.F7.2.m2.1b"><ci id="A4.F7.2.m2.1.1.cmml" xref="A4.F7.2.m2.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.F7.2.m2.1c">a</annotation></semantics></math> and subtract it from <math id="A4.F7.3.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A4.F7.3.m3.1a"><mi id="A4.F7.3.m3.1.1" xref="A4.F7.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A4.F7.3.m3.1b"><ci id="A4.F7.3.m3.1.1.cmml" xref="A4.F7.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.F7.3.m3.1c">n</annotation></semantics></math> such that: <math id="A4.F7.4.m4.1" class="ltx_Math" alttext="1\leq a\leq n" display="inline"><semantics id="A4.F7.4.m4.1a"><mrow id="A4.F7.4.m4.1.1" xref="A4.F7.4.m4.1.1.cmml"><mn id="A4.F7.4.m4.1.1.2" xref="A4.F7.4.m4.1.1.2.cmml">1</mn><mo id="A4.F7.4.m4.1.1.3" xref="A4.F7.4.m4.1.1.3.cmml">≤</mo><mi id="A4.F7.4.m4.1.1.4" xref="A4.F7.4.m4.1.1.4.cmml">a</mi><mo id="A4.F7.4.m4.1.1.5" xref="A4.F7.4.m4.1.1.5.cmml">≤</mo><mi id="A4.F7.4.m4.1.1.6" xref="A4.F7.4.m4.1.1.6.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="A4.F7.4.m4.1b"><apply id="A4.F7.4.m4.1.1.cmml" xref="A4.F7.4.m4.1.1"><and id="A4.F7.4.m4.1.1a.cmml" xref="A4.F7.4.m4.1.1"></and><apply id="A4.F7.4.m4.1.1b.cmml" xref="A4.F7.4.m4.1.1"><leq id="A4.F7.4.m4.1.1.3.cmml" xref="A4.F7.4.m4.1.1.3"></leq><cn type="integer" id="A4.F7.4.m4.1.1.2.cmml" xref="A4.F7.4.m4.1.1.2">1</cn><ci id="A4.F7.4.m4.1.1.4.cmml" xref="A4.F7.4.m4.1.1.4">𝑎</ci></apply><apply id="A4.F7.4.m4.1.1c.cmml" xref="A4.F7.4.m4.1.1"><leq id="A4.F7.4.m4.1.1.5.cmml" xref="A4.F7.4.m4.1.1.5"></leq><share href="#A4.F7.4.m4.1.1.4.cmml" id="A4.F7.4.m4.1.1d.cmml" xref="A4.F7.4.m4.1.1"></share><ci id="A4.F7.4.m4.1.1.6.cmml" xref="A4.F7.4.m4.1.1.6">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.F7.4.m4.1c">1\leq a\leq n</annotation></semantics></math>. If it’s Mahmoud’s turn, <math id="A4.F7.5.m5.1" class="ltx_Math" alttext="a" display="inline"><semantics id="A4.F7.5.m5.1a"><mi id="A4.F7.5.m5.1.1" xref="A4.F7.5.m5.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="A4.F7.5.m5.1b"><ci id="A4.F7.5.m5.1.1.cmml" xref="A4.F7.5.m5.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.F7.5.m5.1c">a</annotation></semantics></math> has to be even, but if it’s Ehab’s turn, <math id="A4.F7.6.m6.1" class="ltx_Math" alttext="a" display="inline"><semantics id="A4.F7.6.m6.1a"><mi id="A4.F7.6.m6.1.1" xref="A4.F7.6.m6.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="A4.F7.6.m6.1b"><ci id="A4.F7.6.m6.1.1.cmml" xref="A4.F7.6.m6.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.F7.6.m6.1c">a</annotation></semantics></math> has to be odd.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F7.9" class="ltx_p ltx_figure_panel">If the current player can’t choose any number satisfying the conditions, he loses. Can you determine the winner if they both play optimally?</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F7.10" class="ltx_p ltx_figure_panel">—–Input—–</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F7.8" class="ltx_p ltx_figure_panel">The only line contains an integer <math id="A4.F7.7.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A4.F7.7.m1.1a"><mi id="A4.F7.7.m1.1.1" xref="A4.F7.7.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A4.F7.7.m1.1b"><ci id="A4.F7.7.m1.1.1.cmml" xref="A4.F7.7.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.F7.7.m1.1c">n</annotation></semantics></math> <math id="A4.F7.8.m2.1" class="ltx_Math" alttext="(1\leq n\leq 10^{9})" display="inline"><semantics id="A4.F7.8.m2.1a"><mrow id="A4.F7.8.m2.1.1.1" xref="A4.F7.8.m2.1.1.1.1.cmml"><mo stretchy="false" id="A4.F7.8.m2.1.1.1.2" xref="A4.F7.8.m2.1.1.1.1.cmml">(</mo><mrow id="A4.F7.8.m2.1.1.1.1" xref="A4.F7.8.m2.1.1.1.1.cmml"><mn id="A4.F7.8.m2.1.1.1.1.2" xref="A4.F7.8.m2.1.1.1.1.2.cmml">1</mn><mo id="A4.F7.8.m2.1.1.1.1.3" xref="A4.F7.8.m2.1.1.1.1.3.cmml">≤</mo><mi id="A4.F7.8.m2.1.1.1.1.4" xref="A4.F7.8.m2.1.1.1.1.4.cmml">n</mi><mo id="A4.F7.8.m2.1.1.1.1.5" xref="A4.F7.8.m2.1.1.1.1.5.cmml">≤</mo><msup id="A4.F7.8.m2.1.1.1.1.6" xref="A4.F7.8.m2.1.1.1.1.6.cmml"><mn id="A4.F7.8.m2.1.1.1.1.6.2" xref="A4.F7.8.m2.1.1.1.1.6.2.cmml">10</mn><mn id="A4.F7.8.m2.1.1.1.1.6.3" xref="A4.F7.8.m2.1.1.1.1.6.3.cmml">9</mn></msup></mrow><mo stretchy="false" id="A4.F7.8.m2.1.1.1.3" xref="A4.F7.8.m2.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.F7.8.m2.1b"><apply id="A4.F7.8.m2.1.1.1.1.cmml" xref="A4.F7.8.m2.1.1.1"><and id="A4.F7.8.m2.1.1.1.1a.cmml" xref="A4.F7.8.m2.1.1.1"></and><apply id="A4.F7.8.m2.1.1.1.1b.cmml" xref="A4.F7.8.m2.1.1.1"><leq id="A4.F7.8.m2.1.1.1.1.3.cmml" xref="A4.F7.8.m2.1.1.1.1.3"></leq><cn type="integer" id="A4.F7.8.m2.1.1.1.1.2.cmml" xref="A4.F7.8.m2.1.1.1.1.2">1</cn><ci id="A4.F7.8.m2.1.1.1.1.4.cmml" xref="A4.F7.8.m2.1.1.1.1.4">𝑛</ci></apply><apply id="A4.F7.8.m2.1.1.1.1c.cmml" xref="A4.F7.8.m2.1.1.1"><leq id="A4.F7.8.m2.1.1.1.1.5.cmml" xref="A4.F7.8.m2.1.1.1.1.5"></leq><share href="#A4.F7.8.m2.1.1.1.1.4.cmml" id="A4.F7.8.m2.1.1.1.1d.cmml" xref="A4.F7.8.m2.1.1.1"></share><apply id="A4.F7.8.m2.1.1.1.1.6.cmml" xref="A4.F7.8.m2.1.1.1.1.6"><csymbol cd="ambiguous" id="A4.F7.8.m2.1.1.1.1.6.1.cmml" xref="A4.F7.8.m2.1.1.1.1.6">superscript</csymbol><cn type="integer" id="A4.F7.8.m2.1.1.1.1.6.2.cmml" xref="A4.F7.8.m2.1.1.1.1.6.2">10</cn><cn type="integer" id="A4.F7.8.m2.1.1.1.1.6.3.cmml" xref="A4.F7.8.m2.1.1.1.1.6.3">9</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.F7.8.m2.1c">(1\leq n\leq 10^{9})</annotation></semantics></math>, the number at the beginning of the game.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F7.11" class="ltx_p ltx_figure_panel">—–Output—–</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F7.12" class="ltx_p ltx_figure_panel">Output “Mahmoud” (without quotes) if Mahmoud wins and “Ehab” (without quotes) otherwise.

<br class="ltx_break">
<br class="ltx_break"><span id="A4.F7.12.1" class="ltx_text ltx_font_bold">Model Output</span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A4.F7.13" class="ltx_listing ltx_figure_panel ltx_lst_numbers_left ltx_lstlisting ltx_listing" style="background-color:#FFFFFF;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,Cm4gPSBpbnQoaW5wdXQoKSkKaWYgbiBwcmludCgnTWFobW91ZCcpCmVsc2U6CnByaW50KCdFaGFiJykK" download="">⬇</a></div>
<div id="lstnumberx21" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>
</div>
<div id="lstnumberx22" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="lstnumberx22.1" class="ltx_text ltx_lst_identifier">n</span><span id="lstnumberx22.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx22.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx22.4" class="ltx_text ltx_lst_identifier">int</span>(<span id="lstnumberx22.5" class="ltx_text ltx_lst_identifier">input</span>())
</div>
<div id="lstnumberx23" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span><span id="lstnumberx23.1" class="ltx_text ltx_lst_identifier">if</span><span id="lstnumberx23.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx23.3" class="ltx_text ltx_lst_identifier">n</span><span id="lstnumberx23.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx23.5" class="ltx_text ltx_lst_identifier">print</span>(’<span id="lstnumberx23.6" class="ltx_text ltx_lst_identifier">Mahmoud</span>’)
</div>
<div id="lstnumberx24" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span><span id="lstnumberx24.1" class="ltx_text ltx_lst_identifier">else</span>:
</div>
<div id="lstnumberx25" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span><span id="lstnumberx25.1" class="ltx_text ltx_lst_identifier">print</span>(’<span id="lstnumberx25.2" class="ltx_text ltx_lst_identifier">Ehab</span>’)
</div>
</div>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>The sole interview problem solved by GPT-3. Legitimately solving this problem requires nontrivial reasoning about the proposed game’s branching possibilities.</figcaption>
</figure>
<figure id="A4.F8" class="ltx_figure">
<div id="A4.F8.1" class="ltx_listing ltx_lst_numbers_left ltx_lstlisting ltx_listing" style="background-color:#FFFFFF;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,CmEsIGIgPSBpbnQoaW5wdXQoKSksIGludChpbnB1dCgpKQppZiBhID09IGI6ClxwYXJwcmludCgnJy5qb2luKG1hcChzdHIsIFthLCBiXSkpKQpccGFyZWxzZToKXHBhcnByaW50KCcnLmpvaW4obWFwKHN0ciwgW2EsIGJdKSkpCg==" download="">⬇</a></div>
<div id="lstnumberx26" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>
</div>
<div id="lstnumberx27" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="lstnumberx27.1" class="ltx_text ltx_lst_identifier">a</span>,<span id="lstnumberx27.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx27.3" class="ltx_text ltx_lst_identifier">b</span><span id="lstnumberx27.4" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx27.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx27.6" class="ltx_text ltx_lst_identifier">int</span>(<span id="lstnumberx27.7" class="ltx_text ltx_lst_identifier">input</span>()),<span id="lstnumberx27.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx27.9" class="ltx_text ltx_lst_identifier">int</span>(<span id="lstnumberx27.10" class="ltx_text ltx_lst_identifier">input</span>())
</div>
<div id="lstnumberx28" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span><span id="lstnumberx28.1" class="ltx_text ltx_lst_identifier">if</span><span id="lstnumberx28.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx28.3" class="ltx_text ltx_lst_identifier">a</span><span id="lstnumberx28.4" class="ltx_text ltx_lst_space"> </span>==<span id="lstnumberx28.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx28.6" class="ltx_text ltx_lst_identifier">b</span>:
</div>
<div id="lstnumberx29" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span>\<span id="lstnumberx29.1" class="ltx_text ltx_lst_identifier">parprint</span>(”.<span id="lstnumberx29.2" class="ltx_text ltx_lst_identifier">join</span>(<span id="lstnumberx29.3" class="ltx_text ltx_lst_identifier">map</span>(<span id="lstnumberx29.4" class="ltx_text ltx_lst_identifier">str</span>,<span id="lstnumberx29.5" class="ltx_text ltx_lst_space"> </span>[<span id="lstnumberx29.6" class="ltx_text ltx_lst_identifier">a</span>,<span id="lstnumberx29.7" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx29.8" class="ltx_text ltx_lst_identifier">b</span>])))
</div>
<div id="lstnumberx30" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span>\<span id="lstnumberx30.1" class="ltx_text ltx_lst_identifier">parelse</span>:
</div>
<div id="lstnumberx31" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6</span>\<span id="lstnumberx31.1" class="ltx_text ltx_lst_identifier">parprint</span>(”.<span id="lstnumberx31.2" class="ltx_text ltx_lst_identifier">join</span>(<span id="lstnumberx31.3" class="ltx_text ltx_lst_identifier">map</span>(<span id="lstnumberx31.4" class="ltx_text ltx_lst_identifier">str</span>,<span id="lstnumberx31.5" class="ltx_text ltx_lst_space"> </span>[<span id="lstnumberx31.6" class="ltx_text ltx_lst_identifier">a</span>,<span id="lstnumberx31.7" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx31.8" class="ltx_text ltx_lst_identifier">b</span>])))
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>Oftentimes, GPT-3 generates outputs with incorrect syntax, such as the above code. Even though the code is reasonable, bad formatting of the if-else block causes a syntax error.</figcaption>
</figure>
</section>
<section id="A4.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">More Generated Code Examples.</h4>

<div id="A4.SS0.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="A4.SS0.SSS0.Px3.p1.1" class="ltx_p">Additional examples of generated code can be found in <a href="#A4.F9" title="In More Generated Code Examples. ‣ Appendix D Additional Results ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">9</span></a>, <a href="#A4.F10" title="In More Generated Code Examples. ‣ Appendix D Additional Results ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">10</span></a>, and <a href="#A4.F11" title="In More Generated Code Examples. ‣ Appendix D Additional Results ‣ Measuring Coding Challenge Competence With APPS" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">11</span></a>.</p>
</div>
<figure id="A4.F9" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F9.1" class="ltx_p ltx_figure_panel"><span id="A4.F9.1.1" class="ltx_text ltx_font_bold">Problem</span> 
<br class="ltx_break">Given is a directed graph G with N vertices and M edges. The vertices are numbered 1 to N, and the i-th edge is directed from Vertex A_i to Vertex B_i. It is guaranteed that the graph contains no self-loops or multiple edges. Determine whether there exists an induced subgraph (see Notes) of G such that the in-degree and out-degree of every vertex are both 1. If the answer is yes, show one such subgraph. Here the null graph is not considered as a subgraph.

<br class="ltx_break">
<br class="ltx_break"><span id="A4.F9.1.2" class="ltx_text ltx_font_bold">Model Output</span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A4.F9.2" class="ltx_listing ltx_figure_panel ltx_lst_numbers_left ltx_lstlisting ltx_listing" style="background-color:#FFFFFF;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,CmltcG9ydCBzeXMKXHBhcnN5cy5zZXRyZWN1cnNpb25saW1pdCgxMCAqKiA3KQppbnB1dCA9IHN5cy5zdGRpbi5yZWFkbGluZQpmX2luZiA9IGZsb2F0KCdpbmYnKQptb2QgPSAxMCAqKiA5ICsgNwpccGFyXHBhcmRlZiByZXNvbHZlKCk6Cm4sIG0gPSBtYXAoaW50LCBpbnB1dCgpLnNwbGl0KCkpCmVkZ2UgPSBbW10gZm9yIF8gaW4gcmFuZ2UobildCmZvciBfIGluIHJhbmdlKG0pOgphLCBiID0gbWFwKGludCwgaW5wdXQoKS5zcGxpdCgpKQplZGdlW2EgLSAxXS5hcHBlbmQoYiAtIDEpCmVkZ2VbYiAtIDFdLmFwcGVuZChhIC0gMSkKXHBhcmRlZiBkZnModSk6CnN0YWNrID0gW3VdCnNlZW4gPSBbRmFsc2VdICogbgp3aGlsZSBzdGFjazoKdSA9IHN0YWNrLnBvcCgpCmlmIHNlZW5bdV06CmNvbnRpbnVlCnNlZW5bdV0gPSBUcnVlCmZvciB2IGluIGVkZ2VbdV06CmlmIHNlZW5bdl06CmNvbnRpbnVlCnNlZW5bdl0gPSBUcnVlCnN0YWNrLmFwcGVuZCh2KQpccGFycmVzID0gTm9uZQpmb3IgaSBpbiByYW5nZShuKToKaWYgbm90IHNlZW5baV06CnNlZW5baV0gPSBUcnVlCnJlcyA9IGRmcyhpKQppZiBub3QgcmVzOgpwcmludCgtMSkKZWxzZToKcHJpbnQocmVzKQpccGFyXHBhcmlmIF9fbmFtZV9fID09ICdfX21haW5fXyc6CnJlc29sdmUoKQo=" download="">⬇</a></div>
<div id="lstnumberx32" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>
</div>
<div id="lstnumberx33" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="lstnumberx33.1" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx33.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx33.3" class="ltx_text ltx_lst_identifier">sys</span>
</div>
<div id="lstnumberx34" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span>\<span id="lstnumberx34.1" class="ltx_text ltx_lst_identifier">parsys</span>.<span id="lstnumberx34.2" class="ltx_text ltx_lst_identifier">setrecursionlimit</span>(10<span id="lstnumberx34.3" class="ltx_text ltx_lst_space"> </span>**<span id="lstnumberx34.4" class="ltx_text ltx_lst_space"> </span>7)
</div>
<div id="lstnumberx35" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span><span id="lstnumberx35.1" class="ltx_text ltx_lst_identifier">input</span><span id="lstnumberx35.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx35.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx35.4" class="ltx_text ltx_lst_identifier">sys</span>.<span id="lstnumberx35.5" class="ltx_text ltx_lst_identifier">stdin</span>.<span id="lstnumberx35.6" class="ltx_text ltx_lst_identifier">readline</span>
</div>
<div id="lstnumberx36" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span><span id="lstnumberx36.1" class="ltx_text ltx_lst_identifier">f_inf</span><span id="lstnumberx36.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx36.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx36.4" class="ltx_text ltx_lst_identifier">float</span>(’<span id="lstnumberx36.5" class="ltx_text ltx_lst_identifier">inf</span>’)
</div>
<div id="lstnumberx37" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6</span><span id="lstnumberx37.1" class="ltx_text ltx_lst_identifier">mod</span><span id="lstnumberx37.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx37.3" class="ltx_text ltx_lst_space"> </span>10<span id="lstnumberx37.4" class="ltx_text ltx_lst_space"> </span>**<span id="lstnumberx37.5" class="ltx_text ltx_lst_space"> </span>9<span id="lstnumberx37.6" class="ltx_text ltx_lst_space"> </span>+<span id="lstnumberx37.7" class="ltx_text ltx_lst_space"> </span>7
</div>
<div id="lstnumberx38" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7</span>\<span id="lstnumberx38.1" class="ltx_text ltx_lst_identifier">par</span>\<span id="lstnumberx38.2" class="ltx_text ltx_lst_identifier">pardef</span><span id="lstnumberx38.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx38.4" class="ltx_text ltx_lst_identifier">resolve</span>():
</div>
<div id="lstnumberx39" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8</span><span id="lstnumberx39.1" class="ltx_text ltx_lst_identifier">n</span>,<span id="lstnumberx39.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx39.3" class="ltx_text ltx_lst_identifier">m</span><span id="lstnumberx39.4" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx39.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx39.6" class="ltx_text ltx_lst_identifier">map</span>(<span id="lstnumberx39.7" class="ltx_text ltx_lst_identifier">int</span>,<span id="lstnumberx39.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx39.9" class="ltx_text ltx_lst_identifier">input</span>().<span id="lstnumberx39.10" class="ltx_text ltx_lst_identifier">split</span>())
</div>
<div id="lstnumberx40" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9</span><span id="lstnumberx40.1" class="ltx_text ltx_lst_identifier">edge</span><span id="lstnumberx40.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx40.3" class="ltx_text ltx_lst_space"> </span>[[]<span id="lstnumberx40.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx40.5" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx40.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx40.7" class="ltx_text ltx_lst_identifier">_</span><span id="lstnumberx40.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx40.9" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx40.10" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx40.11" class="ltx_text ltx_lst_identifier">range</span>(<span id="lstnumberx40.12" class="ltx_text ltx_lst_identifier">n</span>)]
</div>
<div id="lstnumberx41" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10</span><span id="lstnumberx41.1" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx41.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx41.3" class="ltx_text ltx_lst_identifier">_</span><span id="lstnumberx41.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx41.5" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx41.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx41.7" class="ltx_text ltx_lst_identifier">range</span>(<span id="lstnumberx41.8" class="ltx_text ltx_lst_identifier">m</span>):
</div>
<div id="lstnumberx42" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11</span><span id="lstnumberx42.1" class="ltx_text ltx_lst_identifier">a</span>,<span id="lstnumberx42.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx42.3" class="ltx_text ltx_lst_identifier">b</span><span id="lstnumberx42.4" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx42.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx42.6" class="ltx_text ltx_lst_identifier">map</span>(<span id="lstnumberx42.7" class="ltx_text ltx_lst_identifier">int</span>,<span id="lstnumberx42.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx42.9" class="ltx_text ltx_lst_identifier">input</span>().<span id="lstnumberx42.10" class="ltx_text ltx_lst_identifier">split</span>())
</div>
<div id="lstnumberx43" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12</span><span id="lstnumberx43.1" class="ltx_text ltx_lst_identifier">edge</span>[<span id="lstnumberx43.2" class="ltx_text ltx_lst_identifier">a</span><span id="lstnumberx43.3" class="ltx_text ltx_lst_space"> </span>-<span id="lstnumberx43.4" class="ltx_text ltx_lst_space"> </span>1].<span id="lstnumberx43.5" class="ltx_text ltx_lst_identifier">append</span>(<span id="lstnumberx43.6" class="ltx_text ltx_lst_identifier">b</span><span id="lstnumberx43.7" class="ltx_text ltx_lst_space"> </span>-<span id="lstnumberx43.8" class="ltx_text ltx_lst_space"> </span>1)
</div>
<div id="lstnumberx44" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">13</span><span id="lstnumberx44.1" class="ltx_text ltx_lst_identifier">edge</span>[<span id="lstnumberx44.2" class="ltx_text ltx_lst_identifier">b</span><span id="lstnumberx44.3" class="ltx_text ltx_lst_space"> </span>-<span id="lstnumberx44.4" class="ltx_text ltx_lst_space"> </span>1].<span id="lstnumberx44.5" class="ltx_text ltx_lst_identifier">append</span>(<span id="lstnumberx44.6" class="ltx_text ltx_lst_identifier">a</span><span id="lstnumberx44.7" class="ltx_text ltx_lst_space"> </span>-<span id="lstnumberx44.8" class="ltx_text ltx_lst_space"> </span>1)
</div>
<div id="lstnumberx45" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">14</span>\<span id="lstnumberx45.1" class="ltx_text ltx_lst_identifier">pardef</span><span id="lstnumberx45.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx45.3" class="ltx_text ltx_lst_identifier">dfs</span>(<span id="lstnumberx45.4" class="ltx_text ltx_lst_identifier">u</span>):
</div>
<div id="lstnumberx46" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">15</span><span id="lstnumberx46.1" class="ltx_text ltx_lst_identifier">stack</span><span id="lstnumberx46.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx46.3" class="ltx_text ltx_lst_space"> </span>[<span id="lstnumberx46.4" class="ltx_text ltx_lst_identifier">u</span>]
</div>
<div id="lstnumberx47" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">16</span><span id="lstnumberx47.1" class="ltx_text ltx_lst_identifier">seen</span><span id="lstnumberx47.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx47.3" class="ltx_text ltx_lst_space"> </span>[<span id="lstnumberx47.4" class="ltx_text ltx_lst_identifier">False</span>]<span id="lstnumberx47.5" class="ltx_text ltx_lst_space"> </span>*<span id="lstnumberx47.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx47.7" class="ltx_text ltx_lst_identifier">n</span>
</div>
<div id="lstnumberx48" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">17</span><span id="lstnumberx48.1" class="ltx_text ltx_lst_identifier">while</span><span id="lstnumberx48.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx48.3" class="ltx_text ltx_lst_identifier">stack</span>:
</div>
<div id="lstnumberx49" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">18</span><span id="lstnumberx49.1" class="ltx_text ltx_lst_identifier">u</span><span id="lstnumberx49.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx49.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx49.4" class="ltx_text ltx_lst_identifier">stack</span>.<span id="lstnumberx49.5" class="ltx_text ltx_lst_identifier">pop</span>()
</div>
<div id="lstnumberx50" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">19</span><span id="lstnumberx50.1" class="ltx_text ltx_lst_identifier">if</span><span id="lstnumberx50.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx50.3" class="ltx_text ltx_lst_identifier">seen</span>[<span id="lstnumberx50.4" class="ltx_text ltx_lst_identifier">u</span>]:
</div>
<div id="lstnumberx51" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">20</span><span id="lstnumberx51.1" class="ltx_text ltx_lst_identifier">continue</span>
</div>
<div id="lstnumberx52" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">21</span><span id="lstnumberx52.1" class="ltx_text ltx_lst_identifier">seen</span>[<span id="lstnumberx52.2" class="ltx_text ltx_lst_identifier">u</span>]<span id="lstnumberx52.3" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx52.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx52.5" class="ltx_text ltx_lst_identifier">True</span>
</div>
<div id="lstnumberx53" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">22</span><span id="lstnumberx53.1" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx53.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx53.3" class="ltx_text ltx_lst_identifier">v</span><span id="lstnumberx53.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx53.5" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx53.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx53.7" class="ltx_text ltx_lst_identifier">edge</span>[<span id="lstnumberx53.8" class="ltx_text ltx_lst_identifier">u</span>]:
</div>
<div id="lstnumberx54" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">23</span><span id="lstnumberx54.1" class="ltx_text ltx_lst_identifier">if</span><span id="lstnumberx54.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx54.3" class="ltx_text ltx_lst_identifier">seen</span>[<span id="lstnumberx54.4" class="ltx_text ltx_lst_identifier">v</span>]:
</div>
<div id="lstnumberx55" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">24</span><span id="lstnumberx55.1" class="ltx_text ltx_lst_identifier">continue</span>
</div>
<div id="lstnumberx56" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">25</span><span id="lstnumberx56.1" class="ltx_text ltx_lst_identifier">seen</span>[<span id="lstnumberx56.2" class="ltx_text ltx_lst_identifier">v</span>]<span id="lstnumberx56.3" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx56.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx56.5" class="ltx_text ltx_lst_identifier">True</span>
</div>
<div id="lstnumberx57" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">26</span><span id="lstnumberx57.1" class="ltx_text ltx_lst_identifier">stack</span>.<span id="lstnumberx57.2" class="ltx_text ltx_lst_identifier">append</span>(<span id="lstnumberx57.3" class="ltx_text ltx_lst_identifier">v</span>)
</div>
<div id="lstnumberx58" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">27</span>\<span id="lstnumberx58.1" class="ltx_text ltx_lst_identifier">parres</span><span id="lstnumberx58.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx58.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx58.4" class="ltx_text ltx_lst_identifier">None</span>
</div>
<div id="lstnumberx59" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">28</span><span id="lstnumberx59.1" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx59.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx59.3" class="ltx_text ltx_lst_identifier">i</span><span id="lstnumberx59.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx59.5" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx59.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx59.7" class="ltx_text ltx_lst_identifier">range</span>(<span id="lstnumberx59.8" class="ltx_text ltx_lst_identifier">n</span>):
</div>
<div id="lstnumberx60" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">29</span><span id="lstnumberx60.1" class="ltx_text ltx_lst_identifier">if</span><span id="lstnumberx60.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx60.3" class="ltx_text ltx_lst_identifier">not</span><span id="lstnumberx60.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx60.5" class="ltx_text ltx_lst_identifier">seen</span>[<span id="lstnumberx60.6" class="ltx_text ltx_lst_identifier">i</span>]:
</div>
<div id="lstnumberx61" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">30</span><span id="lstnumberx61.1" class="ltx_text ltx_lst_identifier">seen</span>[<span id="lstnumberx61.2" class="ltx_text ltx_lst_identifier">i</span>]<span id="lstnumberx61.3" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx61.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx61.5" class="ltx_text ltx_lst_identifier">True</span>
</div>
<div id="lstnumberx62" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">31</span><span id="lstnumberx62.1" class="ltx_text ltx_lst_identifier">res</span><span id="lstnumberx62.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx62.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx62.4" class="ltx_text ltx_lst_identifier">dfs</span>(<span id="lstnumberx62.5" class="ltx_text ltx_lst_identifier">i</span>)
</div>
<div id="lstnumberx63" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">32</span><span id="lstnumberx63.1" class="ltx_text ltx_lst_identifier">if</span><span id="lstnumberx63.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx63.3" class="ltx_text ltx_lst_identifier">not</span><span id="lstnumberx63.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx63.5" class="ltx_text ltx_lst_identifier">res</span>:
</div>
<div id="lstnumberx64" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">33</span><span id="lstnumberx64.1" class="ltx_text ltx_lst_identifier">print</span>(-1)
</div>
<div id="lstnumberx65" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">34</span><span id="lstnumberx65.1" class="ltx_text ltx_lst_identifier">else</span>:
</div>
<div id="lstnumberx66" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">35</span><span id="lstnumberx66.1" class="ltx_text ltx_lst_identifier">print</span>(<span id="lstnumberx66.2" class="ltx_text ltx_lst_identifier">res</span>)
</div>
<div id="lstnumberx67" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">36</span>\<span id="lstnumberx67.1" class="ltx_text ltx_lst_identifier">par</span>\<span id="lstnumberx67.2" class="ltx_text ltx_lst_identifier">parif</span><span id="lstnumberx67.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx67.4" class="ltx_text ltx_lst_identifier">__name__</span><span id="lstnumberx67.5" class="ltx_text ltx_lst_space"> </span>==<span id="lstnumberx67.6" class="ltx_text ltx_lst_space"> </span>’<span id="lstnumberx67.7" class="ltx_text ltx_lst_identifier">__main__</span>’:
</div>
<div id="lstnumberx68" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">37</span><span id="lstnumberx68.1" class="ltx_text ltx_lst_identifier">resolve</span>()
</div>
</div>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>An example generated by GPT-2 1.5B. The model can sometimes compose function calls.</figcaption>
</figure>
<figure id="A4.F10" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F10.1" class="ltx_p ltx_figure_panel"><span id="A4.F10.1.1" class="ltx_text ltx_font_bold">Problem</span> 
<br class="ltx_break">Given is a tree G with N vertices. The vertices are numbered 1 through N, and the i-th edge connects Vertex a_i and Vertex b_i. Consider painting the edges in G with some number of colors. We want to paint them so that, for each vertex, the colors of the edges incident to that vertex are all different. Among the colorings satisfying the condition above, construct one that uses the minimum number of colors.

<br class="ltx_break">
<br class="ltx_break"><span id="A4.F10.1.2" class="ltx_text ltx_font_bold">Model Output</span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A4.F10.2" class="ltx_listing ltx_figure_panel ltx_lst_numbers_left ltx_lstlisting ltx_listing" style="background-color:#FFFFFF;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,CmltcG9ydCBzeXMKZnJvbSBjb2xsZWN0aW9ucyBpbXBvcnQgZGVxdWUsIGRlZmF1bHRkaWN0LCBDb3VudGVyCmZyb20gaXRlcnRvb2xzIGltcG9ydCBhY2N1bXVsYXRlLCBwcm9kdWN0LCBwZXJtdXRhdGlvbnMsIGNvbWJpbmF0aW9ucwpmcm9tIG9wZXJhdG9yIGltcG9ydCBpdGVtZ2V0dGVyCmZyb20gYmlzZWN0IGltcG9ydCBiaXNlY3RfbGVmdCwgYmlzZWN0X3JpZ2h0CmZyb20gaGVhcHEgaW1wb3J0IGhlYXBwb3AsIGhlYXBwdXNoCmZyb20gbWF0aCBpbXBvcnQgY2VpbCwgZmxvb3IsIHNxcnQsIGdjZCwgaW5mCmZyb20gY29weSBpbXBvcnQgZGVlcGNvcHkKaW1wb3J0IG51bXB5IGFzIG5wCmltcG9ydCBzY2lweSBhcyBzcApccGFySU5GID0gaW5mCk1PRCA9IDEwMDAwMDAwMDcKXHBhcm4gPSBpbnQoaW5wdXQoKSkKYWIgPSBbbGlzdChtYXAoaW50LCBpbnB1dCgpLnNwbGl0KCkpKSBmb3IgaSBpbiByYW5nZShuIC0gMSldClxwYXJncmFwaCA9IFtbXSBmb3IgaSBpbiByYW5nZShuICsgMSldCmZvciBhLCBiIGluIGFiOgpncmFwaFthXS5hcHBlbmQoYikKZ3JhcGhbYl0uYXBwZW5kKGEpClxwYXJ2aXNpdGVkID0gW0ZhbHNlXSAqIChuICsgMSkKY29sb3IgPSBbMF0gKiAobiArIDEpCmNvbG9yWzBdID0gMQpzdGFjayA9IFsxXQp3aGlsZSBzdGFjazoKeCA9IHN0YWNrLnBvcCgpCmZvciB5IGluIGdyYXBoW3hdOgppZiB2aXNpdGVkW3ldOgpjb250aW51ZQp2aXNpdGVkW3ldID0gVHJ1ZQpjb2xvclt5XSA9IGNvbG9yW3hdICsgMQpzdGFjay5hcHBlbmQoeSkKXHBhcmFucyA9IG1pbihjb2xvcikKcHJpbnQoKmFucywgc2VwPSJcbiIpCg==" download="">⬇</a></div>
<div id="lstnumberx69" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>
</div>
<div id="lstnumberx70" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="lstnumberx70.1" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx70.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx70.3" class="ltx_text ltx_lst_identifier">sys</span>
</div>
<div id="lstnumberx71" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span><span id="lstnumberx71.1" class="ltx_text ltx_lst_identifier">from</span><span id="lstnumberx71.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx71.3" class="ltx_text ltx_lst_identifier">collections</span><span id="lstnumberx71.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx71.5" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx71.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx71.7" class="ltx_text ltx_lst_identifier">deque</span>,<span id="lstnumberx71.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx71.9" class="ltx_text ltx_lst_identifier">defaultdict</span>,<span id="lstnumberx71.10" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx71.11" class="ltx_text ltx_lst_identifier">Counter</span>
</div>
<div id="lstnumberx72" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span><span id="lstnumberx72.1" class="ltx_text ltx_lst_identifier">from</span><span id="lstnumberx72.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx72.3" class="ltx_text ltx_lst_identifier">itertools</span><span id="lstnumberx72.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx72.5" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx72.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx72.7" class="ltx_text ltx_lst_identifier">accumulate</span>,<span id="lstnumberx72.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx72.9" class="ltx_text ltx_lst_identifier">product</span>,<span id="lstnumberx72.10" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx72.11" class="ltx_text ltx_lst_identifier">permutations</span>,<span id="lstnumberx72.12" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx72.13" class="ltx_text ltx_lst_identifier">combinations</span>
</div>
<div id="lstnumberx73" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span><span id="lstnumberx73.1" class="ltx_text ltx_lst_identifier">from</span><span id="lstnumberx73.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx73.3" class="ltx_text ltx_lst_identifier">operator</span><span id="lstnumberx73.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx73.5" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx73.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx73.7" class="ltx_text ltx_lst_identifier">itemgetter</span>
</div>
<div id="lstnumberx74" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6</span><span id="lstnumberx74.1" class="ltx_text ltx_lst_identifier">from</span><span id="lstnumberx74.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx74.3" class="ltx_text ltx_lst_identifier">bisect</span><span id="lstnumberx74.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx74.5" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx74.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx74.7" class="ltx_text ltx_lst_identifier">bisect_left</span>,<span id="lstnumberx74.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx74.9" class="ltx_text ltx_lst_identifier">bisect_right</span>
</div>
<div id="lstnumberx75" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7</span><span id="lstnumberx75.1" class="ltx_text ltx_lst_identifier">from</span><span id="lstnumberx75.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx75.3" class="ltx_text ltx_lst_identifier">heapq</span><span id="lstnumberx75.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx75.5" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx75.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx75.7" class="ltx_text ltx_lst_identifier">heappop</span>,<span id="lstnumberx75.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx75.9" class="ltx_text ltx_lst_identifier">heappush</span>
</div>
<div id="lstnumberx76" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8</span><span id="lstnumberx76.1" class="ltx_text ltx_lst_identifier">from</span><span id="lstnumberx76.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx76.3" class="ltx_text ltx_lst_identifier">math</span><span id="lstnumberx76.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx76.5" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx76.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx76.7" class="ltx_text ltx_lst_identifier">ceil</span>,<span id="lstnumberx76.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx76.9" class="ltx_text ltx_lst_identifier">floor</span>,<span id="lstnumberx76.10" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx76.11" class="ltx_text ltx_lst_identifier">sqrt</span>,<span id="lstnumberx76.12" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx76.13" class="ltx_text ltx_lst_identifier">gcd</span>,<span id="lstnumberx76.14" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx76.15" class="ltx_text ltx_lst_identifier">inf</span>
</div>
<div id="lstnumberx77" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9</span><span id="lstnumberx77.1" class="ltx_text ltx_lst_identifier">from</span><span id="lstnumberx77.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx77.3" class="ltx_text ltx_lst_identifier">copy</span><span id="lstnumberx77.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx77.5" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx77.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx77.7" class="ltx_text ltx_lst_identifier">deepcopy</span>
</div>
<div id="lstnumberx78" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10</span><span id="lstnumberx78.1" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx78.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx78.3" class="ltx_text ltx_lst_identifier">numpy</span><span id="lstnumberx78.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx78.5" class="ltx_text ltx_lst_identifier">as</span><span id="lstnumberx78.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx78.7" class="ltx_text ltx_lst_identifier">np</span>
</div>
<div id="lstnumberx79" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11</span><span id="lstnumberx79.1" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx79.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx79.3" class="ltx_text ltx_lst_identifier">scipy</span><span id="lstnumberx79.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx79.5" class="ltx_text ltx_lst_identifier">as</span><span id="lstnumberx79.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx79.7" class="ltx_text ltx_lst_identifier">sp</span>
</div>
<div id="lstnumberx80" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12</span>\<span id="lstnumberx80.1" class="ltx_text ltx_lst_identifier">parINF</span><span id="lstnumberx80.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx80.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx80.4" class="ltx_text ltx_lst_identifier">inf</span>
</div>
<div id="lstnumberx81" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">13</span><span id="lstnumberx81.1" class="ltx_text ltx_lst_identifier">MOD</span><span id="lstnumberx81.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx81.3" class="ltx_text ltx_lst_space"> </span>1000000007
</div>
<div id="lstnumberx82" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">14</span>\<span id="lstnumberx82.1" class="ltx_text ltx_lst_identifier">parn</span><span id="lstnumberx82.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx82.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx82.4" class="ltx_text ltx_lst_identifier">int</span>(<span id="lstnumberx82.5" class="ltx_text ltx_lst_identifier">input</span>())
</div>
<div id="lstnumberx83" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">15</span><span id="lstnumberx83.1" class="ltx_text ltx_lst_identifier">ab</span><span id="lstnumberx83.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx83.3" class="ltx_text ltx_lst_space"> </span>[<span id="lstnumberx83.4" class="ltx_text ltx_lst_identifier">list</span>(<span id="lstnumberx83.5" class="ltx_text ltx_lst_identifier">map</span>(<span id="lstnumberx83.6" class="ltx_text ltx_lst_identifier">int</span>,<span id="lstnumberx83.7" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx83.8" class="ltx_text ltx_lst_identifier">input</span>().<span id="lstnumberx83.9" class="ltx_text ltx_lst_identifier">split</span>()))<span id="lstnumberx83.10" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx83.11" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx83.12" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx83.13" class="ltx_text ltx_lst_identifier">i</span><span id="lstnumberx83.14" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx83.15" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx83.16" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx83.17" class="ltx_text ltx_lst_identifier">range</span>(<span id="lstnumberx83.18" class="ltx_text ltx_lst_identifier">n</span><span id="lstnumberx83.19" class="ltx_text ltx_lst_space"> </span>-<span id="lstnumberx83.20" class="ltx_text ltx_lst_space"> </span>1)]
</div>
<div id="lstnumberx84" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">16</span>\<span id="lstnumberx84.1" class="ltx_text ltx_lst_identifier">pargraph</span><span id="lstnumberx84.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx84.3" class="ltx_text ltx_lst_space"> </span>[[]<span id="lstnumberx84.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx84.5" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx84.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx84.7" class="ltx_text ltx_lst_identifier">i</span><span id="lstnumberx84.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx84.9" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx84.10" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx84.11" class="ltx_text ltx_lst_identifier">range</span>(<span id="lstnumberx84.12" class="ltx_text ltx_lst_identifier">n</span><span id="lstnumberx84.13" class="ltx_text ltx_lst_space"> </span>+<span id="lstnumberx84.14" class="ltx_text ltx_lst_space"> </span>1)]
</div>
<div id="lstnumberx85" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">17</span><span id="lstnumberx85.1" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx85.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx85.3" class="ltx_text ltx_lst_identifier">a</span>,<span id="lstnumberx85.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx85.5" class="ltx_text ltx_lst_identifier">b</span><span id="lstnumberx85.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx85.7" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx85.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx85.9" class="ltx_text ltx_lst_identifier">ab</span>:
</div>
<div id="lstnumberx86" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">18</span><span id="lstnumberx86.1" class="ltx_text ltx_lst_identifier">graph</span>[<span id="lstnumberx86.2" class="ltx_text ltx_lst_identifier">a</span>].<span id="lstnumberx86.3" class="ltx_text ltx_lst_identifier">append</span>(<span id="lstnumberx86.4" class="ltx_text ltx_lst_identifier">b</span>)
</div>
<div id="lstnumberx87" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">19</span><span id="lstnumberx87.1" class="ltx_text ltx_lst_identifier">graph</span>[<span id="lstnumberx87.2" class="ltx_text ltx_lst_identifier">b</span>].<span id="lstnumberx87.3" class="ltx_text ltx_lst_identifier">append</span>(<span id="lstnumberx87.4" class="ltx_text ltx_lst_identifier">a</span>)
</div>
<div id="lstnumberx88" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">20</span>\<span id="lstnumberx88.1" class="ltx_text ltx_lst_identifier">parvisited</span><span id="lstnumberx88.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx88.3" class="ltx_text ltx_lst_space"> </span>[<span id="lstnumberx88.4" class="ltx_text ltx_lst_identifier">False</span>]<span id="lstnumberx88.5" class="ltx_text ltx_lst_space"> </span>*<span id="lstnumberx88.6" class="ltx_text ltx_lst_space"> </span>(<span id="lstnumberx88.7" class="ltx_text ltx_lst_identifier">n</span><span id="lstnumberx88.8" class="ltx_text ltx_lst_space"> </span>+<span id="lstnumberx88.9" class="ltx_text ltx_lst_space"> </span>1)
</div>
<div id="lstnumberx89" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">21</span><span id="lstnumberx89.1" class="ltx_text ltx_lst_identifier">color</span><span id="lstnumberx89.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx89.3" class="ltx_text ltx_lst_space"> </span>[0]<span id="lstnumberx89.4" class="ltx_text ltx_lst_space"> </span>*<span id="lstnumberx89.5" class="ltx_text ltx_lst_space"> </span>(<span id="lstnumberx89.6" class="ltx_text ltx_lst_identifier">n</span><span id="lstnumberx89.7" class="ltx_text ltx_lst_space"> </span>+<span id="lstnumberx89.8" class="ltx_text ltx_lst_space"> </span>1)
</div>
<div id="lstnumberx90" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">22</span><span id="lstnumberx90.1" class="ltx_text ltx_lst_identifier">color</span>[0]<span id="lstnumberx90.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx90.3" class="ltx_text ltx_lst_space"> </span>1
</div>
<div id="lstnumberx91" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">23</span><span id="lstnumberx91.1" class="ltx_text ltx_lst_identifier">stack</span><span id="lstnumberx91.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx91.3" class="ltx_text ltx_lst_space"> </span>[1]
</div>
<div id="lstnumberx92" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">24</span><span id="lstnumberx92.1" class="ltx_text ltx_lst_identifier">while</span><span id="lstnumberx92.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx92.3" class="ltx_text ltx_lst_identifier">stack</span>:
</div>
<div id="lstnumberx93" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">25</span><span id="lstnumberx93.1" class="ltx_text ltx_lst_identifier">x</span><span id="lstnumberx93.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx93.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx93.4" class="ltx_text ltx_lst_identifier">stack</span>.<span id="lstnumberx93.5" class="ltx_text ltx_lst_identifier">pop</span>()
</div>
<div id="lstnumberx94" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">26</span><span id="lstnumberx94.1" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx94.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx94.3" class="ltx_text ltx_lst_identifier">y</span><span id="lstnumberx94.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx94.5" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx94.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx94.7" class="ltx_text ltx_lst_identifier">graph</span>[<span id="lstnumberx94.8" class="ltx_text ltx_lst_identifier">x</span>]:
</div>
<div id="lstnumberx95" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">27</span><span id="lstnumberx95.1" class="ltx_text ltx_lst_identifier">if</span><span id="lstnumberx95.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx95.3" class="ltx_text ltx_lst_identifier">visited</span>[<span id="lstnumberx95.4" class="ltx_text ltx_lst_identifier">y</span>]:
</div>
<div id="lstnumberx96" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">28</span><span id="lstnumberx96.1" class="ltx_text ltx_lst_identifier">continue</span>
</div>
<div id="lstnumberx97" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">29</span><span id="lstnumberx97.1" class="ltx_text ltx_lst_identifier">visited</span>[<span id="lstnumberx97.2" class="ltx_text ltx_lst_identifier">y</span>]<span id="lstnumberx97.3" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx97.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx97.5" class="ltx_text ltx_lst_identifier">True</span>
</div>
<div id="lstnumberx98" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">30</span><span id="lstnumberx98.1" class="ltx_text ltx_lst_identifier">color</span>[<span id="lstnumberx98.2" class="ltx_text ltx_lst_identifier">y</span>]<span id="lstnumberx98.3" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx98.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx98.5" class="ltx_text ltx_lst_identifier">color</span>[<span id="lstnumberx98.6" class="ltx_text ltx_lst_identifier">x</span>]<span id="lstnumberx98.7" class="ltx_text ltx_lst_space"> </span>+<span id="lstnumberx98.8" class="ltx_text ltx_lst_space"> </span>1
</div>
<div id="lstnumberx99" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">31</span><span id="lstnumberx99.1" class="ltx_text ltx_lst_identifier">stack</span>.<span id="lstnumberx99.2" class="ltx_text ltx_lst_identifier">append</span>(<span id="lstnumberx99.3" class="ltx_text ltx_lst_identifier">y</span>)
</div>
<div id="lstnumberx100" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">32</span>\<span id="lstnumberx100.1" class="ltx_text ltx_lst_identifier">parans</span><span id="lstnumberx100.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx100.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx100.4" class="ltx_text ltx_lst_identifier">min</span>(<span id="lstnumberx100.5" class="ltx_text ltx_lst_identifier">color</span>)
</div>
<div id="lstnumberx101" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">33</span><span id="lstnumberx101.1" class="ltx_text ltx_lst_identifier">print</span>(*<span id="lstnumberx101.2" class="ltx_text ltx_lst_identifier">ans</span>,<span id="lstnumberx101.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx101.4" class="ltx_text ltx_lst_identifier">sep</span>="\<span id="lstnumberx101.5" class="ltx_text ltx_lst_identifier">n</span>")
</div>
</div>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>An additional generated example. Here, the model output has many import statements.</figcaption>
</figure>
<figure id="A4.F11" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F11.1" class="ltx_p ltx_figure_panel"><span id="A4.F11.1.1" class="ltx_text ltx_font_bold">Problem</span> 
<br class="ltx_break">There is a special square room with mirrors on each of the four walls.  Except for the southwest corner, there are receptors on each of the remaining corners, numbered 0, 1, and 2. The square room has walls of length p, and a laser ray from the southwest corner first meets the east wall at a distance q from the 0th receptor. Return the number of the receptor that the ray meets first.  (It is guaranteed that the ray will meet a receptor eventually.)

<br class="ltx_break"></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F11.2" class="ltx_p ltx_figure_panel">Example 1:</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F11.3" class="ltx_p ltx_figure_panel">Input: p = 2, q = 1</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F11.4" class="ltx_p ltx_figure_panel">Output: 2</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F11.5" class="ltx_p ltx_figure_panel">Explanation: The ray meets receptor 2 the first time it gets reflected back to the left wall.

<br class="ltx_break"></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F11.6" class="ltx_p ltx_figure_panel"><span id="A4.F11.6.1" class="ltx_text ltx_font_bold">Starter Code</span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A4.F11.7" class="ltx_listing ltx_figure_panel ltx_lst_numbers_left ltx_lstlisting ltx_listing" style="background-color:#FFFFFF;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,CmNsYXNzIFNvbHV0aW9uOgpkZWYgbWlycm9yUmVmbGVjdGlvbihzZWxmLCBwOiBpbnQsIHE6IGludCkgLT4gaW50Ogo=" download="">⬇</a></div>
<div id="lstnumberx102" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>
</div>
<div id="lstnumberx103" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="lstnumberx103.1" class="ltx_text ltx_lst_identifier">class</span><span id="lstnumberx103.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx103.3" class="ltx_text ltx_lst_identifier">Solution</span>:
</div>
<div id="lstnumberx104" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span><span id="lstnumberx104.1" class="ltx_text ltx_lst_identifier">def</span><span id="lstnumberx104.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx104.3" class="ltx_text ltx_lst_identifier">mirrorReflection</span>(<span id="lstnumberx104.4" class="ltx_text ltx_lst_identifier">self</span>,<span id="lstnumberx104.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx104.6" class="ltx_text ltx_lst_identifier">p</span>:<span id="lstnumberx104.7" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx104.8" class="ltx_text ltx_lst_identifier">int</span>,<span id="lstnumberx104.9" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx104.10" class="ltx_text ltx_lst_identifier">q</span>:<span id="lstnumberx104.11" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx104.12" class="ltx_text ltx_lst_identifier">int</span>)<span id="lstnumberx104.13" class="ltx_text ltx_lst_space"> </span>-&gt;<span id="lstnumberx104.14" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx104.15" class="ltx_text ltx_lst_identifier">int</span>:
</div>
</div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.F11.8" class="ltx_p ltx_figure_panel"><span id="A4.F11.8.1" class="ltx_text ltx_font_bold">Model Output</span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A4.F11.9" class="ltx_listing ltx_figure_panel ltx_lst_numbers_left ltx_lstlisting ltx_listing" style="background-color:#FFFFFF;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,CmNsYXNzIFNvbHV0aW9uOgpkZWYgbWlycm9yUmVmbGVjdGlvbihzZWxmLCBwOiBpbnQsIHE6IGludCkgLT4gaW50OgpzZWxmLnAgPSBwCnNlbGYucSA9IHEKc2VsZi5taXJyb3JzID0gWygwLCAxKSwgKDAsIC0xKSwgKDEsIDApLCAoLTEsIDApXQpyZXR1cm4gc2VsZi5taXJyb3JzW3NlbGYucCAtIDFdW3NlbGYucSAtIDFdCg==" download="">⬇</a></div>
<div id="lstnumberx105" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>
</div>
<div id="lstnumberx106" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="lstnumberx106.1" class="ltx_text ltx_lst_identifier">class</span><span id="lstnumberx106.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx106.3" class="ltx_text ltx_lst_identifier">Solution</span>:
</div>
<div id="lstnumberx107" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span><span id="lstnumberx107.1" class="ltx_text ltx_lst_identifier">def</span><span id="lstnumberx107.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx107.3" class="ltx_text ltx_lst_identifier">mirrorReflection</span>(<span id="lstnumberx107.4" class="ltx_text ltx_lst_identifier">self</span>,<span id="lstnumberx107.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx107.6" class="ltx_text ltx_lst_identifier">p</span>:<span id="lstnumberx107.7" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx107.8" class="ltx_text ltx_lst_identifier">int</span>,<span id="lstnumberx107.9" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx107.10" class="ltx_text ltx_lst_identifier">q</span>:<span id="lstnumberx107.11" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx107.12" class="ltx_text ltx_lst_identifier">int</span>)<span id="lstnumberx107.13" class="ltx_text ltx_lst_space"> </span>-&gt;<span id="lstnumberx107.14" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx107.15" class="ltx_text ltx_lst_identifier">int</span>:
</div>
<div id="lstnumberx108" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span><span id="lstnumberx108.1" class="ltx_text ltx_lst_identifier">self</span>.<span id="lstnumberx108.2" class="ltx_text ltx_lst_identifier">p</span><span id="lstnumberx108.3" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx108.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx108.5" class="ltx_text ltx_lst_identifier">p</span>
</div>
<div id="lstnumberx109" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span><span id="lstnumberx109.1" class="ltx_text ltx_lst_identifier">self</span>.<span id="lstnumberx109.2" class="ltx_text ltx_lst_identifier">q</span><span id="lstnumberx109.3" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx109.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx109.5" class="ltx_text ltx_lst_identifier">q</span>
</div>
<div id="lstnumberx110" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6</span><span id="lstnumberx110.1" class="ltx_text ltx_lst_identifier">self</span>.<span id="lstnumberx110.2" class="ltx_text ltx_lst_identifier">mirrors</span><span id="lstnumberx110.3" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx110.4" class="ltx_text ltx_lst_space"> </span>[(0,<span id="lstnumberx110.5" class="ltx_text ltx_lst_space"> </span>1),<span id="lstnumberx110.6" class="ltx_text ltx_lst_space"> </span>(0,<span id="lstnumberx110.7" class="ltx_text ltx_lst_space"> </span>-1),<span id="lstnumberx110.8" class="ltx_text ltx_lst_space"> </span>(1,<span id="lstnumberx110.9" class="ltx_text ltx_lst_space"> </span>0),<span id="lstnumberx110.10" class="ltx_text ltx_lst_space"> </span>(-1,<span id="lstnumberx110.11" class="ltx_text ltx_lst_space"> </span>0)]
</div>
<div id="lstnumberx111" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7</span><span id="lstnumberx111.1" class="ltx_text ltx_lst_identifier">return</span><span id="lstnumberx111.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx111.3" class="ltx_text ltx_lst_identifier">self</span>.<span id="lstnumberx111.4" class="ltx_text ltx_lst_identifier">mirrors</span>[<span id="lstnumberx111.5" class="ltx_text ltx_lst_identifier">self</span>.<span id="lstnumberx111.6" class="ltx_text ltx_lst_identifier">p</span><span id="lstnumberx111.7" class="ltx_text ltx_lst_space"> </span>-<span id="lstnumberx111.8" class="ltx_text ltx_lst_space"> </span>1][<span id="lstnumberx111.9" class="ltx_text ltx_lst_identifier">self</span>.<span id="lstnumberx111.10" class="ltx_text ltx_lst_identifier">q</span><span id="lstnumberx111.11" class="ltx_text ltx_lst_space"> </span>-<span id="lstnumberx111.12" class="ltx_text ltx_lst_space"> </span>1]
</div>
</div>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>An example using the call-based format with some starter code.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2105.09937" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2105.09938" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2105.09938">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2105.09938" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2105.09939" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Thu Mar  7 00:35:40 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2105.09938
While programming is one of the most broadly applicable skills in modern society, it is unclear how well state-of-the-art machine learning models can write code. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to assess code generation performance in an accurate and rigorous manner. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we evaluate models by checking their generated code on test cases. Our benchmark includesproblems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximatelyof the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an objective measure for tracking advancements.
abstract
Measuring Coding Challenge Competence With APPS
Abstract
2105.09938
1IntroductionComputer programming can be found in nearly all parts of society. Spanning entertainment, healthcare, education, and more, programming is an extraordinarily general tool with applications that are vast in scope. As computers are becoming more ubiquitous in modern life, rising demand for high-quality code draws an ever-greater number of aspiring programmers to the profession. After years of study to become proficient coders, human experts are are able to convert abstract specifications of diverse cognitive tasks into concrete programs.In the past few years, large-scale language models have shown promise in generalizing to various cognitive tasks, including linguistic inference, commonsense reasoning, logical deduction, mathematics, and general understanding of multiple domains of human knowledge. However, whether large-scale language models can reliably write code remains an open question.Motivated by the potential of language models and the need for thorough code generation evaluation, we introduce APPS, a benchmark for code generation from natural language specifications. Unlike prior work on code generation with Transformer language models, which mostly focuses on code translationand pseudocode-to-code, we evaluate models on their ability to take specifications given in natural language and write code that meets these specifications. This setting mirrors how human coders are evaluated and is a more realistic and informative setting in which to benchmark models.Figure 1:An example problem from APPS (left)  along with possible generated code (middle)  and two example test cases we use to evaluate the generated code (right) . Our evaluation framework has test cases andcode generation problems of varying difficulty levels.APPS provides a precise and comprehensive view of code generation. APPS evaluates models not only on their ability to code syntactically correct programs, but also on their ability to understand task descriptions and devise algorithms to solve these tasks. It containsprogramming problems at various levels of difficulty, covering simple introductory problems, interview-level problems, and coding competition challenges. If a model were to perform well on APPS, this would indicate an ability to flexibly use data structures and programming techniques, as well as an ability to correctly interpret diverse task specifications, follow instructions, and understand human intent.For most text generation tasks, high-quality evaluation requires human feedback, which can be time-consuming or carry pecuniary costs. As a result, automatic metrics such as BLEUare often used to compare methods, but these metrics do not necessarily track program correctness. Since the objective for code generation is to produce correct programs, we assess programs not with BLEU but with test cases and error catching. Evaluating code generation on APPS is facilitated by a large bank of overtest cases. The test cases are specifically chosen to probe correct functionality across the input space. By using test cases, we provide a gold-standard metric for code generation quality.In our experiments, we find that models are now starting to exhibit nonzero accuracy and solve some coding problems. Additionally, as models improve, we observe that syntax errors are exponentially decreasing. We also find further evidence that BLEU is a problematic metric for code generation, sometimes being anticorrelated with gold-standard accuracy. We find that accuracy decreases with difficulty level and improves through fine-tuning and model size increases. The strongest model that we evaluate on introductory problems passes almostof test cases given five attempts. These results position code generation as a challenging but now tractable testbed for large-scale language models.Writing code to meet specifications in natural language is an economically valuable task with widespread social implications should it be solved, as it could eventually facilitate malicious code generation and one day result in job automation. As large-scale language models have the potential to make significant progress on code generation, it is essential that we begin to track advancements on this task. Our new benchmark facilitates measuring performance in an accurate and rigorous manner. Using APPS, we find that programming is very difficult for modern language models, though performance is improving. Thus, the APPS benchmark can provide foresight about the performance of future large-scale language models at the critical task of program synthesis from natural language. The dataset is available athttps://github.com/hendrycks/apps.PY150CONCODESPoCAPPSProgramming LanguagePythonJavaC++PythonTest Cases✕✕Number of ProgramsN/A104,00018,356232,421Lines per Program (Avg.) 126.314.718.0Number of Exercises3,000104,00067710,000Text InputPythonDocstringsPseudocodeProblem DescriptionsTable 1:A comparison of the APPS dataset to existing datasets for converting between text and code. APPS has over an order of magnitude more ground-truth solutions than these datasets, test cases, and natural language problem descriptions.
introduction
Measuring Coding Challenge Competence With APPS
1Introduction
2105.09938
2Related WorkProgram Synthesis.Program synthesis is the task of generating a computer program that satisfies given specifications.
Deductive program synthesis uses formal logic specifications to define a search problem. Complex optimization techniques are used to generate programs satisfying these specifications. Because specifications must be converted into a formal language, these approaches can be rigid. Inductive synthesis from example input-output behavior can provide an alternative to formal specification, but it is often hard to full specify behavior with examples, as any machine learning practitioner is well-aware.An alternative to formal or inductive specification is to specify program behavior in natural language, which prior work has considered in constrained settings.andgenerate short programs using ad-hoc programming languages to solve specifications such as “Any 2 letters followed by any combination of 6 whole numbers.”introduce the Spider dataset for converting natural language queries into short SQL database commands. In contrast, we consider long natural language specifications and general-purpose programming languages.Code Understanding Datasets.Language modeling is a compelling tool for code generation, and several works have achieved success generating code with language models in limited settings.use unsupervised machine translation techniques to translate functions across programming languages, attaining identical behavior after translation in many cases.introduce SPoC, a method for converting pseudocode to code utilizing seq2seq machine translation with an additional search step. To train SPoC, they collect line-by-line descriptions of C++ programs using Amazon Mechanical Turk. Recently,introduce the CodeXGLUE benchmark which aggregates various previous benchmarks and use CodeBLEUand CONCODE.investigate generating Java code from docstrings and evaluate performance with BLEU. The docstrings are often incomplete specifications of what should be coded and onlywords long on average, e.g. “Convert mixed case to underscores.” By comparison, problem specifications in our new APPS benchmark are self-contained and have a much larger average length ofwords. Unlike, APPS contains test cases for every exercise, enabling a high-quality evaluation of code correctness. Further comparisons are in the Appendix.Evaluating Large-Scale Language Models.Modern large-scale language models have demonstrated impressive capabilities across a variety of text-based tasks. On the SuperGLUE benchmark, some models now exceed human performance. On many commonsense reasoning benchmarks, performance is rising quickly. Even when language models are evaluated across diverse technical areas such as law and medicine, performance is surprisingly high and poised to improve as models are scaled up further. With rapid improvements across numerous datasets, finding resilient benchmarks on which models significantly underperform humans is challenging. APPS represents an attempt to fill this gap and cleanly separate model performance from that of expert humans.
body
Measuring Coding Challenge Competence With APPS
2Related Work
2105.09938
3The APPS DatasetThe APPS dataset consists of problems collected from different open-access coding websites such as Codeforces, Kattis, and more. The APPS benchmark attempts to mirror how humans programmers are evaluated by posing coding problems in unrestricted natural language and using test cases to evaluate solution correctness. The problems range in difficulty from introductory to collegiate competition level and measure coding and problem-solving ability.The Automated Programming Progress Standard, abbreviated APPS, consists ofcoding problems in total, withtest cases for checking solutions andground-truth solutions written by humans. Problems can be complicated, as the average length of a problem iswords.
The data are split evenly into training and test sets, withproblems each. In the test set, every problem has multiple test cases, and the average number of test cases is. Each test case is specifically designed for the corresponding problem, enabling us to rigorously evaluate program functionality.Dataset Construction.To create the APPS dataset, we manually curate problems from open-access sites where programmers share problems with each other, including Codewars, AtCoder, Kattis, and Codeforces. Problems are posed as natural language specifications of what should be coded, and they come in various formats. To improve quality and consistency, we wrote custom HTML parsers for each source of problems, which allows us to properly format LaTeX expressions, lists, and sections in the question text. Where necessary, we convert equation images to LaTeX using the MathPix API, and we remove problems that rely on image figures. We also perform deduplication using tf-idf features with SVD dimensionality reduction and cosine similarity. Several graduate and undergraduate student authors polished and refined this dataset over the course of six months, ensuring a high-quality set of problems.Executing and evaluating arbitrary Python code is challenging. On the websites we source data from, human solutions are allowed to run arbitrary code, including import statements for common modules and libraries. To handle this, each website implements a custom judging system for solutions. We design a testing framework with this in mind, which merges the judging functionality of several websites. We also standardize the format of test cases. The end result is that solutions are allowed to execute arbitrary Python code, and the results are compared against test cases for a given problem.Dataset Difficulty.Each of our problem sources uses a separate scale for measuring difficulty. We place problems from these different sources into three categories. For example, problems from Kattis with difficulty less thanare categorized as “introductory,” problems with difficulty betweenandas “interview,” and problems with difficulty greater thanas “competition.”1.Introductory Level. These are problems that most programmers with 1-2 years of experience can answer without requiring complicated algorithms. Examples of such problems include counting the number of vowels in a string, or returning the running sum of a list of integers. There areproblems classified as introductory level andin the test set.2.Interview Level. These are problems that are more algorithmic and difficult in nature and would be at the level of questions asked in programming technical interviews. Examples of such problems might include those involving data structures such as trees or graphs, or problems that requiring nontrivial algorithms. There areproblems classified as interview level andin the test set.3.Competition Level. These are problems are the most challenging and are at the level of the most advanced high school and collegiate programming competitions, including USACO, IOI, and ACM. There arecompetition level problems andin the test set.Problem Formats.To accommodate a broad range of problem sources, problems in APPS come in two formats.•Call-Based Formatproblems generally provide initial starter code, usually in the form of a function header, and ask for the solution to be provided as the function’s return value.•Standard Input Formatproblems generally lack starter code. Instead, the model is only provided with the problem and must output its answers to the STDOUT stream, such as by using print statements.For the call-based format problems, we prompt models using the following inputs:{spverbatim}"\nQUESTION:\n" + q_str + "\n" + starter_code_str + "\n" + "\nUseCall-Based Format\n\nANSWER:\n"For the above prompt, the variableq_strrepresents the raw text of the problem statement. The variablestarter_code_strrepresents the starter code given in the problem definition, or the empty string if no starter code was provided. For the standard input format problems, we prompt the model with the input string as before, but we replace “Call-Based Format” with “Standard Input Format.” Note that if starter code is given, it is only part of the input. This means that to use the starter code, a model must learn to copy the starter code at the beginning of its outputted answer in order to get the question correct. We find that fine-tuned models are able to do this without difficulty.ProblemYou are given a stringof length, which only contains digits,,…,. A substringofis a string. A substringofis called even if the number represented by it is even. Find the number of even substrings of. Note, that even if some substrings are equal as strings, but have differentand, they are counted as different substrings. The first line contains an integer()  — the length of the string. The second line contains a stringof length. The stringconsists only of digits,,…,. Print the number of even substrings of.Model Output⬇12n=int(input() ) 3s=input() 4ans=05foriinrange(n) :6forjinrange(i,n) :7ifint(s[i:j+1]) ans+=18print(ans) Figure 2:Model-generated code from GPT-2 1.5B. 18/18 test cases pass for the problem. Note that models were given in problems exactly as they they are formatted here, which even meant parsingLaTeX. For brevity, we exclude formatting instructions from this depicted problem.ProblemYou are given two integersand. Calculate the number of pairs of arrayssuch that: the length of both arrays is equal to; each element of each array is an integer betweenand(inclusive) ;for any indexfromto; arrayis sorted in non-descending order; arrayis sorted in non-ascending order. As the result can be very large, you should print it modulo. Input: The only line contains two integersand(,) . Output: Print one integer – the number of arraysandsatisfying the conditions described above modulo.Model Output⬇12n,m=map(int,input() .split() ) 3mod=10**9+74\parfact=[1,1]5factinv=[1,1]6inv=[0,1]7\parforiinrange(2,n+1) :8fact.append((fact[-1]*i) inv.append((-inv[modfactinv.append((factinv[-1]*inv[-1]) \pardefcmb(n,r,mod) :9if(r<0) or(n<r) :10return011r=min(r,n-r) 12returnfact[n]*factinv[r]*factinv[n-r]\parprint((cmb(n,m,mod) *cmb(m-1,n-1,mod) ) Figure 3:An example from GPT-2 1.5B. Although the code generated passes 0 test cases, it looks plausible at first glance.Test Case Quality.In the APPS test split, the average number of test cases is, but some problems only have two test cases. These problems mainly come from Kattis and were chosen for the test split due to limited numbers of competition problems. A potential concern is that these problems could result in false positives if models happen to guess both test cases correctly. This is very unlikely in problems with large output spaces, but some problems have small output spaces, such as. Although the model must narrow down these two options from the space of all possible strings, we find that this is not difficult for current models to accomplish, and this may cause false positives in our evaluation.To quantify the impact of these problems, we took all 2- and 3-test-case problems that GPT-Neo 2.7B passed and manually checked for false positives. Out of 12 problems that the model got correct with 2 test cases, 8 were false positives. Out of 6 problems with 3 test cases, only 1 was a false positive. Since the test cases are from coding challenge websites with many users, we can assume that false negatives are absent. Thus, the false positive rate with 2 test cases isandwith 3 test cases. The upshot is that the potential for noise is significantly less than that of most naturally annotated datasets.
body
Measuring Coding Challenge Competence With APPS
3The APPS Dataset
2105.09938
4Experiments4.1Experimental SetupModels.We use GPT-2, GPT-3, and GPT-Neomodels. The GPT architecture is especially appropriate for text generation because it is autoregressive. However, GPT-2 was not pretrained on code, so we pretrain it on GitHub as described in the next paragraph. Anecdotal evidence indicates that GPT-3 can generate code. To determine the extent of its code generation ability, we use the ‘davinci’ (Instruct series)  model, the largest publicly available model speculated to have 175 billion parameters. Finally, GPT-Neo has an architecture similar to GPT-3, and it was pretrained on the Pilewhich includes GitHub. Unlike GPT-3, GPT-Neo’s weights are publicly available, hence we are able to fine-tune it with APPS.GPT-2 Pretraining.Since GPT-2 was trained on natural language and not code, we collected GitHub code to further pretrain GPT-2. GitHub repositories with fewer than one star were filtered out.
While Neo’s GitHub pretraining data didnotundergo an APPS data decontamination process, our GPT-2 models are trained on decontaminated data.
Specifically, all repositories matching certain keywords that would suggest overlap with common programming exercises were removed. We provide the list of keywords in the Supplementary Materials. We also discard any GitHub code that contains functions with the same signatures as functions in the starter code in many of our APPS problems. This leaves us with 30 GB of Python code. To improve the efficiency of pretraining, we process all Python code in the pretraining dataset by converting from spaces to tabs, which saves the character conversion when running model tokenizers.Fine-tuning.During fine-tuning with APPS, the objective is to predict the entire code solution, given both the English text problem statement and the problem format (call-based format or standard input format) . For problems with starter code, we exclude the starter code from the training loss.Across pretraining and fine-tuning, we use the AdamW optimizer, a batch size of, and a weight decay of. We fine-tune forepochs. We use DeepSpeed and its implementation of the ZeRO optimizer to reduce memory consumption while training large models. Unless otherwise specified, we use the default HuggingFace generation parameters, except that we use beam search with a beam size of. Models are fine-tuned on 8 A100 GPUs.4.2MetricsTo obtain a comprehensive evaluation of code generation ability, we use the large bank of test cases and ground-truth solutions provided with APPS. Test cases allow forautomaticevaluation, even though the the space of possible programs can be combinatorially large. Therefore, unlike many other text generation tasks, manual analysis is not necessary. We aggregate the generated code’s performance on test cases with two metrics, “test case average” and “strict accuracy.”Test Case Average.We compute the average fraction of test cases passed.
Concretely, let the number of problems in the test set be. For a given problem, let the code generated to solve problembe denoted, and set of test cases for problembe. Then the test case average isOftentimes, solutions can successfully pass a subset of the test cases but not cover every corner case. This allows for less stringent model evaluation, as strict accuracy may currently obscure model improvements.Test Case AverageStrict AccuracyModelIntroductoryInterviewCompetitiveAverageIntroductoryInterviewCompetitionAverageGPT-2 0.1B5.646.934.376.161.000.330.000.40GPT-2 1.5B7.409.115.057.961.300.700.000.68GPT-Neo 2.7B14.689.856.5410.153.900.570.001.12GPT-3 175B0.570.650.210.550.200.030.000.06Table 2:Average percentage of test cases passed and strict accuracy for each model and difficulty level. All values are percentages. Note ‘0.1B’ indicates the number of model parameters in billions. GPT-3 is afew-shotmodel and not fine-tuned, unlike the other models. GPT-Neo does best and attains approximately 4% strict accuracy on Introductory problems, and for these problems it passes approximately 15% of the test cases.Strict Accuracy.Eventually, generated solutions should pass all test cases including corner cases.
To compute the strict accuracy which requires programs pass every test case, we run the code generated by the model on every test case of every problem. Strict accuracy is then computed by taking the number of solutions passing every test case divided by the total number of exercises. Using the notation from before, we can write the strict accuracy asFuture research may only use strict accuracy when models become sufficiently capable.4.3Model Performance AnalysisQualitative Output Analysis.Models can sometimes generate correct or superficially plausible code.Figure2shows code generated by GPT-2 1.5B that passes all test cases. When models do not pass the test cases, sometimes their generated code still appears plausible at first glance. For example, inFigure3, we see that the 1.5B parameter model generates code that is related to the problem statement and makes a plausible attempt to solve it.Test Case Evaluation.We show the main results inTable2. We observe that models are able to generate code that passed some test cases, implying many generated programs are free of syntax errors and can successfully process inputs test cases to produce correct answers. Note that for Introductory questions, GPT-Neo passes approximatelyof the test cases. We visualize Test Case Average results inFigure5. This demonstrates models are showing marked improvements on code generation and now starting to have traction on code generation.Top-1Top-5Test Case AverageStrict AccuracyTable 3:GPT-Neo 2.7B performance on introductory problems using one generated program (Top-1)  and the best of five generated programs (Top-5) . Full results are in the Supplementary Materials.Performance can be further improved by sampling multiple solutions and selecting the best. Here, we perform beam search with beam widthand evaluate itsbeams, so that each model has five attempts to get a problem correct rather than one. With this setup, GPT-Neo’s strict accuracy on Introductory problem then exceeds, as shown inTable3. Our results in the Supplementary Materials show that the top-5 test case average GPT-2 0.1B is 10.75 while the top-1 test case average of GPT-2 1.5B is 7.96. This highlights that simply sampling multiple candidate solutions is a powerful way to markedly improve performance.Our results also provide us with information about the importance of model choice. Evidently existing few-shot GPT-3 models are not necessarily better at code generation than fine-tuned models that are smaller by two orders of magnitude. Additionally, performance improvement from GPT-2 1.5B to GPT-Neo 2.7B is larger than that from GPT-2 0.1B to GPT-2 1.5B. Potential causes of GPT-Neo’s better performance are that GPT-Neo is trained on more code from GitHub, it has more parameters, or its architecture hyperparameters were chosen better. Memorization explaining all performance is an implausible explanation as performance tracks problem difficulty; were models just memorizing, we would expect uniform performance across difficulties. Since models still have large room for improvement, solving the APPS benchmark without unreasonable amounts of computational resources may require architectural or algorithmic improvements.Syntax Errors.We now assess the frequency of syntax errors, errors that prevent the program from being interpreted including inconsistent spacing, unbalanced brackets, missing colons, and so on. Syntax errors are identified in our testing framework based on the heuristic of whether pyext is able to load the generated code as a Python module. For our purposes, this almost exclusively occurs for syntax errors. We visualize the prevalence of syntax errors inFigure5. While approximatelyof GPT-3’s generated solutions for introductory problems have syntax errors, GPT-Neo syntax error frequency is approximately. Note that recent work such ascreate a separate model to repair source code to fix compilation issues, but our results suggest that such efforts may be unnecessary in the future as syntax error frequency is sharply decreasing automatically.Figure 4:The average percentage of test cases passed increases with larger fine-tuned models.Figure 5:Syntax errors decrease exponentially with fine-tuning and increased model sizes. GPT-Neo 2.7B has very few syntax errors.BLEU.We find that assessing model performance with BLEU is a poor substitute for evaluating with test cases. To evaluate BLEU, we take the generated solution and compute its BLEU with each human-written solution for a given problem; we then record the highest BLEU score. Observe inFigure6that BLEU increases as problem sources become more difficult, even though models actually perform worse on harder problems. Moreover, worse models can have similar or higher BLEU scores. For example, GPT-2 0.1B has,, andas BLEU scores for introductory, interview, and competition problems, respectively. Meanwhile GPT-Neo 2.7B has,, andas its BLEU scores, respectively. Hence BLEU wrongly suggests GPT-Neo is a worse model.Figure 6:BLEU scores for GPT-Neo 2.7B increase with difficulty level and are anticorrelated with a gold-standard accuracy metric.Evaluating GPT-3.We evaluate GPT-3 175B on APPS in a few-shot setting. A separate prompt is used for standard input and call-based questions, and each prompt includes instruction text along with two example questions and solutions from the corresponding question type. We find that GPT-3 only solvesproblems out of: two introductory problems and one interview problem. The two introductory problems are simple interpretation tasks, such as implementing a specified algebraic expression. The interview problem requires higher-level thinking that suggests nontrivial reasoning. However, it is possible that GPT-3 memorized the solution during pretraining, or that it took a lucky guess based on heuristics in the question. One potential factor in GPT-3’s poor performance is that it handles syntax poorly. Namely, we observed cases where improper formatting of otherwise functioning code causes a syntax error. For specific examples and more details, see the Supplementary Materials.Evaluations on Larger Models.Since the public release of APPS, several others have trained even larger models on APPS than we evaluate here. OpenAI Codex is a 12B parameter Transformer language model pre-trained on large quantities of public code and comments.evaluate Codex on APPS under various configurations and achieve top-1 and top-5 accuracy on introductory problems of 4.14% and 9.65% respectively, close to double the top-5 accuracy of GPT-Neo 2.7B. Furthermore, by scaling up to a top-1000 evaluation they obtain 25% accuracy. This demonstrates that larger models trained specifically for code generation can improve APPS performance even further, but are still far from solving the task.
body
Measuring Coding Challenge Competence With APPS
4Experiments
2105.09938
5ConclusionWe introduced APPS, a benchmark ofPython programming problems. Unlike prior work that focused on pseudocode to code generation or translation between programming languages, our benchmark measures how well language models can generate python code given natural language specifications. By performing extensive quality assurance and including hundreds of thousands of test cases and ground-truth solutions across different difficulty levels, we created a comprehensive and rigorous testbed for evaluating models. We assessed state-of-the-art generative models on our benchmark and found that overall performance was low. However, the prevalence of syntax errors decreased exponentially as models improved, and recent models such as GPT-Neo solved overof our introductory problems. As models become more competent at code generation, it is important to have a proxy for tracking this capability which could one day result in automation or malicious code generation. The APPS benchmark can provide an important measure for tracking upstream program synthesis advancements.
conclusion
Measuring Coding Challenge Competence With APPS
5Conclusion