<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.07565] On Leakage of Code Generation Evaluation Datasets</title><meta property="og:description" content="In this paper we consider contamination by code generation test sets, in particular in their use in modern large language models.
We discuss three possible sources of such contamination and show findings supporting eac…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="On Leakage of Code Generation Evaluation Datasets">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="On Leakage of Code Generation Evaluation Datasets">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.07565">

<!--Generated on Mon Aug  5 15:54:47 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">On Leakage of Code Generation Evaluation Datasets</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname"><span id="id1.1.id1" class="ltx_text ltx_font_bold">Alexandre Matton</span>, <span id="id2.2.id2" class="ltx_text ltx_font_bold">Tom Sherborne</span>, <span id="id3.3.id3" class="ltx_text ltx_font_bold">Dennis Aumiller</span>, 
<br class="ltx_break"><span id="id4.4.id4" class="ltx_text ltx_font_bold">Elena Tommasone</span>, <span id="id5.5.id5" class="ltx_text ltx_font_bold">Milad Alizadeh</span>,
<span id="id6.6.id6" class="ltx_text ltx_font_bold">Jingyi He</span>, 
<br class="ltx_break"><span id="id7.7.id7" class="ltx_text ltx_font_bold">Raymond Ma</span>, <span id="id8.8.id8" class="ltx_text ltx_font_bold">Maxime Voisin</span>, <span id="id9.9.id9" class="ltx_text ltx_font_bold">Ellen Gilsenan-McMahon</span> 
<br class="ltx_break"><span id="id10.10.id10" class="ltx_text ltx_font_bold">Matthias Gallé</span> 
<br class="ltx_break">Cohere
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id11.id1" class="ltx_p">In this paper we consider contamination by code generation test sets, in particular in their use in modern large language models.
We discuss three possible sources of such contamination and show findings supporting each of them: (i) direct data leakage, (ii) indirect data leakage through the use of synthetic data and (iii) overfitting to evaluation sets during model selection.</p>
<p id="id12.id2" class="ltx_p">Key to our findings is a new dataset of 161 prompts with their associated python solutions, dataset which is released at <a target="_blank" href="https://huggingface.co/datasets/CohereForAI/lbpp" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/CohereForAI/lbpp</a>.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Code generation has emerged as an important skill for large language models to master.
Measuring recent progress in code generation has relied on few, critical benchmarks to judge performance between model families and checkpoints. While many recent sophisticated evaluation datasets have been proposed <cite class="ltx_cite ltx_citemacro_citep">(Jain et al., <a href="#bib.bib6" title="" class="ltx_ref">2024</a>; Jimenez et al., <a href="#bib.bib7" title="" class="ltx_ref">2024</a>)</cite>, the community largely relies on <span id="S1.p1.1.1" class="ltx_text ltx_font_typewriter">HumanEval</span> <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib3" title="" class="ltx_ref">2021</a>)</cite> and <span id="S1.p1.1.2" class="ltx_text ltx_font_typewriter">MBPP</span> <cite class="ltx_cite ltx_citemacro_citep">(Austin et al., <a href="#bib.bib1" title="" class="ltx_ref">2021</a>)</cite> to judge a new model’s code capability. In fact, all major announcements in 2023-2024 claiming advanced code capabilities—from academic and industry labs—use at least one of these two datasets. Practically, reporting <span id="S1.p1.1.3" class="ltx_text ltx_font_typewriter">HumanEval</span> and <span id="S1.p1.1.4" class="ltx_text ltx_font_typewriter">MBPP</span> is mandatory for a model to report competitive code generation.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">However, the importance of these benchmarks has led to a conflict between popularity and utility.
On one side, obtaining competitive numbers comes with significant scientific and economic reward—made increasingly easy with the proliferation of public replicas of these datasets.
However, this prevalence has led to data leakage beyond the original evaluation scope, i.e., <span id="S1.p2.1.1" class="ltx_text ltx_font_italic">data contamination</span>, and once this evaluation data <span id="S1.p2.1.2" class="ltx_text ltx_font_italic">contaminates</span> model training, the validity of the metrics as a measure of generalization capability becomes unreliable.
If a model has been trained on the same data we use for out-of-distribution generalization (or is selected based on its performance on that data), we break an implicit tenet of how model capability can be measured. We argue that understanding the effect of contamination is critical to accurately interpreting scores on these benchmarks.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this paper, we review the evidence that these two benchmarks have contaminated most large LLMs, which we define as any procedure that leaked those datasets <span id="S1.p3.1.1" class="ltx_text ltx_font_italic">during</span> model training.
The most obvious method of contamination is presence inside training data, and we provide evidence that it is highly probably that this occurs at a scale too large to be avoidable.
A second possibility is that contamination happens indirectly through the use of synthetic data—a widespread paradigm used in particular to increase code capabilities by generating additional code training tokens.
Finally, we argue that final model selection might have been overly influenced by their performance on these datasets, overfitting to performance on these metrics over general-purpose code-oriented skills.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">To measure this contamination, we propose <span id="S1.p4.1.1" class="ltx_text ltx_font_bold">Less</span> <span id="S1.p4.1.2" class="ltx_text ltx_font_bold">Basic</span> <span id="S1.p4.1.3" class="ltx_text ltx_font_bold">Python</span> <span id="S1.p4.1.4" class="ltx_text ltx_font_bold">Problems</span> (LBPP), a code generation benchmark similar to <span id="S1.p4.1.5" class="ltx_text ltx_font_typewriter">HumanEval</span> and <span id="S1.p4.1.6" class="ltx_text ltx_font_typewriter">MBPP</span> in style and scale, but more difficult.
LBPP is similarly portable, but is produced in a manner to reduce any likelihood of leakage into present code training data. We contribute LBPP to act as a genuinely held-out test set to measure <span id="S1.p4.1.7" class="ltx_text ltx_font_italic">current</span> code generation capability, and potential overfitting to <span id="S1.p4.1.8" class="ltx_text ltx_font_typewriter">HumanEval</span> and <span id="S1.p4.1.9" class="ltx_text ltx_font_typewriter">MBPP</span>.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p"><span id="S2.p1.1.1" class="ltx_text ltx_font_typewriter">HumanEval</span> <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib3" title="" class="ltx_ref">2021</a>)</cite> and <span id="S2.p1.1.2" class="ltx_text ltx_font_typewriter">MBPP</span> <cite class="ltx_cite ltx_citemacro_citep">(Austin et al., <a href="#bib.bib1" title="" class="ltx_ref">2021</a>)</cite> remain the most reported results on public leaderboards, but others similar datasets exist <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib5" title="" class="ltx_ref">2021</a>; Li et al., <a href="#bib.bib11" title="" class="ltx_ref">2022</a>)</cite>.
They consist of short and mostly simple (not programming competition level) instructions with completions in Python.
Translation into other programming languages exist for those datasets <cite class="ltx_cite ltx_citemacro_citep">(Muennighoff et al., <a href="#bib.bib13" title="" class="ltx_ref">2023</a>; Cassano et al., <a href="#bib.bib2" title="" class="ltx_ref">2022</a>)</cite>, as well as versions with additional tests <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib12" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.2" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citep">(Jain et al., <a href="#bib.bib6" title="" class="ltx_ref">2024</a>)</cite> proposed a continuously updated set of leetcode to improve dataset challenge by including harder and novel (unseen) prompts.
<cite class="ltx_cite ltx_citemacro_citep">(Jimenez et al., <a href="#bib.bib7" title="" class="ltx_ref">2024</a>)</cite> aims for challenging software engineering problems, that require understanding of full repositories.
In a similar vein, RepoQA<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span><a target="_blank" href="https://github.com/evalplus/repoqa" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/evalplus/repoqa</a></span></span></span> and Bug In The Code Stack<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span><a target="_blank" href="https://github.com/HammingHQ/bug-in-the-code-stack" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/HammingHQ/bug-in-the-code-stack</a></span></span></span> focus on understanding long contexts within code tasks.
One proposed solution is to use hidden evaluation sets <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib20" title="" class="ltx_ref">2024</a>)</cite>, however, these do not allow inspection of failure cases and requires trusting the quality and correctness of an opaque ‘black-box’ evaluation setup.
Recently, <cite class="ltx_cite ltx_citemacro_citet">Riddell et al. (<a href="#bib.bib14" title="" class="ltx_ref">2024</a>)</cite> analyzed data contamination in popular pretraining datasets: reporting that <math id="S2.p2.1.m1.1" class="ltx_Math" alttext="12.2\%" display="inline"><semantics id="S2.p2.1.m1.1a"><mrow id="S2.p2.1.m1.1.1" xref="S2.p2.1.m1.1.1.cmml"><mn id="S2.p2.1.m1.1.1.2" xref="S2.p2.1.m1.1.1.2.cmml">12.2</mn><mo id="S2.p2.1.m1.1.1.1" xref="S2.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.p2.1.m1.1b"><apply id="S2.p2.1.m1.1.1.cmml" xref="S2.p2.1.m1.1.1"><csymbol cd="latexml" id="S2.p2.1.m1.1.1.1.cmml" xref="S2.p2.1.m1.1.1.1">percent</csymbol><cn type="float" id="S2.p2.1.m1.1.1.2.cmml" xref="S2.p2.1.m1.1.1.2">12.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.1.m1.1c">12.2\%</annotation></semantics></math> of <span id="S2.p2.2.1" class="ltx_text ltx_font_typewriter">HumanEval</span> samples are present in The Pile <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib4" title="" class="ltx_ref">2020</a>)</cite>, and <math id="S2.p2.2.m2.1" class="ltx_Math" alttext="18.9\%" display="inline"><semantics id="S2.p2.2.m2.1a"><mrow id="S2.p2.2.m2.1.1" xref="S2.p2.2.m2.1.1.cmml"><mn id="S2.p2.2.m2.1.1.2" xref="S2.p2.2.m2.1.1.2.cmml">18.9</mn><mo id="S2.p2.2.m2.1.1.1" xref="S2.p2.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.p2.2.m2.1b"><apply id="S2.p2.2.m2.1.1.cmml" xref="S2.p2.2.m2.1.1"><csymbol cd="latexml" id="S2.p2.2.m2.1.1.1.cmml" xref="S2.p2.2.m2.1.1.1">percent</csymbol><cn type="float" id="S2.p2.2.m2.1.1.2.cmml" xref="S2.p2.2.m2.1.1.2">18.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.2.m2.1c">18.9\%</annotation></semantics></math> in The Stack <cite class="ltx_cite ltx_citemacro_citep">(Kocetkov et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite>.
Different from our analysis however, they conclude: “we do not find the performance rankings of the models to change with decontaminated results”.</p>
</div>
<figure id="S2.F1" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F1.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.07565/assets/humaneval.png" id="S2.F1.sf1.g1" class="ltx_graphics ltx_img_landscape" width="598" height="472" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span><span id="S2.F1.sf1.2.1" class="ltx_text ltx_font_typewriter">HumanEval</span> </figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F1.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.07565/assets/mbpp.png" id="S2.F1.sf2.g1" class="ltx_graphics ltx_img_landscape" width="598" height="472" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span><span id="S2.F1.sf2.2.1" class="ltx_text ltx_font_typewriter">MBPP</span> </figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Pass@1 rate of popular datasets and the 161 prompts in LBPP.</figcaption>
</figure>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Possible sources of contamination</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">We provide three hypotheses—with evidence for each one—on why existing models might be over-optimized towards existing leaked benchmarks.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Direct data leakage</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.2" class="ltx_p">The most obvious reason is the simplest: many of the test datasets are of widespread use and the simplest answer might be that modern LLMs are just trained on this evaluation data.
We note that intentional (i.e., to <span id="S3.SS1.p1.2.1" class="ltx_text ltx_font_italic">cheat</span>) or unintentional contamination has the same net effect: training on evaluation data limits the confidence and utility of the benchmark results.
For code tasks, it is very expensive to curate datasets of natural language to code instructions (one example generally costing several dozen US dollars).
For any group aiming to minimize data cost to improve coding performance, the dollar value of creating new datasets can be very high.
This leads to a common practice of web scraping code-oriented resources (e.g., GitHub or Stackoverflow) for data. However, these resources are also likely sources of contamination.
Since the release of <span id="S3.SS1.p1.2.2" class="ltx_text ltx_font_typewriter">HumanEval</span> and <span id="S3.SS1.p1.2.3" class="ltx_text ltx_font_typewriter">MBPP</span> in 2021 these datasets have been branched, re-used and copied all across the Internet.
The small data size and portability of such benchmarks encourages replication within code repositories.
For example, searching for the prompts from <span id="S3.SS1.p1.2.4" class="ltx_text ltx_font_typewriter">HumanEval</span> on GitHub returns a hit in all cases—the median hits is <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="99" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mn id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">99</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><cn type="integer" id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">99</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">99</annotation></semantics></math> and the minimum <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="43" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mn id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml">43</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><cn type="integer" id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">43</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">43</annotation></semantics></math> (see Fig. <a href="#S3.F2" title="Figure 2 ‣ 3.1 Direct data leakage ‣ 3 Possible sources of contamination ‣ On Leakage of Code Generation Evaluation Datasets" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>). In many cases, these hits are exact duplicates and indications of a fork of the original dataset.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p">While decontamination of training sets is becoming more common, present decontamination filters designed for natural text adapts poorly to code.
To operate efficiently at scale, most filters rely on generic deduplication algorithms e.g., such as <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mi id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><ci id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">n</annotation></semantics></math>-gram matching or hashing functions <cite class="ltx_cite ltx_citemacro_cite">Lee et al. (<a href="#bib.bib9" title="" class="ltx_ref">2022</a>)</cite>.
Such surface-level matching does not adequately capture code similarity where a simple variable name change leaves program semantics unchanged, but changing a single keyword can have profound changes.<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>E.g., compare the instruction “<span id="footnote3.1" class="ltx_text ltx_font_typewriter">return true if the string is a float</span>” with “<span id="footnote3.2" class="ltx_text ltx_font_typewriter">return true if the string is a verb</span>”.</span></span></span>
The same shortcomings of decontamination efforts apply to the creation of large-scale synthetic datasets: for example the model-generated dataset of Starcoder <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite> is decontaminated only by removing exact docstrings or solutions that match <span id="S3.SS1.p2.1.1" class="ltx_text ltx_font_typewriter">HumanEval</span> or <span id="S3.SS1.p2.1.2" class="ltx_text ltx_font_typewriter">MBPP</span>.</p>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2407.07565/assets/x1.png" id="S3.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="232" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Histogram (excluding outliers) of occurrences for <span id="S3.F2.4.1" class="ltx_text ltx_font_typewriter">HumanEval</span> prompts in public GitHub repositories. Every prompt occurs at least <math id="S3.F2.2.m1.1" class="ltx_Math" alttext="43" display="inline"><semantics id="S3.F2.2.m1.1b"><mn id="S3.F2.2.m1.1.1" xref="S3.F2.2.m1.1.1.cmml">43</mn><annotation-xml encoding="MathML-Content" id="S3.F2.2.m1.1c"><cn type="integer" id="S3.F2.2.m1.1.1.cmml" xref="S3.F2.2.m1.1.1">43</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.2.m1.1d">43</annotation></semantics></math> times.</figcaption>
</figure>
<div id="S3.SS1.p3" class="ltx_para">
<p id="S3.SS1.p3.1" class="ltx_p">The recent exploration of <cite class="ltx_cite ltx_citemacro_citet">Riddell et al. (<a href="#bib.bib14" title="" class="ltx_ref">2024</a>)</cite> aims to quantify the proportion of this data leakage in existing datasets using plagiarism tools specifically designed for code.
Even when static training datasets are cleaned, contamination may persist.
Entities who serve models through an API may encounter these benchmark tasks when evaluated by third party users. When a sample of real model usage is annotated for future training data, samples from benchmark evaluation can leak into future training corpora. Furthermore, these samples may include subtle phrasing variations and format changes that further complicate heuristic deduplication.
In this scenario, a model may easily memorize completions to purportedly novel prompts. As evidence of this phenomena, we prompted one popular commercial system with partial prompts from <span id="S3.SS1.p3.1.1" class="ltx_text ltx_font_typewriter">HumanEval</span> that were designed to keep the instruction under-specified.
Table <a href="#Ax1" title="Appendix ‣ On Leakage of Code Generation Evaluation Datasets" class="ltx_ref"><span class="ltx_text ltx_ref_title">Appendix</span></a> in the Appendix shows the outcome and evidence that—despite the ambiguity of the prompt—the resulting completion matches exactly the gold solution from the test set of <span id="S3.SS1.p3.1.2" class="ltx_text ltx_font_typewriter">HumanEval</span>.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Data leakage through synthetic data</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">The most capable of code language models rely heavily on the use of synthetic training data <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>; Wei et al., <a href="#bib.bib17" title="" class="ltx_ref">2023</a>, <a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">A typical pipeline generally consists of: curating prompts related to code generation, inferring completions with a previously trained LLM, and synthesizing unit tests for relevant prompts using LLMs.
Completions that pass the respective unit tests are considered valid code solutions and can be used as future training examples.
Alternatively, if a sufficiently powerful model is used, completions might be used as-is.</p>
</div>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.1" class="ltx_p"><span id="S3.SS2.p3.1.1" class="ltx_text ltx_font_typewriter">evol-instruct</span> for example comprises <math id="S3.SS2.p3.1.m1.1" class="ltx_Math" alttext="110k" display="inline"><semantics id="S3.SS2.p3.1.m1.1a"><mrow id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml"><mn id="S3.SS2.p3.1.m1.1.1.2" xref="S3.SS2.p3.1.m1.1.1.2.cmml">110</mn><mo lspace="0em" rspace="0em" id="S3.SS2.p3.1.m1.1.1.1" xref="S3.SS2.p3.1.m1.1.1.1.cmml">​</mo><mi id="S3.SS2.p3.1.m1.1.1.3" xref="S3.SS2.p3.1.m1.1.1.3.cmml">k</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.1b"><apply id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1"><times id="S3.SS2.p3.1.m1.1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1.1"></times><cn type="integer" id="S3.SS2.p3.1.m1.1.1.2.cmml" xref="S3.SS2.p3.1.m1.1.1.2">110</cn><ci id="S3.SS2.p3.1.m1.1.1.3.cmml" xref="S3.SS2.p3.1.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.1c">110k</annotation></semantics></math> complex query prompts coupled with completions from numerous closed and open-source models.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>Per downloads, the most popular version is a ‘lightly decontaminated’ version <a target="_blank" href="https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K" title="" class="ltx_ref ltx_href">on HuggingFace here</a>.</span></span></span>
It is widely used by many code LLMs such as WizardCoder <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite>.
Prior reports  <cite class="ltx_cite ltx_citemacro_citep">(Yu et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>, page 8)</cite>, <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib17" title="" class="ltx_ref">2023</a>, page 4)</cite> discuss an apparent high similarity between some of those examples in <span id="S3.SS2.p3.1.2" class="ltx_text ltx_font_typewriter">evol-instruct</span> and <span id="S3.SS2.p3.1.3" class="ltx_text ltx_font_typewriter">HumanEval</span>.
We extend this analysis by studying the similarity between embedded representations of the <span id="S3.SS2.p3.1.4" class="ltx_text ltx_font_italic">prompts<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note"><span id="footnote5.1.1.1" class="ltx_text ltx_font_upright">5</span></span><span id="footnote5.5" class="ltx_text ltx_font_upright">Embedded using Cohere </span><span id="footnote5.6" class="ltx_text ltx_font_typewriter ltx_font_upright">embed v3</span><span id="footnote5.7" class="ltx_text ltx_font_upright"> </span><cite class="ltx_cite ltx_citemacro_citep"><span id="footnote5.8.1" class="ltx_text ltx_font_upright">(</span>Team<span id="footnote5.9.2.1.1" class="ltx_text ltx_font_upright">, </span><a href="#bib.bib15" title="" class="ltx_ref">2024</a><span id="footnote5.10.3" class="ltx_text ltx_font_upright">)</span></cite><span id="footnote5.11" class="ltx_text ltx_font_upright">.</span></span></span></span></span> of <span id="S3.SS2.p3.1.5" class="ltx_text ltx_font_typewriter">HumanEval</span> and <span id="S3.SS2.p3.1.6" class="ltx_text ltx_font_typewriter">MBPP</span> with nearest neighbors from <span id="S3.SS2.p3.1.7" class="ltx_text ltx_font_italic">evol-instruct</span> and <span id="S3.SS2.p3.1.8" class="ltx_text ltx_font_italic">Starcoder-Instruct</span>. Fig <a href="#S3.F3" title="Figure 3 ‣ 3.2 Data leakage through synthetic data ‣ 3 Possible sources of contamination ‣ On Leakage of Code Generation Evaluation Datasets" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> highlights widespread similarity between evaluation data and synthetic training datasets ‘Starcoder V2 OSS Instruct’ and ‘evol instruct’. Even if unintentional, this contamination further damages the utility of these benchmarks as a held-out test of code generation capability.</p>
</div>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2407.07565/assets/x2.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="230" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Histogram of cosine similarities for prompts in <span id="S3.F3.5.1" class="ltx_text ltx_font_typewriter">HumanEval</span>, <span id="S3.F3.6.2" class="ltx_text ltx_font_typewriter">MBPP</span> and LBPP relative to two popular synthetic code training datasets. We note the high similarity between most <span id="S3.F3.7.3" class="ltx_text ltx_font_typewriter">HumanEval</span> prompts to <span id="S3.F3.8.4" class="ltx_text ltx_font_italic">evol-instruct</span>, and how LBPP has reduced overall similarity to either training dataset.
</figcaption>
</figure>
<div id="S3.SS2.p4" class="ltx_para">
<p id="S3.SS2.p4.1" class="ltx_p">Strikingly, we saw that training on this dataset can increase performance of one of our models by 14 absolute points on <span id="S3.SS2.p4.1.1" class="ltx_text ltx_font_typewriter">HumanEval</span> (from 0.52 pass@1 to 0.66), while only by 1 point on <span id="S3.SS2.p4.1.2" class="ltx_text ltx_font_typewriter">MBPP</span> (from 0.52 to 0.53).
Inspecting closest neighbours for almost all examples in <span id="S3.SS2.p4.1.3" class="ltx_text ltx_font_typewriter">HumanEval</span> (see examples in Table <a href="#Ax1" title="Appendix ‣ On Leakage of Code Generation Evaluation Datasets" class="ltx_ref"><span class="ltx_text ltx_ref_title">Appendix</span></a> in the Appendix)—we identify a semantically equivalent version in <span id="S3.SS2.p4.1.4" class="ltx_text ltx_font_typewriter">evol-instruct</span>.</p>
</div>
<div id="S3.SS2.p5" class="ltx_para">
<p id="S3.SS2.p5.1" class="ltx_p">Even when the original evaluation datasets were not used as inspiration for the creation of the synthetic datasets the simple nature of the evaluation datasets might make duplication unavoidable.
There are only so many short natural language prompts describing typical interview-style programming questions that can be used.
While synthetically generated prompts may not be explicitly based on provided examples from a given test set in the prompt context, the massive scale of these datasets (<math id="S3.SS2.p5.1.m1.1" class="ltx_Math" alttext="238k" display="inline"><semantics id="S3.SS2.p5.1.m1.1a"><mrow id="S3.SS2.p5.1.m1.1.1" xref="S3.SS2.p5.1.m1.1.1.cmml"><mn id="S3.SS2.p5.1.m1.1.1.2" xref="S3.SS2.p5.1.m1.1.1.2.cmml">238</mn><mo lspace="0em" rspace="0em" id="S3.SS2.p5.1.m1.1.1.1" xref="S3.SS2.p5.1.m1.1.1.1.cmml">​</mo><mi id="S3.SS2.p5.1.m1.1.1.3" xref="S3.SS2.p5.1.m1.1.1.3.cmml">k</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.1.m1.1b"><apply id="S3.SS2.p5.1.m1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1"><times id="S3.SS2.p5.1.m1.1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1.1"></times><cn type="integer" id="S3.SS2.p5.1.m1.1.1.2.cmml" xref="S3.SS2.p5.1.m1.1.1.2">238</cn><ci id="S3.SS2.p5.1.m1.1.1.3.cmml" xref="S3.SS2.p5.1.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.1.m1.1c">238k</annotation></semantics></math> instances for StarCoder-Instruct prior to deduplication) runs the risk of exhausting the possible number of variations on questions.
Table <a href="#Ax1.T1" title="Table 1 ‣ Appendix ‣ On Leakage of Code Generation Evaluation Datasets" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows examples where generated prompts are extremely similar to questions in the <span id="S3.SS2.p5.1.1" class="ltx_text ltx_font_typewriter">MBPP</span> test set.
Note that despite that similarity training on this dataset did not improve substantially performance on <span id="S3.SS2.p5.1.2" class="ltx_text ltx_font_typewriter">MBPP</span>.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Overfitting to test sets</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">The exaggerated importance of these benchmarks encourages an incentive structure where model selection prioritizes gain on a narrow suite of metrics. While it may be tempting to use such benchmarks as a deciding factor between similar checkpoints, there is weak evidence for these benchmarks correlating with ‘solving code generation’. While the meaning and measurement of this unscientific objective is subject to constant revision, selecting for optimal <span id="S3.SS3.p1.1.1" class="ltx_text ltx_font_typewriter">HumanEval</span> performance may be akin to <math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="p" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><mi id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><ci id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1">𝑝</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">p</annotation></semantics></math>-hacking in other fields.
This practice can be justified by assuming that these benchmarks are the new dev sets, while the true test is the usage of users over time.
The risk remains however that some models overfit to those test sets more than others, distorting perception on relative performance.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.1" class="ltx_p">In order to measure this, we created <span id="S3.SS3.p2.1.1" class="ltx_text ltx_font_bold">Less Basic Python Problems (LBPP)</span>, a dataset of 161 code completion problems in the style of <span id="S3.SS3.p2.1.2" class="ltx_text ltx_font_typewriter">HumanEval</span>.
Human annotators were instructed to create totally fresh problems, which were not solvable by an internal model<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>We will update references to internal models in future revisions.</span></span></span> they had access to.
Annotators had competitive programming experience and could use programming books as inspiration, but were instructed not to copy existing solutions on the Internet and not to use any LLMs.
All annotators were paid above minimum wage in their respective countries, and all final prompt-completion pairs were manually reviewed by the authors.
This adversarial collection resulted in more difficult problems, with most models solving less than <math id="S3.SS3.p2.1.m1.1" class="ltx_Math" alttext="50\%" display="inline"><semantics id="S3.SS3.p2.1.m1.1a"><mrow id="S3.SS3.p2.1.m1.1.1" xref="S3.SS3.p2.1.m1.1.1.cmml"><mn id="S3.SS3.p2.1.m1.1.1.2" xref="S3.SS3.p2.1.m1.1.1.2.cmml">50</mn><mo id="S3.SS3.p2.1.m1.1.1.1" xref="S3.SS3.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.1b"><apply id="S3.SS3.p2.1.m1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1"><csymbol cd="latexml" id="S3.SS3.p2.1.m1.1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S3.SS3.p2.1.m1.1.1.2.cmml" xref="S3.SS3.p2.1.m1.1.1.2">50</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.1c">50\%</annotation></semantics></math> of the dataset.
Results on a selection of models are in Table <a href="#Ax1.T4" title="Table 4 ‣ Appendix ‣ On Leakage of Code Generation Evaluation Datasets" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> in the Appendix.</p>
</div>
<div id="S3.SS3.p3" class="ltx_para">
<p id="S3.SS3.p3.2" class="ltx_p">Using this dataset we can correlate the performances of existing models on them against performance of the well-known benchmarks.
The two plots comparing them to <span id="S3.SS3.p3.2.1" class="ltx_text ltx_font_typewriter">HumanEval</span> and <span id="S3.SS3.p3.2.2" class="ltx_text ltx_font_typewriter">MBPP</span> can be seen in Fig. <a href="#S2.F1" title="Figure 1 ‣ 2 Related Work ‣ On Leakage of Code Generation Evaluation Datasets" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
There is a clear correlation on both data-sets, indicating that the public benchmarks are still a valuable target signal.
However, when zooming in — in particular for models from the same family — the correlation often becomes negative, which might indicate that the selected checkpoint to release performs better on those public datasets while under-performing on new datapoints.
Note in particular in Fig. <a href="#S2.F1.sf1" title="In Figure 1 ‣ 2 Related Work ‣ On Leakage of Code Generation Evaluation Datasets" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1(a)</span></a> the crowded space between 0.75 and 0.8 of the <math id="S3.SS3.p3.1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS3.p3.1.m1.1a"><mi id="S3.SS3.p3.1.m1.1.1" xref="S3.SS3.p3.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.1.m1.1b"><ci id="S3.SS3.p3.1.m1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.1.m1.1c">x</annotation></semantics></math>-axis (pass@1 of <span id="S3.SS3.p3.2.3" class="ltx_text ltx_font_typewriter">HumanEval</span>).
Despite very similar performance on that public evaluation set, the numbers of LBPP vary wildly.
On the other side, in Fig. <a href="#S2.F1.sf2" title="In Figure 1 ‣ 2 Related Work ‣ On Leakage of Code Generation Evaluation Datasets" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1(b)</span></a> a vertical line can be traced at around <math id="S3.SS3.p3.2.m2.1" class="ltx_Math" alttext="0.38" display="inline"><semantics id="S3.SS3.p3.2.m2.1a"><mn id="S3.SS3.p3.2.m2.1.1" xref="S3.SS3.p3.2.m2.1.1.cmml">0.38</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.2.m2.1b"><cn type="float" id="S3.SS3.p3.2.m2.1.1.cmml" xref="S3.SS3.p3.2.m2.1.1">0.38</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.2.m2.1c">0.38</annotation></semantics></math> of LBPP that crosses various points – indicating maybe an overfit to <span id="S3.SS3.p3.2.4" class="ltx_text ltx_font_typewriter">MBPP</span>.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Conclusion</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">We study the cause and effect of data contamination by two popular code generation benchmarks. Our analysis highlights that contamination is likely unavoidable at the LLM scale given the difficulty of filtering every potential permutation of a benchmark dataset.
This insight motivates our contribution of a novel code generation benchmark to evaluate contemporary LLMs in a contamination-free setting.
We are well aware that our decision to release this dataset will make future leakage impossible to control.
However, with the context of the fast-paced model development cycles that LLMs are currently undergoing we believe that releasing this increases trustworthiness and usefulness of this dataset.
It is conveniently designed to serve as drop-in replacement (or addition) of current evaluation sets.
On top of its newness, the more challenging nature of this dataset also allows it to provide more signal for model comparison.</p>
</div>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Limitations</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">All the model analysis was done <span id="S5.p1.1.1" class="ltx_text ltx_font_italic">black-box</span>, without inspecting the model weights or the training set (excepting the work on synthetic data).
There is no reason why this dataset will not follow the same path than the two studied here.
As mentioned in the Conclusion we believe there is more value in that than in an alternative solution (not releasing or keeping it behind an API access).</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Austin et al. (2021)</span>
<span class="ltx_bibblock">
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2108.07732" title="" class="ltx_ref ltx_href">Program synthesis with large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2108.07732.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cassano et al. (2022)</span>
<span class="ltx_bibblock">
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2022.

</span>
<span class="ltx_bibblock">Multipl-e: A scalable and extensible approach to benchmarking neural code generation.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2208.08227</em>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2021)</span>
<span class="ltx_bibblock">
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021.

</span>
<span class="ltx_bibblock">Evaluating large language models trained on code.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2107.03374</em>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2020)</span>
<span class="ltx_bibblock">
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.

</span>
<span class="ltx_bibblock">The pile: An 800gb dataset of diverse text for language modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2101.00027</em>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021.

</span>
<span class="ltx_bibblock">Measuring coding challenge competence with apps.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2105.09938</em>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jain et al. (2024)</span>
<span class="ltx_bibblock">
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024.

</span>
<span class="ltx_bibblock">Livecodebench: Holistic and contamination free evaluation of large language models for code.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.07974</em>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jimenez et al. (2024)</span>
<span class="ltx_bibblock">
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=VTF8yNQM66" title="" class="ltx_ref ltx_href">SWE-bench: Can language models resolve real-world github issues?</a>

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">The Twelfth International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kocetkov et al. (2022)</span>
<span class="ltx_bibblock">
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. 2022.

</span>
<span class="ltx_bibblock">The stack: 3 tb of permissively licensed source code.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2211.15533</em>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2022)</span>
<span class="ltx_bibblock">
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.acl-long.577" title="" class="ltx_ref ltx_href">Deduplicating training data makes language models better</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>, pages 8424–8445, Dublin, Ireland. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023)</span>
<span class="ltx_bibblock">
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023.

</span>
<span class="ltx_bibblock">Starcoder: may the source be with you!

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.06161</em>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2022)</span>
<span class="ltx_bibblock">
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022.

</span>
<span class="ltx_bibblock">Competition-level code generation with alphacode.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Science</em>, 378(6624):1092–1097.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2024)</span>
<span class="ltx_bibblock">
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024.

</span>
<span class="ltx_bibblock">Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Muennighoff et al. (2023)</span>
<span class="ltx_bibblock">
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023.

</span>
<span class="ltx_bibblock">Octopack: Instruction tuning code large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.07124</em>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Riddell et al. (2024)</span>
<span class="ltx_bibblock">
Martin Riddell, Ansong Ni, and Arman Cohan. 2024.

</span>
<span class="ltx_bibblock">Quantifying contamination in evaluating code generation capabilities of language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.04811</em>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Team (2024)</span>
<span class="ltx_bibblock">
Cohere Embedding Team. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://huggingface.co/Cohere/Cohere-embed-english-v3.0" title="" class="ltx_ref ltx_href">Cohere embed-english-v3.0</a>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2024)</span>
<span class="ltx_bibblock">
Yuxiang Wei, Federico Cassano, Yifeng Ding, Naman Jain, Harm de Vries, Leandro von Werra, Arjun Guha, and Lingming Zhang. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://huggingface.co/blog/sc2-instruct" title="" class="ltx_ref ltx_href">Starcoder2-instruct: Fully transparent and permissive self-alignment for code generation</a>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2023)</span>
<span class="ltx_bibblock">
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023.

</span>
<span class="ltx_bibblock">Magicoder: Source code is all you need.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.02120</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2023)</span>
<span class="ltx_bibblock">
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2023.

</span>
<span class="ltx_bibblock">Wizardlm: Empowering large pre-trained language models to follow complex instructions.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">The Twelfth International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. (2023)</span>
<span class="ltx_bibblock">
Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. 2023.

</span>
<span class="ltx_bibblock">Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.14187</em>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2024)</span>
<span class="ltx_bibblock">
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. 2024.

</span>
<span class="ltx_bibblock">A careful examination of large language model performance on grade school arithmetic.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2405.00332</em>.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="Ax1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">Appendix</h2>

<figure id="Ax1.T1" class="ltx_table">
<table id="Ax1.T1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="Ax1.T1.1.1.1" class="ltx_tr">
<th id="Ax1.T1.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_l ltx_border_r ltx_border_t">
<span id="Ax1.T1.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.T1.1.1.1.1.1.1" class="ltx_p" style="width:108.4pt;"><span id="Ax1.T1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">MBPP test prompt</span></span>
</span>
</th>
<th id="Ax1.T1.1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_r ltx_border_t">
<span id="Ax1.T1.1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.T1.1.1.1.2.1.1" class="ltx_p" style="width:108.4pt;"><span id="Ax1.T1.1.1.1.2.1.1.1" class="ltx_text ltx_font_typewriter ltx_font_bold">evol-instruct<span id="Ax1.T1.1.1.1.2.1.1.1.1" class="ltx_text ltx_font_serif"> prompt</span></span></span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="Ax1.T1.1.2.1" class="ltx_tr">
<td id="Ax1.T1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_l ltx_border_r ltx_border_t">
<span id="Ax1.T1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.T1.1.2.1.1.1.1" class="ltx_p" style="width:108.4pt;">Write a function to find the perimeter of a square</span>
</span>
</td>
<td id="Ax1.T1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r ltx_border_t">
<span id="Ax1.T1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.T1.1.2.1.2.1.1" class="ltx_p" style="width:108.4pt;">Calculate the perimeter of a square</span>
</span>
</td>
</tr>
<tr id="Ax1.T1.1.3.2" class="ltx_tr">
<td id="Ax1.T1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_l ltx_border_r ltx_border_t">
<span id="Ax1.T1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.T1.1.3.2.1.1.1" class="ltx_p" style="width:108.4pt;">Write a python function to find the volume of a triangular prism</span>
</span>
</td>
<td id="Ax1.T1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r ltx_border_t">
<span id="Ax1.T1.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.T1.1.3.2.2.1.1" class="ltx_p" style="width:108.4pt;">How can I use JavaScript to calculate the volume of a triangular prism with an irregular base and uneven side lengths?</span>
</span>
</td>
</tr>
<tr id="Ax1.T1.1.4.3" class="ltx_tr">
<td id="Ax1.T1.1.4.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_b ltx_border_l ltx_border_r ltx_border_t">
<span id="Ax1.T1.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.T1.1.4.3.1.1.1" class="ltx_p" style="width:108.4pt;">Write a function to convert snake case string to camel case string.</span>
</span>
</td>
<td id="Ax1.T1.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_b ltx_border_r ltx_border_t">
<span id="Ax1.T1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.T1.1.4.3.2.1.1" class="ltx_p" style="width:108.4pt;">Convert the sentence to camel case.</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 1: </span>Examples of similar prompts from the test set of <span id="Ax1.T1.3.1" class="ltx_text ltx_font_typewriter">MBPP</span> found in evol_instruct <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite>.</figcaption>
</figure>
<figure id="Ax1.1" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 2: </span>Original human evaluation prompts with the completion from a major LLM provider.</figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<table id="Ax1.1.1" class="ltx_tabular ltx_figure_panel ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="Ax1.1.1.1.1" class="ltx_tr">
<th id="Ax1.1.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_r ltx_border_tt">
<span id="Ax1.1.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.1.1.1.1.1" class="ltx_p" style="width:227.6pt;"><span id="Ax1.1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Original human-eval prompt</span></span>
</span>
</th>
<th id="Ax1.1.1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_r ltx_border_tt">
<span id="Ax1.1.1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.1.1.2.1.1" class="ltx_p" style="width:227.6pt;"><span id="Ax1.1.1.1.1.2.1.1.1" class="ltx_text ltx_font_bold">Submitted Prompt</span></span>
</span>
</th>
<th id="Ax1.1.1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_tt">
<span id="Ax1.1.1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.1.1.3.1.1" class="ltx_p" style="width:227.6pt;"><span id="Ax1.1.1.1.1.3.1.1.1" class="ltx_text ltx_font_bold">Opus completion</span></span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="Ax1.1.1.2.1" class="ltx_tr">
<td id="Ax1.1.1.2.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="Ax1.1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.2.1.1.1.1" class="ltx_p" style="width:227.6pt;">
<span id="Ax1.1.1.2.1.1.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGlzX2hhcHB5KHMpOgoiIiJZb3UgYXJlIGdpdmVuIGEgc3RyaW5nIHMuCllvdXIgdGFzayBpcyB0byBjaGVjayBpZiB0aGUgc3RyaW5nIGlzIGhhcHB5IG9yIG5vdC4KQSBzdHJpbmcgaXMgaGFwcHkgaWYgaXRzIGxlbmd0aCBpcyBhdCBsZWFzdCAzCmFuZCBldmVyeSAzIGNvbnNlY3V0aXZlIGxldHRlcnMgYXJlIGRpc3RpbmN0CkZvciBleGFtcGxlOgppc19oYXBweShhKSA9PiBGYWxzZQppc19oYXBweShhYSkgPT4gRmFsc2UKaXNfaGFwcHkoYWJjZCkgPT4gVHJ1ZQppc19oYXBweShhYWJiKSA9PiBGYWxzZQppc19oYXBweShhZGIpID0+IFRydWUKaXNfaGFwcHkoeHl5KSA9PiBGYWxzZSAiIiI=" download="">⬇</a></span>
<span id="lstnumberx1" class="ltx_listingline"><span id="lstnumberx1.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx1.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_happy</span><span id="lstnumberx1.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx1.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx1.6" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx2" class="ltx_listingline"><span id="lstnumberx2.1" class="ltx_text ltx_font_typewriter">"""</span><span id="lstnumberx2.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">You</span><span id="lstnumberx2.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx2.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx2.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx2.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span><span id="lstnumberx2.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx2.13" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx3" class="ltx_listingline"><span id="lstnumberx3.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Your</span><span id="lstnumberx3.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">task</span><span id="lstnumberx3.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx3.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx3.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">check</span><span id="lstnumberx3.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx3.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx3.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span><span id="lstnumberx3.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx3.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">happy</span><span id="lstnumberx3.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">or</span><span id="lstnumberx3.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">not</span><span id="lstnumberx3.24" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx4" class="ltx_listingline"><span id="lstnumberx4.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">A</span><span id="lstnumberx4.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span><span id="lstnumberx4.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx4.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">happy</span><span id="lstnumberx4.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx4.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">its</span><span id="lstnumberx4.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">length</span><span id="lstnumberx4.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx4.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">at</span><span id="lstnumberx4.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">least</span><span id="lstnumberx4.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.21" class="ltx_text ltx_font_typewriter">3</span>
</span>
<span id="lstnumberx5" class="ltx_listingline"><span id="lstnumberx5.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx5.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx5.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">every</span><span id="lstnumberx5.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx5.5" class="ltx_text ltx_font_typewriter">3</span><span id="lstnumberx5.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx5.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">consecutive</span><span id="lstnumberx5.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx5.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">letters</span><span id="lstnumberx5.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx5.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx5.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx5.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">distinct</span>
</span>
<span id="lstnumberx6" class="ltx_listingline"><span id="lstnumberx6.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">For</span><span id="lstnumberx6.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx6.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">example</span><span id="lstnumberx6.4" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx7" class="ltx_listingline"><span id="lstnumberx7.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_happy</span><span id="lstnumberx7.2" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx7.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx7.4" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx7.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx7.6" class="ltx_text ltx_font_typewriter">=&gt;</span><span id="lstnumberx7.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx7.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">False</span>
</span>
<span id="lstnumberx8" class="ltx_listingline"><span id="lstnumberx8.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_happy</span><span id="lstnumberx8.2" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx8.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">aa</span><span id="lstnumberx8.4" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx8.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.6" class="ltx_text ltx_font_typewriter">=&gt;</span><span id="lstnumberx8.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">False</span>
</span>
<span id="lstnumberx9" class="ltx_listingline"><span id="lstnumberx9.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_happy</span><span id="lstnumberx9.2" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx9.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">abcd</span><span id="lstnumberx9.4" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx9.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx9.6" class="ltx_text ltx_font_typewriter">=&gt;</span><span id="lstnumberx9.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx9.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">True</span>
</span>
<span id="lstnumberx10" class="ltx_listingline"><span id="lstnumberx10.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_happy</span><span id="lstnumberx10.2" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx10.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">aabb</span><span id="lstnumberx10.4" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx10.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx10.6" class="ltx_text ltx_font_typewriter">=&gt;</span><span id="lstnumberx10.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx10.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">False</span>
</span>
<span id="lstnumberx11" class="ltx_listingline"><span id="lstnumberx11.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_happy</span><span id="lstnumberx11.2" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx11.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">adb</span><span id="lstnumberx11.4" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx11.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.6" class="ltx_text ltx_font_typewriter">=&gt;</span><span id="lstnumberx11.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">True</span>
</span>
<span id="lstnumberx12" class="ltx_listingline"><span id="lstnumberx12.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_happy</span><span id="lstnumberx12.2" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx12.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">xyy</span><span id="lstnumberx12.4" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx12.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.6" class="ltx_text ltx_font_typewriter">=&gt;</span><span id="lstnumberx12.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">False</span><span id="lstnumberx12.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.10" class="ltx_text ltx_font_typewriter">"""</span>
</span>
</span></span>
</span>
</td>
<td id="Ax1.1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r ltx_border_t">
<span id="Ax1.1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.2.1.2.1.1" class="ltx_p" style="width:227.6pt;">Continue this code. Do not add any test, just continue it: 
<br class="ltx_break">
<span id="Ax1.1.1.2.1.2.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGlzX2hhcHB5KHMpOgoiIiIgWW91IGFyZSBnaXZlbiBhIHN0cmluZyBzLiBZb3VyIHRhc2sgaXMgdG8gY2hlY2sgaWYgdGhlIHN0cg==" download="">⬇</a></span>
<span id="lstnumberx13" class="ltx_listingline"><span id="lstnumberx13.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx13.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_happy</span><span id="lstnumberx13.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx13.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx13.6" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx14" class="ltx_listingline"><span id="lstnumberx14.1" class="ltx_text ltx_font_typewriter">"""</span><span id="lstnumberx14.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">You</span><span id="lstnumberx14.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx14.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx14.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx14.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span><span id="lstnumberx14.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx14.14" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx14.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Your</span><span id="lstnumberx14.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">task</span><span id="lstnumberx14.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx14.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx14.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">check</span><span id="lstnumberx14.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx14.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx14.29" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter">str</span>
</span>
</span></span>
</span>
</td>
<td id="Ax1.1.1.2.1.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="Ax1.1.1.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.2.1.3.1.1" class="ltx_p" style="width:227.6pt;">Here’s the continuation of the code without adding any tests:

<span id="Ax1.1.1.2.1.3.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGlzX2hhcHB5KHMpOgoiIiIKWW91IGFyZSBnaXZlbiBhIHN0cmluZyBzLiBZb3VyIHRhc2sgaXMgdG8gY2hlY2sgaWYgdGhlIHN0cmluZwppcyBoYXBweS4gQSBzdHJpbmcgaXMgY2FsbGVkIGhhcHB5IGlmIGl0cyBsZW5ndGggaXMgYXQgbGVhc3QgMwphbmQgZXZlcnkgMyBjb25zZWN1dGl2ZSBsZXR0ZXJzIGFyZSBkaXN0aW5jdAoKWy4uLl0=" download="">⬇</a></span>
<span id="lstnumberx15" class="ltx_listingline"><span id="lstnumberx15.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx15.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx15.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_happy</span><span id="lstnumberx15.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx15.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx15.6" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx16" class="ltx_listingline"><span id="lstnumberx16.1" class="ltx_text ltx_font_typewriter">"""</span>
</span>
<span id="lstnumberx17" class="ltx_listingline"><span id="lstnumberx17.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">You</span><span id="lstnumberx17.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx17.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx17.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx17.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span><span id="lstnumberx17.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx17.12" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx17.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Your</span><span id="lstnumberx17.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">task</span><span id="lstnumberx17.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx17.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx17.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">check</span><span id="lstnumberx17.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx17.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx17.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span>
</span>
<span id="lstnumberx18" class="ltx_listingline"><span id="lstnumberx18.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx18.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">happy</span><span id="lstnumberx18.4" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx18.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">A</span><span id="lstnumberx18.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span><span id="lstnumberx18.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx18.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">called</span><span id="lstnumberx18.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">happy</span><span id="lstnumberx18.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx18.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">its</span><span id="lstnumberx18.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">length</span><span id="lstnumberx18.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx18.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">at</span><span id="lstnumberx18.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">least</span><span id="lstnumberx18.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.28" class="ltx_text ltx_font_typewriter">3</span>
</span>
<span id="lstnumberx19" class="ltx_listingline"><span id="lstnumberx19.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx19.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx19.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">every</span><span id="lstnumberx19.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx19.5" class="ltx_text ltx_font_typewriter">3</span><span id="lstnumberx19.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx19.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">consecutive</span><span id="lstnumberx19.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx19.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">letters</span><span id="lstnumberx19.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx19.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx19.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx19.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">distinct</span>
</span>
<span id="lstnumberx20" class="ltx_listingline">
</span>
<span id="lstnumberx21" class="ltx_listingline"><span id="lstnumberx21.1" class="ltx_text ltx_font_typewriter">[...]</span>
</span>
</span></span>
</span>
</td>
</tr>
<tr id="Ax1.1.1.3.2" class="ltx_tr">
<td id="Ax1.1.1.3.2.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="Ax1.1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.3.2.1.1.1" class="ltx_p" style="width:227.6pt;">
<span id="Ax1.1.1.3.2.1.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGNvdW50X3VwcGVyKHMpOgoiIiIgR2l2ZW4gYSBzdHJpbmcgcywgY291bnQgdGhlIG51bWJlciBvZiB1cHBlcmNhc2Ugdm93ZWxzCmluIGV2ZW4gaW5kaWNlcy4KCkZvciBleGFtcGxlOgpjb3VudF91cHBlcignYUJDZEVmJykgcmV0dXJucyAxCmNvdW50X3VwcGVyKCdhYmNkZWZnJykgcmV0dXJucyAwCmNvdW50X3VwcGVyKCdkQkJFJykgcmV0dXJucyAwCiIiIg==" download="">⬇</a></span>
<span id="lstnumberx22" class="ltx_listingline"><span id="lstnumberx22.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx22.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx22.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">count_upper</span><span id="lstnumberx22.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx22.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx22.6" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx23" class="ltx_listingline"><span id="lstnumberx23.1" class="ltx_text ltx_font_typewriter">"""</span><span id="lstnumberx23.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx23.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Given</span><span id="lstnumberx23.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx23.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx23.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx23.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span><span id="lstnumberx23.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx23.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx23.10" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx23.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx23.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">count</span><span id="lstnumberx23.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx23.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx23.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx23.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx23.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx23.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx23.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx23.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">uppercase</span><span id="lstnumberx23.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx23.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">vowels</span>
</span>
<span id="lstnumberx24" class="ltx_listingline"><span id="lstnumberx24.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx24.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">even</span><span id="lstnumberx24.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">indices</span><span id="lstnumberx24.6" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx25" class="ltx_listingline">
</span>
<span id="lstnumberx26" class="ltx_listingline"><span id="lstnumberx26.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">For</span><span id="lstnumberx26.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">example</span><span id="lstnumberx26.4" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx27" class="ltx_listingline"><span id="lstnumberx27.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">count_upper</span><span id="lstnumberx27.2" class="ltx_text ltx_font_typewriter">(’</span><span id="lstnumberx27.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">aBCdEf</span><span id="lstnumberx27.4" class="ltx_text ltx_font_typewriter">’)</span><span id="lstnumberx27.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx27.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">returns</span><span id="lstnumberx27.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx27.8" class="ltx_text ltx_font_typewriter">1</span>
</span>
<span id="lstnumberx28" class="ltx_listingline"><span id="lstnumberx28.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">count_upper</span><span id="lstnumberx28.2" class="ltx_text ltx_font_typewriter">(’</span><span id="lstnumberx28.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">abcdefg</span><span id="lstnumberx28.4" class="ltx_text ltx_font_typewriter">’)</span><span id="lstnumberx28.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx28.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">returns</span><span id="lstnumberx28.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx28.8" class="ltx_text ltx_font_typewriter">0</span>
</span>
<span id="lstnumberx29" class="ltx_listingline"><span id="lstnumberx29.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">count_upper</span><span id="lstnumberx29.2" class="ltx_text ltx_font_typewriter">(’</span><span id="lstnumberx29.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dBBE</span><span id="lstnumberx29.4" class="ltx_text ltx_font_typewriter">’)</span><span id="lstnumberx29.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">returns</span><span id="lstnumberx29.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.8" class="ltx_text ltx_font_typewriter">0</span>
</span>
<span id="lstnumberx30" class="ltx_listingline"><span id="lstnumberx30.1" class="ltx_text ltx_font_typewriter">"""</span>
</span>
</span></span>
</span>
</td>
<td id="Ax1.1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r ltx_border_t">
<span id="Ax1.1.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.3.2.2.1.1" class="ltx_p" style="width:227.6pt;">Complete this instruction, that is taken from the Human Eval test set:

<span id="Ax1.1.1.3.2.2.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,CmRlZiBjb3VudF91cHBlcihzKToKIiIiIEdpdmVuIGEgc3RyaW5nIHMsIGNvdW50IHRoZSBudW1iZXIgb2Y=" download="">⬇</a></span>
<span id="lstnumberx31" class="ltx_listingline">
</span>
<span id="lstnumberx32" class="ltx_listingline"><span id="lstnumberx32.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx32.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">count_upper</span><span id="lstnumberx32.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx32.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx32.6" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx33" class="ltx_listingline"><span id="lstnumberx33.1" class="ltx_text ltx_font_typewriter">"""</span><span id="lstnumberx33.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx33.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Given</span><span id="lstnumberx33.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx33.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx33.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx33.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span><span id="lstnumberx33.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx33.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx33.10" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx33.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx33.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">count</span><span id="lstnumberx33.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx33.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx33.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx33.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx33.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx33.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span>
</span>
</span></span>
</span>
</td>
<td id="Ax1.1.1.3.2.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="Ax1.1.1.3.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.3.2.3.1.1" class="ltx_p" style="width:227.6pt;">Here is the completion of the given function signature and docstring to count the number of uppercase letters in a string:

<span id="Ax1.1.1.3.2.3.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGNvdW50X3VwcGVyKHMpOgoiIiIKR2l2ZW4gYSBzdHJpbmcgcywgY291bnQgdGhlIG51bWJlciBvZiB1cHBlcmNhc2Ugdm93ZWxzIGluCnRoZSBzdHJpbmcuCgpbLi4uXQ==" download="">⬇</a></span>
<span id="lstnumberx34" class="ltx_listingline"><span id="lstnumberx34.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx34.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx34.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">count_upper</span><span id="lstnumberx34.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx34.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx34.6" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx35" class="ltx_listingline"><span id="lstnumberx35.1" class="ltx_text ltx_font_typewriter">"""</span>
</span>
<span id="lstnumberx36" class="ltx_listingline"><span id="lstnumberx36.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Given</span><span id="lstnumberx36.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx36.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span><span id="lstnumberx36.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx36.8" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx36.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">count</span><span id="lstnumberx36.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx36.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx36.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx36.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">uppercase</span><span id="lstnumberx36.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">vowels</span><span id="lstnumberx36.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span>
</span>
<span id="lstnumberx37" class="ltx_listingline"><span id="lstnumberx37.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx37.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx37.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">string</span><span id="lstnumberx37.4" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx38" class="ltx_listingline">
</span>
<span id="lstnumberx39" class="ltx_listingline"><span id="lstnumberx39.1" class="ltx_text ltx_font_typewriter">[...]</span>
</span>
</span></span>
</span>
</td>
</tr>
<tr id="Ax1.1.1.4.3" class="ltx_tr">
<td id="Ax1.1.1.4.3.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="Ax1.1.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.4.3.1.1.1" class="ltx_p" style="width:227.6pt;">
<span id="Ax1.1.1.4.3.1.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGFueV9pbnQoeCwgeSwgeik6CicnJyBDcmVhdGUgYSBmdW5jdGlvbiB0aGF0IHRha2VzIDMgbnVtYmVycy4KUmV0dXJucyB0cnVlIGlmIG9uZSBvZiB0aGUgbnVtYmVycyBpcyBlcXVhbCB0byB0aGUgc3VtIG9mIHRoZQpvdGhlciB0d28sIGFuZCBhbGwgbnVtYmVycyBhcmUgaW50ZWdlcnMuIFJldHVybnMgZmFsc2UgaW4gYW55Cm90aGVyIGNhc2VzLgoKRXhhbXBsZXM6CmFueV9pbnQoNSwgMiwgNykgLT4gVHJ1ZQphbnlfaW50KDMsIDIsIDIpIC0+IEZhbHNlCmFueV9pbnQoMywgLTIsIDEpIC0+IFRydWUKYW55X2ludCgzLjYsIC0yLjIsIDIpIC0+IEZhbHNlICcnJw==" download="">⬇</a></span>
<span id="lstnumberx40" class="ltx_listingline"><span id="lstnumberx40.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx40.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">any_int</span><span id="lstnumberx40.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx40.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x</span><span id="lstnumberx40.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx40.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">y</span><span id="lstnumberx40.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx40.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">z</span><span id="lstnumberx40.12" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx41" class="ltx_listingline"><span id="lstnumberx41.1" class="ltx_text ltx_font_typewriter">’’’</span><span id="lstnumberx41.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Create</span><span id="lstnumberx41.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx41.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx41.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">that</span><span id="lstnumberx41.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">takes</span><span id="lstnumberx41.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.13" class="ltx_text ltx_font_typewriter">3</span><span id="lstnumberx41.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">numbers</span><span id="lstnumberx41.16" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx42" class="ltx_listingline"><span id="lstnumberx42.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Returns</span><span id="lstnumberx42.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">true</span><span id="lstnumberx42.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx42.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">one</span><span id="lstnumberx42.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx42.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx42.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">numbers</span><span id="lstnumberx42.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx42.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">equal</span><span id="lstnumberx42.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx42.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx42.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sum</span><span id="lstnumberx42.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx42.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span>
</span>
<span id="lstnumberx43" class="ltx_listingline"><span id="lstnumberx43.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">other</span><span id="lstnumberx43.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">two</span><span id="lstnumberx43.4" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx43.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx43.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">all</span><span id="lstnumberx43.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">numbers</span><span id="lstnumberx43.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx43.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">integers</span><span id="lstnumberx43.15" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx43.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Returns</span><span id="lstnumberx43.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">false</span><span id="lstnumberx43.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx43.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">any</span>
</span>
<span id="lstnumberx44" class="ltx_listingline"><span id="lstnumberx44.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">other</span><span id="lstnumberx44.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx44.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cases</span><span id="lstnumberx44.4" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx45" class="ltx_listingline">
</span>
<span id="lstnumberx46" class="ltx_listingline"><span id="lstnumberx46.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Examples</span><span id="lstnumberx46.2" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx47" class="ltx_listingline"><span id="lstnumberx47.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">any_int</span><span id="lstnumberx47.2" class="ltx_text ltx_font_typewriter">(5,</span><span id="lstnumberx47.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.4" class="ltx_text ltx_font_typewriter">2,</span><span id="lstnumberx47.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.6" class="ltx_text ltx_font_typewriter">7)</span><span id="lstnumberx47.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.8" class="ltx_text ltx_font_typewriter">-&gt;</span><span id="lstnumberx47.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">True</span>
</span>
<span id="lstnumberx48" class="ltx_listingline"><span id="lstnumberx48.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">any_int</span><span id="lstnumberx48.2" class="ltx_text ltx_font_typewriter">(3,</span><span id="lstnumberx48.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx48.4" class="ltx_text ltx_font_typewriter">2,</span><span id="lstnumberx48.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx48.6" class="ltx_text ltx_font_typewriter">2)</span><span id="lstnumberx48.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx48.8" class="ltx_text ltx_font_typewriter">-&gt;</span><span id="lstnumberx48.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx48.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">False</span>
</span>
<span id="lstnumberx49" class="ltx_listingline"><span id="lstnumberx49.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">any_int</span><span id="lstnumberx49.2" class="ltx_text ltx_font_typewriter">(3,</span><span id="lstnumberx49.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx49.4" class="ltx_text ltx_font_typewriter">-2,</span><span id="lstnumberx49.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx49.6" class="ltx_text ltx_font_typewriter">1)</span><span id="lstnumberx49.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx49.8" class="ltx_text ltx_font_typewriter">-&gt;</span><span id="lstnumberx49.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx49.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">True</span>
</span>
<span id="lstnumberx50" class="ltx_listingline"><span id="lstnumberx50.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">any_int</span><span id="lstnumberx50.2" class="ltx_text ltx_font_typewriter">(3.6,</span><span id="lstnumberx50.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.4" class="ltx_text ltx_font_typewriter">-2.2,</span><span id="lstnumberx50.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.6" class="ltx_text ltx_font_typewriter">2)</span><span id="lstnumberx50.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.8" class="ltx_text ltx_font_typewriter">-&gt;</span><span id="lstnumberx50.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">False</span><span id="lstnumberx50.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.12" class="ltx_text ltx_font_typewriter">’’’</span>
</span>
</span></span>
</span>
</td>
<td id="Ax1.1.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r ltx_border_t">
<span id="Ax1.1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.4.3.2.1.1" class="ltx_p" style="width:227.6pt;">continue this code.

<span id="Ax1.1.1.4.3.2.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGFueV9pbnQoeCwgeSwgeik6CiIiIkNyZWF0ZSBhIGZ1bmN0aW9uIHRoYXQgdGFrZXMgMyBudW1iZXJzLiBSZXR1cm4gdHJ1ZSBp" download="">⬇</a></span>
<span id="lstnumberx51" class="ltx_listingline"><span id="lstnumberx51.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx51.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">any_int</span><span id="lstnumberx51.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx51.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x</span><span id="lstnumberx51.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx51.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">y</span><span id="lstnumberx51.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx51.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">z</span><span id="lstnumberx51.12" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx52" class="ltx_listingline"><span id="lstnumberx52.1" class="ltx_text ltx_font_typewriter">"""</span><span id="lstnumberx52.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Create</span><span id="lstnumberx52.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx52.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx52.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">that</span><span id="lstnumberx52.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">takes</span><span id="lstnumberx52.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.12" class="ltx_text ltx_font_typewriter">3</span><span id="lstnumberx52.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">numbers</span><span id="lstnumberx52.15" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx52.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Return</span><span id="lstnumberx52.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">true</span><span id="lstnumberx52.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">i</span>
</span>
</span></span>
</span>
</td>
<td id="Ax1.1.1.4.3.3" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="Ax1.1.1.4.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.4.3.3.1.1" class="ltx_p" style="width:227.6pt;">
<span id="Ax1.1.1.4.3.3.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGFueV9pbnQoeCwgeSwgeik6CiIiIgpDcmVhdGUgYSBmdW5jdGlvbiB0aGF0IHRha2VzIDMgbnVtYmVycy4KUmV0dXJucyB0cnVlIGlmIG9uZSBvZiB0aGUgbnVtYmVycyBpcyBlcXVhbCB0byB0aGUgc3VtIG9mIHRoZQpvdGhlciB0d28gbnVtYmVycy4gT3RoZXJ3aXNlIHJldHVybnMgZmFsc2UuCiIiIgpbLi4uXQ==" download="">⬇</a></span>
<span id="lstnumberx53" class="ltx_listingline"><span id="lstnumberx53.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx53.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">any_int</span><span id="lstnumberx53.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx53.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x</span><span id="lstnumberx53.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx53.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">y</span><span id="lstnumberx53.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx53.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">z</span><span id="lstnumberx53.12" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx54" class="ltx_listingline"><span id="lstnumberx54.1" class="ltx_text ltx_font_typewriter">"""</span>
</span>
<span id="lstnumberx55" class="ltx_listingline"><span id="lstnumberx55.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Create</span><span id="lstnumberx55.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx55.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx55.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx55.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx55.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx55.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">that</span><span id="lstnumberx55.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx55.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">takes</span><span id="lstnumberx55.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx55.11" class="ltx_text ltx_font_typewriter">3</span><span id="lstnumberx55.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx55.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">numbers</span><span id="lstnumberx55.14" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx56" class="ltx_listingline"><span id="lstnumberx56.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Returns</span><span id="lstnumberx56.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">true</span><span id="lstnumberx56.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx56.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">one</span><span id="lstnumberx56.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx56.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx56.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">numbers</span><span id="lstnumberx56.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx56.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">equal</span><span id="lstnumberx56.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx56.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx56.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sum</span><span id="lstnumberx56.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx56.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span>
</span>
<span id="lstnumberx57" class="ltx_listingline"><span id="lstnumberx57.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">other</span><span id="lstnumberx57.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">two</span><span id="lstnumberx57.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">numbers</span><span id="lstnumberx57.6" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx57.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Otherwise</span><span id="lstnumberx57.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">returns</span><span id="lstnumberx57.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">false</span><span id="lstnumberx57.13" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx58" class="ltx_listingline"><span id="lstnumberx58.1" class="ltx_text ltx_font_typewriter">"""</span>
</span>
<span id="lstnumberx59" class="ltx_listingline"><span id="lstnumberx59.1" class="ltx_text ltx_font_typewriter">[...]</span>
</span>
</span></span>
</span>
</td>
</tr>
<tr id="Ax1.1.1.5.4" class="ltx_tr">
<td id="Ax1.1.1.5.4.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t">
<span id="Ax1.1.1.5.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.5.4.1.1.1" class="ltx_p" style="width:227.6pt;">
<span id="Ax1.1.1.5.4.1.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIG9yZGVyX2J5X3BvaW50cyhudW1zKToKIiIiIFdyaXRlIGEgZnVuY3Rpb24gd2hpY2ggc29ydHMgdGhlIGdpdmVuIGxpc3Qgb2YgaW50ZWdlcnMKaW4gYXNjZW5kaW5nIG9yZGVyIGFjY29yZGluZyB0byB0aGUgc3VtIG9mIHRoZWlyIGRpZ2l0cy4KCk5vdGU6IGlmIHRoZXJlIGFyZSBzZXZlcmFsIGl0ZW1zIHdpdGggc2ltaWxhciBzdW0gb2YgdGhlaXIKZGlnaXRzLCBvcmRlciB0aGVtIGJhc2VkIG9uIHRoZWlyIGluZGV4IGluIG9yaWdpbmFsIGxpc3QuCgpGb3IgZXhhbXBsZToKCm9yZGVyX2J5X3BvaW50cyhbMSwgMTEsIC0xLCAtMTEsIC0xMl0pID09IFstMSwgLTExLCAxLCAtMTIsIDExXQpvcmRlcl9ieV9wb2ludHMoW10pID09IFtdCiIiIg==" download="">⬇</a></span>
<span id="lstnumberx60" class="ltx_listingline"><span id="lstnumberx60.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx60.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx60.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">order_by_points</span><span id="lstnumberx60.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx60.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nums</span><span id="lstnumberx60.6" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx61" class="ltx_listingline"><span id="lstnumberx61.1" class="ltx_text ltx_font_typewriter">"""</span><span id="lstnumberx61.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Write</span><span id="lstnumberx61.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx61.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx61.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">which</span><span id="lstnumberx61.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sorts</span><span id="lstnumberx61.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx61.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx61.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx61.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx61.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">integers</span>
</span>
<span id="lstnumberx62" class="ltx_listingline"><span id="lstnumberx62.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx62.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ascending</span><span id="lstnumberx62.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">order</span><span id="lstnumberx62.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">according</span><span id="lstnumberx62.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx62.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx62.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sum</span><span id="lstnumberx62.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx62.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">their</span><span id="lstnumberx62.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">digits</span><span id="lstnumberx62.20" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx63" class="ltx_listingline">
</span>
<span id="lstnumberx64" class="ltx_listingline"><span id="lstnumberx64.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Note</span><span id="lstnumberx64.2" class="ltx_text ltx_font_typewriter">:</span><span id="lstnumberx64.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx64.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">there</span><span id="lstnumberx64.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx64.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">several</span><span id="lstnumberx64.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">items</span><span id="lstnumberx64.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx64.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">similar</span><span id="lstnumberx64.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sum</span><span id="lstnumberx64.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx64.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">their</span>
</span>
<span id="lstnumberx65" class="ltx_listingline"><span id="lstnumberx65.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">digits</span><span id="lstnumberx65.2" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx65.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">order</span><span id="lstnumberx65.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">them</span><span id="lstnumberx65.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">based</span><span id="lstnumberx65.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">on</span><span id="lstnumberx65.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">their</span><span id="lstnumberx65.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">index</span><span id="lstnumberx65.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx65.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">original</span><span id="lstnumberx65.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx65.21" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx66" class="ltx_listingline">
</span>
<span id="lstnumberx67" class="ltx_listingline"><span id="lstnumberx67.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">For</span><span id="lstnumberx67.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx67.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">example</span><span id="lstnumberx67.4" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx68" class="ltx_listingline">
</span>
<span id="lstnumberx69" class="ltx_listingline"><span id="lstnumberx69.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">order_by_points</span><span id="lstnumberx69.2" class="ltx_text ltx_font_typewriter">([1,</span><span id="lstnumberx69.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.4" class="ltx_text ltx_font_typewriter">11,</span><span id="lstnumberx69.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.6" class="ltx_text ltx_font_typewriter">-1,</span><span id="lstnumberx69.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.8" class="ltx_text ltx_font_typewriter">-11,</span><span id="lstnumberx69.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.10" class="ltx_text ltx_font_typewriter">-12])</span><span id="lstnumberx69.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.12" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx69.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.14" class="ltx_text ltx_font_typewriter">[-1,</span><span id="lstnumberx69.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.16" class="ltx_text ltx_font_typewriter">-11,</span><span id="lstnumberx69.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.18" class="ltx_text ltx_font_typewriter">1,</span><span id="lstnumberx69.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.20" class="ltx_text ltx_font_typewriter">-12,</span><span id="lstnumberx69.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.22" class="ltx_text ltx_font_typewriter">11]</span>
</span>
<span id="lstnumberx70" class="ltx_listingline"><span id="lstnumberx70.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">order_by_points</span><span id="lstnumberx70.2" class="ltx_text ltx_font_typewriter">([])</span><span id="lstnumberx70.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.4" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx70.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.6" class="ltx_text ltx_font_typewriter">[]</span>
</span>
<span id="lstnumberx71" class="ltx_listingline"><span id="lstnumberx71.1" class="ltx_text ltx_font_typewriter">"""</span>
</span>
</span></span>
</span>
</td>
<td id="Ax1.1.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_r ltx_border_t">
<span id="Ax1.1.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.5.4.2.1.1" class="ltx_p" style="width:227.6pt;">You will be given an instruction to complete a python function. This instruction is cut in half, so you need to complete it. No need of generating the final code, only the instruction is required.</span>
<span id="Ax1.1.1.5.4.2.1.2" class="ltx_p">The incomplete instruction is:</span>
<span id="Ax1.1.1.5.4.2.1.3" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIG9yZGVyX2J5X3BvaW50cyhudW1zKToKIiIiIFdyaXRlIGEgZnVuY3Rpb24gd2hpY2ggc29ydHMgdGhlIGdpdmVuIGxpc3Qgb2YgaW50ZWdlcnMgaW4KYXNjZW5kaW5nIG9yZGVyIGFjY29yZGluZw==" download="">⬇</a></span>
<span id="lstnumberx72" class="ltx_listingline"><span id="lstnumberx72.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx72.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx72.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">order_by_points</span><span id="lstnumberx72.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx72.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nums</span><span id="lstnumberx72.6" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx73" class="ltx_listingline"><span id="lstnumberx73.1" class="ltx_text ltx_font_typewriter">"""</span><span id="lstnumberx73.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Write</span><span id="lstnumberx73.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx73.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx73.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">which</span><span id="lstnumberx73.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sorts</span><span id="lstnumberx73.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx73.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx73.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx73.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx73.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">integers</span><span id="lstnumberx73.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span>
</span>
<span id="lstnumberx74" class="ltx_listingline"><span id="lstnumberx74.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ascending</span><span id="lstnumberx74.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">order</span><span id="lstnumberx74.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">according</span>
</span>
</span>
</span>
</td>
<td id="Ax1.1.1.5.4.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t">
<span id="Ax1.1.1.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.1.5.4.3.1.1" class="ltx_p" style="width:227.6pt;">Here’s the completed instruction for the Python function:</span>
<span id="Ax1.1.1.5.4.3.1.2" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIG9yZGVyX2J5X3BvaW50cyhudW1zKToKIiIiIFdyaXRlIGEgZnVuY3Rpb24gd2hpY2ggc29ydHMgdGhlIGdpdmVuIGxpc3Qgb2YgaW50ZWdlcnMgaW4KYXNjZW5kaW5nIG9yZGVyIGFjY29yZGluZyB0byB0aGUgc3VtIG9mIHRoZWlyIGRpZ2l0cy4KSWYgdHdvIGludGVnZXJzIGhhdmUgdGhlIHNhbWUgc3VtIG9mIHRoZWlyIGRpZ2l0cywgb3JkZXIgdGhlbQpiYXNlZCBvbiB0aGVpciBpbmRleCBpbiB0aGUgb3JpZ2luYWwgbGlzdC4KIiIi" download="">⬇</a></span>
<span id="lstnumberx75" class="ltx_listingline"><span id="lstnumberx75.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx75.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">order_by_points</span><span id="lstnumberx75.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx75.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nums</span><span id="lstnumberx75.6" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx76" class="ltx_listingline"><span id="lstnumberx76.1" class="ltx_text ltx_font_typewriter">"""</span><span id="lstnumberx76.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Write</span><span id="lstnumberx76.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx76.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx76.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">which</span><span id="lstnumberx76.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sorts</span><span id="lstnumberx76.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx76.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx76.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx76.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx76.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">integers</span><span id="lstnumberx76.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span>
</span>
<span id="lstnumberx77" class="ltx_listingline"><span id="lstnumberx77.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ascending</span><span id="lstnumberx77.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx77.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">order</span><span id="lstnumberx77.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx77.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">according</span><span id="lstnumberx77.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx77.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx77.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx77.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx77.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx77.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sum</span><span id="lstnumberx77.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx77.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx77.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx77.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">their</span><span id="lstnumberx77.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx77.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">digits</span><span id="lstnumberx77.18" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx78" class="ltx_listingline"><span id="lstnumberx78.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">If</span><span id="lstnumberx78.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">two</span><span id="lstnumberx78.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">integers</span><span id="lstnumberx78.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">have</span><span id="lstnumberx78.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx78.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">same</span><span id="lstnumberx78.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sum</span><span id="lstnumberx78.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx78.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">their</span><span id="lstnumberx78.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">digits</span><span id="lstnumberx78.20" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx78.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">order</span><span id="lstnumberx78.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">them</span>
</span>
<span id="lstnumberx79" class="ltx_listingline"><span id="lstnumberx79.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">based</span><span id="lstnumberx79.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx79.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">on</span><span id="lstnumberx79.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx79.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">their</span><span id="lstnumberx79.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx79.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">index</span><span id="lstnumberx79.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx79.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx79.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx79.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx79.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx79.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">original</span><span id="lstnumberx79.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx79.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx79.16" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx80" class="ltx_listingline"><span id="lstnumberx80.1" class="ltx_text ltx_font_typewriter">"""</span>
</span>
</span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="Ax1.1.tab1" class="ltx_table ltx_figure_panel">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 3: </span>Most similar prompt in <span id="Ax1.1.tab1.3.1" class="ltx_text ltx_font_typewriter">evol-instruct</span> for a non-handpicked selection of <span id="Ax1.1.tab1.4.2" class="ltx_text ltx_font_typewriter">HumanEval</span> prompts.</figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<table id="Ax1.1.tab1.5" class="ltx_tabular ltx_figure_panel ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="Ax1.1.tab1.5.1.1" class="ltx_tr">
<th id="Ax1.1.tab1.5.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_r ltx_border_tt">
<span id="Ax1.1.tab1.5.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.tab1.5.1.1.1.1.1" class="ltx_p" style="width:341.4pt;"><span id="Ax1.1.tab1.5.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter ltx_font_bold">HumanEval<span id="Ax1.1.tab1.5.1.1.1.1.1.1.1" class="ltx_text ltx_font_serif"> prompt</span></span></span>
</span>
</th>
<th id="Ax1.1.tab1.5.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_tt">
<span id="Ax1.1.tab1.5.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.tab1.5.1.1.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="Ax1.1.tab1.5.1.1.2.1.1.1" class="ltx_text ltx_font_bold">Closest <span id="Ax1.1.tab1.5.1.1.2.1.1.1.1" class="ltx_text ltx_font_typewriter">evol-instrct</span> prompt</span></span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="Ax1.1.tab1.5.2.1" class="ltx_tr">
<td id="Ax1.1.tab1.5.2.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="Ax1.1.tab1.5.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.tab1.5.2.1.1.1.1" class="ltx_p" style="width:341.4pt;">
<span id="Ax1.1.tab1.5.2.1.1.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,V3JpdGUgYSBQeXRob24gZnVuY3Rpb24gYGxvbmdlc3Qoc3RyaW5nczogTGlzdFtzdHJdKSAtPiBPcHRpb25hbFtzdHJdYCB0byBzb2x2ZSB0aGUgZm9sbG93aW5nCnByb2JsZW06Ck91dCBvZiBsaXN0IG9mIHN0cmluZ3MsIHJldHVybiB0aGUgbG9uZ2VzdCBvbmUuIFJldHVybiB0aGUgZmlyc3Qgb25lIGluIGNhc2Ugb2YgbXVsdGlwbGUKc3RyaW5ncyBvZiB0aGUgc2FtZSBsZW5ndGguIFJldHVybiBOb25lIGluIGNhc2UgdGhlIGlucHV0IGxpc3QgaXMgZW1wdHkuCj4+PiBsb25nZXN0KFtdKQo+Pj4gbG9uZ2VzdChbJ2EnLCAnYicsICdjJ10pCidhJwo+Pj4gbG9uZ2VzdChbJ2EnLCAnYmInLCAnY2NjJ10pCidjY2Mn" download="">⬇</a></span>
<span id="lstnumberx81" class="ltx_listingline"><span id="lstnumberx81.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Write</span><span id="lstnumberx81.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx81.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Python</span><span id="lstnumberx81.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx81.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.9" class="ltx_text ltx_font_typewriter">‘</span><span id="lstnumberx81.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longest</span><span id="lstnumberx81.11" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx81.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">strings</span><span id="lstnumberx81.13" class="ltx_text ltx_font_typewriter">:</span><span id="lstnumberx81.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">List</span><span id="lstnumberx81.16" class="ltx_text ltx_font_typewriter">[</span><span id="lstnumberx81.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">str</span><span id="lstnumberx81.18" class="ltx_text ltx_font_typewriter">])</span><span id="lstnumberx81.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.20" class="ltx_text ltx_font_typewriter">-&gt;</span><span id="lstnumberx81.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Optional</span><span id="lstnumberx81.23" class="ltx_text ltx_font_typewriter">[</span><span id="lstnumberx81.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">str</span><span id="lstnumberx81.25" class="ltx_text ltx_font_typewriter">]‘</span><span id="lstnumberx81.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx81.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">solve</span><span id="lstnumberx81.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx81.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span>
</span>
<span id="lstnumberx82" class="ltx_listingline"><span id="lstnumberx82.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">problem</span><span id="lstnumberx82.2" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx83" class="ltx_listingline"><span id="lstnumberx83.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Out</span><span id="lstnumberx83.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx83.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx83.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx83.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">strings</span><span id="lstnumberx83.10" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx83.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">return</span><span id="lstnumberx83.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx83.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longest</span><span id="lstnumberx83.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">one</span><span id="lstnumberx83.19" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx83.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Return</span><span id="lstnumberx83.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx83.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">first</span><span id="lstnumberx83.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">one</span><span id="lstnumberx83.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx83.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">case</span><span id="lstnumberx83.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx83.34" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter">multiple</span>
</span>
<span id="lstnumberx84" class="ltx_listingline"><span id="lstnumberx84.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">strings</span><span id="lstnumberx84.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx84.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx84.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">same</span><span id="lstnumberx84.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">length</span><span id="lstnumberx84.10" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx84.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Return</span><span id="lstnumberx84.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span><span id="lstnumberx84.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx84.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">case</span><span id="lstnumberx84.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx84.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">input</span><span id="lstnumberx84.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx84.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx84.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">empty</span><span id="lstnumberx84.29" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx85" class="ltx_listingline"><span id="lstnumberx85.1" class="ltx_text ltx_font_typewriter">&gt;&gt;&gt;</span><span id="lstnumberx85.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx85.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longest</span><span id="lstnumberx85.4" class="ltx_text ltx_font_typewriter">([])</span>
</span>
<span id="lstnumberx86" class="ltx_listingline"><span id="lstnumberx86.1" class="ltx_text ltx_font_typewriter">&gt;&gt;&gt;</span><span id="lstnumberx86.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx86.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longest</span><span id="lstnumberx86.4" class="ltx_text ltx_font_typewriter">([’</span><span id="lstnumberx86.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx86.6" class="ltx_text ltx_font_typewriter">’,</span><span id="lstnumberx86.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx86.8" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx86.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">b</span><span id="lstnumberx86.10" class="ltx_text ltx_font_typewriter">’,</span><span id="lstnumberx86.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx86.12" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx86.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">c</span><span id="lstnumberx86.14" class="ltx_text ltx_font_typewriter">’])</span>
</span>
<span id="lstnumberx87" class="ltx_listingline"><span id="lstnumberx87.1" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx87.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx87.3" class="ltx_text ltx_font_typewriter">’</span>
</span>
<span id="lstnumberx88" class="ltx_listingline"><span id="lstnumberx88.1" class="ltx_text ltx_font_typewriter">&gt;&gt;&gt;</span><span id="lstnumberx88.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx88.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longest</span><span id="lstnumberx88.4" class="ltx_text ltx_font_typewriter">([’</span><span id="lstnumberx88.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx88.6" class="ltx_text ltx_font_typewriter">’,</span><span id="lstnumberx88.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx88.8" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx88.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">bb</span><span id="lstnumberx88.10" class="ltx_text ltx_font_typewriter">’,</span><span id="lstnumberx88.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx88.12" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx88.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ccc</span><span id="lstnumberx88.14" class="ltx_text ltx_font_typewriter">’])</span>
</span>
<span id="lstnumberx89" class="ltx_listingline"><span id="lstnumberx89.1" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx89.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ccc</span><span id="lstnumberx89.3" class="ltx_text ltx_font_typewriter">’</span>
</span>
</span></span>
</span>
</td>
<td id="Ax1.1.tab1.5.2.1.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="Ax1.1.tab1.5.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.tab1.5.2.1.2.1.1" class="ltx_p" style="width:341.4pt;">
<span id="Ax1.1.tab1.5.2.1.2.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,Q29tcGxldGUgdGhlIGNvZGUgYmVsb3csIGNvbnNpZGVyaW5nIGFuIGF1Z21lbnRlZCBkZXB0aCBvZiBpbnF1aXJ5IGFuZCBtYWludGFpbmluZyBicmV2aXR5Ogpmcm9tIHR5cGluZyBpbXBvcnQgTGlzdCwgT3B0aW9uYWwKCgpkZWYgbG9uZ2VzdChzdHJpbmdzOiBMaXN0W3N0cl0pIC0+IE9wdGlvbmFsW3N0cl06CiAgICAiIiIgRnJvbSBhIGxpc3Qgb2Ygc3RyaW5ncywgcmV0dXJuIHRoZSBsb25nZXN0IG9uZS4gRm9yIG11bHRpcGxlIHN0cmluZ3Mgd2l0aCBlcXVhbAogICAgbGVuZ3RoLCByZXR1cm4gdGhlIGZpcnN0LiBGb3IgYW4gZW1wdHkgbGlzdCwgcmV0dXJuIE5vbmUuCiAgICA+Pj4gbG9uZ2VzdChbXSkKCiAgICA+Pj4gbG9uZ2VzdChbJ2EnLCAnYicsICdjJ10pCiAgICAnYScKICAgID4+PiBsb25nZXN0KFsnYScsICdiYicsICdjY2MnXSkKICAgICdjY2Mn" download="">⬇</a></span>
<span id="lstnumberx90" class="ltx_listingline"><span id="lstnumberx90.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Complete</span><span id="lstnumberx90.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx90.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">code</span><span id="lstnumberx90.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">below</span><span id="lstnumberx90.8" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx90.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">considering</span><span id="lstnumberx90.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">an</span><span id="lstnumberx90.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">augmented</span><span id="lstnumberx90.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">depth</span><span id="lstnumberx90.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx90.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inquiry</span><span id="lstnumberx90.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx90.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">maintaining</span><span id="lstnumberx90.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">brevity</span><span id="lstnumberx90.27" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx91" class="ltx_listingline"><span id="lstnumberx91.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">from</span><span id="lstnumberx91.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx91.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">typing</span><span id="lstnumberx91.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx91.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">import</span><span id="lstnumberx91.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx91.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">List</span><span id="lstnumberx91.8" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx91.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx91.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Optional</span>
</span>
<span id="lstnumberx92" class="ltx_listingline">
</span>
<span id="lstnumberx93" class="ltx_listingline">
</span>
<span id="lstnumberx94" class="ltx_listingline"><span id="lstnumberx94.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx94.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longest</span><span id="lstnumberx94.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx94.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">strings</span><span id="lstnumberx94.6" class="ltx_text ltx_font_typewriter">:</span><span id="lstnumberx94.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">List</span><span id="lstnumberx94.9" class="ltx_text ltx_font_typewriter">[</span><span id="lstnumberx94.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">str</span><span id="lstnumberx94.11" class="ltx_text ltx_font_typewriter">])</span><span id="lstnumberx94.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.13" class="ltx_text ltx_font_typewriter">-&gt;</span><span id="lstnumberx94.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Optional</span><span id="lstnumberx94.16" class="ltx_text ltx_font_typewriter">[</span><span id="lstnumberx94.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">str</span><span id="lstnumberx94.18" class="ltx_text ltx_font_typewriter">]:</span>
</span>
<span id="lstnumberx95" class="ltx_listingline"><span id="lstnumberx95.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx95.2" class="ltx_text ltx_font_typewriter">"""</span><span id="lstnumberx95.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">From</span><span id="lstnumberx95.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx95.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx95.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx95.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">strings</span><span id="lstnumberx95.13" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx95.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">return</span><span id="lstnumberx95.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx95.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longest</span><span id="lstnumberx95.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">one</span><span id="lstnumberx95.22" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx95.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">For</span><span id="lstnumberx95.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">multiple</span><span id="lstnumberx95.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">strings</span><span id="lstnumberx95.29" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx95.31" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.32" class="ltx_text ltx_lst_identifier ltx_font_typewriter">equal</span>
</span>
<span id="lstnumberx96" class="ltx_listingline"><span id="lstnumberx96.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx96.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">length</span><span id="lstnumberx96.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx96.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">return</span><span id="lstnumberx96.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx96.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">first</span><span id="lstnumberx96.10" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx96.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">For</span><span id="lstnumberx96.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">an</span><span id="lstnumberx96.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">empty</span><span id="lstnumberx96.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx96.19" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx96.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">return</span><span id="lstnumberx96.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span><span id="lstnumberx96.24" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx97" class="ltx_listingline"><span id="lstnumberx97.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx97.2" class="ltx_text ltx_font_typewriter">&gt;&gt;&gt;</span><span id="lstnumberx97.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longest</span><span id="lstnumberx97.5" class="ltx_text ltx_font_typewriter">([])</span>
</span>
<span id="lstnumberx98" class="ltx_listingline">
</span>
<span id="lstnumberx99" class="ltx_listingline"><span id="lstnumberx99.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx99.2" class="ltx_text ltx_font_typewriter">&gt;&gt;&gt;</span><span id="lstnumberx99.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longest</span><span id="lstnumberx99.5" class="ltx_text ltx_font_typewriter">([’</span><span id="lstnumberx99.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx99.7" class="ltx_text ltx_font_typewriter">’,</span><span id="lstnumberx99.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.9" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx99.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">b</span><span id="lstnumberx99.11" class="ltx_text ltx_font_typewriter">’,</span><span id="lstnumberx99.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.13" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx99.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">c</span><span id="lstnumberx99.15" class="ltx_text ltx_font_typewriter">’])</span>
</span>
<span id="lstnumberx100" class="ltx_listingline"><span id="lstnumberx100.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx100.2" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx100.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx100.4" class="ltx_text ltx_font_typewriter">’</span>
</span>
<span id="lstnumberx101" class="ltx_listingline"><span id="lstnumberx101.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx101.2" class="ltx_text ltx_font_typewriter">&gt;&gt;&gt;</span><span id="lstnumberx101.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx101.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longest</span><span id="lstnumberx101.5" class="ltx_text ltx_font_typewriter">([’</span><span id="lstnumberx101.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx101.7" class="ltx_text ltx_font_typewriter">’,</span><span id="lstnumberx101.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx101.9" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx101.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">bb</span><span id="lstnumberx101.11" class="ltx_text ltx_font_typewriter">’,</span><span id="lstnumberx101.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx101.13" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx101.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ccc</span><span id="lstnumberx101.15" class="ltx_text ltx_font_typewriter">’])</span>
</span>
<span id="lstnumberx102" class="ltx_listingline"><span id="lstnumberx102.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx102.2" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx102.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ccc</span><span id="lstnumberx102.4" class="ltx_text ltx_font_typewriter">’</span>
</span>
</span></span>
</span>
</td>
</tr>
<tr id="Ax1.1.tab1.5.3.2" class="ltx_tr">
<td id="Ax1.1.tab1.5.3.2.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="Ax1.1.tab1.5.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.tab1.5.3.2.1.1.1" class="ltx_p" style="width:341.4pt;">
<span id="Ax1.1.tab1.5.3.2.1.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,V3JpdGUgYSBQeXRob24gZnVuY3Rpb24gYG1ha2VfYV9waWxlKG4pYCB0byBzb2x2ZSB0aGUgZm9sbG93aW5nIHByb2JsZW06CkdpdmVuIGEgcG9zaXRpdmUgaW50ZWdlciBuLCB5b3UgaGF2ZSB0byBtYWtlIGEgcGlsZSBvZiBuIGxldmVscyBvZiBzdG9uZXMuClRoZSBmaXJzdCBsZXZlbCBoYXMgbiBzdG9uZXMuClRoZSBudW1iZXIgb2Ygc3RvbmVzIGluIHRoZSBuZXh0IGxldmVsIGlzOgotIHRoZSBuZXh0IG9kZCBudW1iZXIgaWYgbiBpcyBvZGQuCi0gdGhlIG5leHQgZXZlbiBudW1iZXIgaWYgbiBpcyBldmVuLgpSZXR1cm4gdGhlIG51bWJlciBvZiBzdG9uZXMgaW4gZWFjaCBsZXZlbCBpbiBhIGxpc3QsIHdoZXJlIGVsZW1lbnQgYXQgaW5kZXgKaSByZXByZXNlbnRzIHRoZSBudW1iZXIgb2Ygc3RvbmVzIGluIHRoZSBsZXZlbCAoaSsxKS4KRXhhbXBsZXM6Cj4+PiBtYWtlX2FfcGlsZSgzKQpbMywgNSwgN10=" download="">⬇</a></span>
<span id="lstnumberx103" class="ltx_listingline"><span id="lstnumberx103.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Write</span><span id="lstnumberx103.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx103.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Python</span><span id="lstnumberx103.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx103.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.9" class="ltx_text ltx_font_typewriter">‘</span><span id="lstnumberx103.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">make_a_pile</span><span id="lstnumberx103.11" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx103.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx103.13" class="ltx_text ltx_font_typewriter">)‘</span><span id="lstnumberx103.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx103.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">solve</span><span id="lstnumberx103.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx103.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx103.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">problem</span><span id="lstnumberx103.24" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx104" class="ltx_listingline"><span id="lstnumberx104.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Given</span><span id="lstnumberx104.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx104.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">positive</span><span id="lstnumberx104.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">integer</span><span id="lstnumberx104.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx104.10" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx104.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">you</span><span id="lstnumberx104.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">have</span><span id="lstnumberx104.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx104.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">make</span><span id="lstnumberx104.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx104.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">pile</span><span id="lstnumberx104.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx104.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx104.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">levels</span><span id="lstnumberx104.29" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx104.31" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.32" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx104.33" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx105" class="ltx_listingline"><span id="lstnumberx105.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">The</span><span id="lstnumberx105.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx105.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">first</span><span id="lstnumberx105.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx105.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">level</span><span id="lstnumberx105.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx105.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">has</span><span id="lstnumberx105.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx105.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx105.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx105.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx105.12" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx106" class="ltx_listingline"><span id="lstnumberx106.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">The</span><span id="lstnumberx106.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx106.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx106.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx106.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx106.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx106.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx106.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx106.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx106.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx106.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx106.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx106.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx106.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx106.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">level</span><span id="lstnumberx106.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx106.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx106.18" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx107" class="ltx_listingline"><span id="lstnumberx107.1" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx107.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx107.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx107.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">odd</span><span id="lstnumberx107.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx107.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx107.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx107.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx107.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">odd</span><span id="lstnumberx107.18" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx108" class="ltx_listingline"><span id="lstnumberx108.1" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx108.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx108.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx108.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">even</span><span id="lstnumberx108.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx108.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx108.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx108.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx108.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">even</span><span id="lstnumberx108.18" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx109" class="ltx_listingline"><span id="lstnumberx109.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Return</span><span id="lstnumberx109.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx109.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx109.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx109.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx109.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx109.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">each</span><span id="lstnumberx109.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">level</span><span id="lstnumberx109.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx109.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx109.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx109.22" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx109.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">where</span><span id="lstnumberx109.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">element</span><span id="lstnumberx109.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">at</span><span id="lstnumberx109.29" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter">index</span>
</span>
<span id="lstnumberx110" class="ltx_listingline"><span id="lstnumberx110.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">i</span><span id="lstnumberx110.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">represents</span><span id="lstnumberx110.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx110.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx110.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx110.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx110.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx110.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx110.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">level</span><span id="lstnumberx110.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.19" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx110.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">i</span><span id="lstnumberx110.21" class="ltx_text ltx_font_typewriter">+1).</span>
</span>
<span id="lstnumberx111" class="ltx_listingline"><span id="lstnumberx111.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Examples</span><span id="lstnumberx111.2" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx112" class="ltx_listingline"><span id="lstnumberx112.1" class="ltx_text ltx_font_typewriter">&gt;&gt;&gt;</span><span id="lstnumberx112.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx112.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">make_a_pile</span><span id="lstnumberx112.4" class="ltx_text ltx_font_typewriter">(3)</span>
</span>
<span id="lstnumberx113" class="ltx_listingline"><span id="lstnumberx113.1" class="ltx_text ltx_font_typewriter">[3,</span><span id="lstnumberx113.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx113.3" class="ltx_text ltx_font_typewriter">5,</span><span id="lstnumberx113.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx113.5" class="ltx_text ltx_font_typewriter">7]</span>
</span>
</span></span>
</span>
</td>
<td id="Ax1.1.tab1.5.3.2.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="Ax1.1.tab1.5.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.tab1.5.3.2.2.1.1" class="ltx_p" style="width:341.4pt;">
<span id="Ax1.1.tab1.5.3.2.2.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,UGxlYXNlIGNvbXBsZXRlIHRoZSBmb2xsb3dpbmcgY29kZSB3aXRoIGFkZGVkIGRpZmZpY3VsdHk6CgpkZWYgbWFrZV9hX3BpbGUobiwgcGF0dGVybik6CiAgICAiIiIKICAgIEdpdmVuIGEgcG9zaXRpdmUgaW50ZWdlciBuLCB5b3UgaGF2ZSB0byBtYWtlIGEgcGlsZSBvZiBuIGxldmVscyBvZiBzdG9uZXMuCiAgICBUaGUgZmlyc3QgbGV2ZWwgaGFzIG4gc3RvbmVzLiBUaGUgbnVtYmVyIG9mIHN0b25lcyBpbiB0aGUgbmV4dCBsZXZlbCBpcyBkZXRlcm1pbmVkCiAgICBieSB0aGUgZ2l2ZW4gcGF0dGVybiAnb2RkJyBvciAnZXZlbic6CiAgICAgICAgLSBJZiBwYXR0ZXJuIGlzICdvZGQnLCBhZGQgdGhlIG5leHQgb2RkIG51bWJlciB0byB0aGUgcHJldmlvdXMgbGV2ZWwgc3RvbmVzLgogICAgICAgIC0gSWYgcGF0dGVybiBpcyAnZXZlbicsIGFkZCB0aGUgbmV4dCBldmVuIG51bWJlciB0byB0aGUgcHJldmlvdXMgbGV2ZWwgc3RvbmVzLgogICAgUmV0dXJuIHRoZSBudW1iZXIgb2Ygc3RvbmVzIGluIGVhY2ggbGV2ZWwgaW4gYSBsaXN0LCB3aGVyZSBlbGVtZW50IGF0IGluZGV4CiAgICBpIHJlcHJlc2VudHMgdGhlIG51bWJlciBvZiBzdG9uZXMgaW4gdGhlIGxldmVsIChpKzEpLgoKICAgIEV4YW1wbGVzOgogICAgPj4+IG1ha2VfYV9waWxlKDMsICdvZGQnKQogICAgWzMsIDUsIDddCiAgICA+Pj4gbWFrZV9hX3BpbGUoMywgJ2V2ZW4nKQogICAgWzMsIDYsIDldCiAgICAiIiI=" download="">⬇</a></span>
<span id="lstnumberx114" class="ltx_listingline"><span id="lstnumberx114.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Please</span><span id="lstnumberx114.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx114.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">complete</span><span id="lstnumberx114.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx114.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx114.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx114.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx114.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx114.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">code</span><span id="lstnumberx114.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx114.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx114.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx114.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">added</span><span id="lstnumberx114.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx114.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">difficulty</span><span id="lstnumberx114.16" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx115" class="ltx_listingline">
</span>
<span id="lstnumberx116" class="ltx_listingline"><span id="lstnumberx116.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">def</span><span id="lstnumberx116.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx116.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">make_a_pile</span><span id="lstnumberx116.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx116.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx116.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx116.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx116.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">pattern</span><span id="lstnumberx116.9" class="ltx_text ltx_font_typewriter">):</span>
</span>
<span id="lstnumberx117" class="ltx_listingline"><span id="lstnumberx117.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx117.2" class="ltx_text ltx_font_typewriter">"""</span>
</span>
<span id="lstnumberx118" class="ltx_listingline"><span id="lstnumberx118.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx118.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Given</span><span id="lstnumberx118.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx118.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">positive</span><span id="lstnumberx118.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">integer</span><span id="lstnumberx118.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx118.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx118.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">you</span><span id="lstnumberx118.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">have</span><span id="lstnumberx118.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx118.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">make</span><span id="lstnumberx118.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx118.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">pile</span><span id="lstnumberx118.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx118.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx118.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">levels</span><span id="lstnumberx118.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx118.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx118.34" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx119" class="ltx_listingline"><span id="lstnumberx119.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx119.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">The</span><span id="lstnumberx119.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">first</span><span id="lstnumberx119.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">level</span><span id="lstnumberx119.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">has</span><span id="lstnumberx119.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx119.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx119.13" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx119.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">The</span><span id="lstnumberx119.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx119.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx119.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx119.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx119.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx119.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx119.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">level</span><span id="lstnumberx119.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx119.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">determined</span>
</span>
<span id="lstnumberx120" class="ltx_listingline"><span id="lstnumberx120.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx120.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">by</span><span id="lstnumberx120.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx120.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx120.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">pattern</span><span id="lstnumberx120.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.10" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx120.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">odd</span><span id="lstnumberx120.12" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx120.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">or</span><span id="lstnumberx120.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.16" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx120.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">even</span><span id="lstnumberx120.18" class="ltx_text ltx_font_typewriter">’:</span>
</span>
<span id="lstnumberx121" class="ltx_listingline"><span id="lstnumberx121.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx121.2" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx121.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">If</span><span id="lstnumberx121.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">pattern</span><span id="lstnumberx121.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx121.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.10" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx121.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">odd</span><span id="lstnumberx121.12" class="ltx_text ltx_font_typewriter">’,</span><span id="lstnumberx121.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">add</span><span id="lstnumberx121.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx121.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx121.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">odd</span><span id="lstnumberx121.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx121.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx121.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx121.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">previous</span><span id="lstnumberx121.29" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter">level</span><span id="lstnumberx121.31" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.32" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx121.33" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx122" class="ltx_listingline"><span id="lstnumberx122.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx122.2" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx122.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">If</span><span id="lstnumberx122.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">pattern</span><span id="lstnumberx122.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx122.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.10" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx122.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">even</span><span id="lstnumberx122.12" class="ltx_text ltx_font_typewriter">’,</span><span id="lstnumberx122.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">add</span><span id="lstnumberx122.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx122.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx122.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">even</span><span id="lstnumberx122.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx122.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx122.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx122.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">previous</span><span id="lstnumberx122.29" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter">level</span><span id="lstnumberx122.31" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.32" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx122.33" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx123" class="ltx_listingline"><span id="lstnumberx123.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx123.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Return</span><span id="lstnumberx123.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx123.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx123.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx123.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx123.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx123.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">each</span><span id="lstnumberx123.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">level</span><span id="lstnumberx123.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx123.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx123.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx123.23" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx123.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">where</span><span id="lstnumberx123.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">element</span><span id="lstnumberx123.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">at</span><span id="lstnumberx123.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">index</span>
</span>
<span id="lstnumberx124" class="ltx_listingline"><span id="lstnumberx124.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx124.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">i</span><span id="lstnumberx124.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">represents</span><span id="lstnumberx124.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx124.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx124.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx124.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stones</span><span id="lstnumberx124.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx124.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx124.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">level</span><span id="lstnumberx124.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.20" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx124.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">i</span><span id="lstnumberx124.22" class="ltx_text ltx_font_typewriter">+1).</span>
</span>
<span id="lstnumberx125" class="ltx_listingline">
</span>
<span id="lstnumberx126" class="ltx_listingline"><span id="lstnumberx126.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx126.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Examples</span><span id="lstnumberx126.3" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx127" class="ltx_listingline"><span id="lstnumberx127.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx127.2" class="ltx_text ltx_font_typewriter">&gt;&gt;&gt;</span><span id="lstnumberx127.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx127.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">make_a_pile</span><span id="lstnumberx127.5" class="ltx_text ltx_font_typewriter">(3,</span><span id="lstnumberx127.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx127.7" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx127.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">odd</span><span id="lstnumberx127.9" class="ltx_text ltx_font_typewriter">’)</span>
</span>
<span id="lstnumberx128" class="ltx_listingline"><span id="lstnumberx128.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx128.2" class="ltx_text ltx_font_typewriter">[3,</span><span id="lstnumberx128.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx128.4" class="ltx_text ltx_font_typewriter">5,</span><span id="lstnumberx128.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx128.6" class="ltx_text ltx_font_typewriter">7]</span>
</span>
<span id="lstnumberx129" class="ltx_listingline"><span id="lstnumberx129.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx129.2" class="ltx_text ltx_font_typewriter">&gt;&gt;&gt;</span><span id="lstnumberx129.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx129.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">make_a_pile</span><span id="lstnumberx129.5" class="ltx_text ltx_font_typewriter">(3,</span><span id="lstnumberx129.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx129.7" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx129.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">even</span><span id="lstnumberx129.9" class="ltx_text ltx_font_typewriter">’)</span>
</span>
<span id="lstnumberx130" class="ltx_listingline"><span id="lstnumberx130.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx130.2" class="ltx_text ltx_font_typewriter">[3,</span><span id="lstnumberx130.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx130.4" class="ltx_text ltx_font_typewriter">6,</span><span id="lstnumberx130.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx130.6" class="ltx_text ltx_font_typewriter">9]</span>
</span>
<span id="lstnumberx131" class="ltx_listingline"><span id="lstnumberx131.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx131.2" class="ltx_text ltx_font_typewriter">"""</span>
</span>
</span></span>
</span>
</td>
</tr>
<tr id="Ax1.1.tab1.5.4.3" class="ltx_tr">
<td id="Ax1.1.tab1.5.4.3.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t">
<span id="Ax1.1.tab1.5.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.tab1.5.4.3.1.1.1" class="ltx_p" style="width:341.4pt;">
<span id="Ax1.1.tab1.5.4.3.1.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,V3JpdGUgYSBQeXRob24gZnVuY3Rpb24gYHhfb3JfeShuLCB4LCB5KWAgdG8gc29sdmUgdGhlIGZvbGxvd2luZyBwcm9ibGVtOgpBIHNpbXBsZSBwcm9ncmFtIHdoaWNoIHNob3VsZCByZXR1cm4gdGhlIHZhbHVlIG9mIHggaWYgbiBpcwphIHByaW1lIG51bWJlciBhbmQgc2hvdWxkIHJldHVybiB0aGUgdmFsdWUgb2YgeSBvdGhlcndpc2UuCkV4YW1wbGVzOgpmb3IgeF9vcl95KDcsIDM0LCAxMikgPT0gMzQKZm9yIHhfb3JfeSgxNSwgOCwgNSkgPT0gNQ==" download="">⬇</a></span>
<span id="lstnumberx132" class="ltx_listingline"><span id="lstnumberx132.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Write</span><span id="lstnumberx132.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx132.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Python</span><span id="lstnumberx132.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx132.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.9" class="ltx_text ltx_font_typewriter">‘</span><span id="lstnumberx132.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x_or_y</span><span id="lstnumberx132.11" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx132.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx132.13" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx132.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x</span><span id="lstnumberx132.16" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx132.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">y</span><span id="lstnumberx132.19" class="ltx_text ltx_font_typewriter">)‘</span><span id="lstnumberx132.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx132.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">solve</span><span id="lstnumberx132.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx132.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx132.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx132.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">problem</span><span id="lstnumberx132.30" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx133" class="ltx_listingline"><span id="lstnumberx133.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">A</span><span id="lstnumberx133.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">simple</span><span id="lstnumberx133.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">program</span><span id="lstnumberx133.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">which</span><span id="lstnumberx133.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">should</span><span id="lstnumberx133.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">return</span><span id="lstnumberx133.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx133.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">value</span><span id="lstnumberx133.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx133.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x</span><span id="lstnumberx133.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">if</span><span id="lstnumberx133.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx133.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx133.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span>
</span>
<span id="lstnumberx134" class="ltx_listingline"><span id="lstnumberx134.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx134.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx134.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">prime</span><span id="lstnumberx134.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx134.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number</span><span id="lstnumberx134.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx134.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx134.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx134.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">should</span><span id="lstnumberx134.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx134.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">return</span><span id="lstnumberx134.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx134.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx134.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx134.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">value</span><span id="lstnumberx134.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx134.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx134.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx134.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">y</span><span id="lstnumberx134.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx134.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">otherwise</span><span id="lstnumberx134.22" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx135" class="ltx_listingline"><span id="lstnumberx135.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Examples</span><span id="lstnumberx135.2" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx136" class="ltx_listingline"><span id="lstnumberx136.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">for</span><span id="lstnumberx136.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx136.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x_or_y</span><span id="lstnumberx136.4" class="ltx_text ltx_font_typewriter">(7,</span><span id="lstnumberx136.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx136.6" class="ltx_text ltx_font_typewriter">34,</span><span id="lstnumberx136.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx136.8" class="ltx_text ltx_font_typewriter">12)</span><span id="lstnumberx136.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx136.10" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx136.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx136.12" class="ltx_text ltx_font_typewriter">34</span>
</span>
<span id="lstnumberx137" class="ltx_listingline"><span id="lstnumberx137.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">for</span><span id="lstnumberx137.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx137.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x_or_y</span><span id="lstnumberx137.4" class="ltx_text ltx_font_typewriter">(15,</span><span id="lstnumberx137.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx137.6" class="ltx_text ltx_font_typewriter">8,</span><span id="lstnumberx137.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx137.8" class="ltx_text ltx_font_typewriter">5)</span><span id="lstnumberx137.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx137.10" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx137.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx137.12" class="ltx_text ltx_font_typewriter">5</span>
</span>
</span></span>
</span>
</td>
<td id="Ax1.1.tab1.5.4.3.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t">
<span id="Ax1.1.tab1.5.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="Ax1.1.tab1.5.4.3.2.1.1" class="ltx_p" style="width:341.4pt;">
<span id="Ax1.1.tab1.5.4.3.2.1.1.1" class="ltx_listing ltx_lstlisting ltx_listing"><span class="ltx_listing_data"><a href="data:text/plain;base64,Q29tcGxldGUgdGhlIHN1YnNlcXVlbnQgbGluZXMgb2YgcHJvZ3JhbW1pbmcgY29kZToKLyoKQW4gZWxlbWVudGFyeSBwcm9ncmFtIGludGVuZGVkIHRvLCBnaXZlbiBhIHNpbmdsZSBuIHZhbHVlLCBkZWNpcGhlciBiZXR3ZWVuIHR3bwpkaXN0aW5jdCBwb3NzaWJpbGl0aWVzOgotIElmIHRoZSBwcm92aWRlZCBuIGlzIGEgbnVtZXJpY2FsIHByaW1lLCB0aGUgcHJvZ3JhbSBzaG91bGQgeWllbGQgdGhlCm91dHB1dCB2YWx1ZSBlcXVpdmFsZW50IHRvIHZhcmlhYmxlIHguCi0gU2hvdWxkIHRoZSBuIGZhaWwgdG8gbWVldCB0aGUgcHJpbWUgY3JpdGVyaWEsIHRoZSBwcm9ncmFtIHNob3VsZCB0aGVuCnByZXNlbnQgdGhlIHZhbHVlIGNvbnRhaW5lZCB3aXRoaW4gdmFyaWFibGUgeSBhcyBpdHMgb3V0cHV0LgoKRXhhbXBsZXM6ClVwb24gcnVubmluZyB4X29yX3koNywgMzQsIDEyKSB0aGUgb3V0cHV0IHdpbGwgYmUgMzQKV2hpbGUgZXhlY3V0aW5nIHhfb3JfeSgxNSwgOCwgNSkgdGhlIG91dHB1dCB3aWxsIGJlIDUKCiovCiNpbmNsdWRlPHN0ZGlvLmg+CnVzaW5nIG5hbWVzcGFjZSBzdGQ7CmludCB4X29yX3koaW50IG4saW50IHgsaW50IHkpew==" download="">⬇</a></span>
<span id="lstnumberx138" class="ltx_listingline"><span id="lstnumberx138.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Complete</span><span id="lstnumberx138.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx138.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx138.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx138.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">subsequent</span><span id="lstnumberx138.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx138.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">lines</span><span id="lstnumberx138.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx138.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx138.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx138.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">programming</span><span id="lstnumberx138.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx138.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">code</span><span id="lstnumberx138.14" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx139" class="ltx_listingline"><span id="lstnumberx139.1" class="ltx_text ltx_font_typewriter">/*</span>
</span>
<span id="lstnumberx140" class="ltx_listingline"><span id="lstnumberx140.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">An</span><span id="lstnumberx140.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">elementary</span><span id="lstnumberx140.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">program</span><span id="lstnumberx140.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">intended</span><span id="lstnumberx140.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx140.10" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx140.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx140.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx140.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">single</span><span id="lstnumberx140.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx140.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">value</span><span id="lstnumberx140.21" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx140.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">decipher</span><span id="lstnumberx140.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">between</span><span id="lstnumberx140.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx140.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">two</span>
</span>
<span id="lstnumberx141" class="ltx_listingline"><span id="lstnumberx141.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">distinct</span><span id="lstnumberx141.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx141.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">possibilities</span><span id="lstnumberx141.4" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx142" class="ltx_listingline"><span id="lstnumberx142.1" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx142.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">If</span><span id="lstnumberx142.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx142.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">provided</span><span id="lstnumberx142.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx142.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx142.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx142.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">numerical</span><span id="lstnumberx142.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">prime</span><span id="lstnumberx142.18" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx142.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx142.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">program</span><span id="lstnumberx142.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">should</span><span id="lstnumberx142.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">yield</span><span id="lstnumberx142.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx142.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span>
</span>
<span id="lstnumberx143" class="ltx_listingline"><span id="lstnumberx143.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">output</span><span id="lstnumberx143.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx143.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">value</span><span id="lstnumberx143.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx143.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">equivalent</span><span id="lstnumberx143.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx143.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx143.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx143.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">variable</span><span id="lstnumberx143.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx143.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x</span><span id="lstnumberx143.12" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx144" class="ltx_listingline"><span id="lstnumberx144.1" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx144.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Should</span><span id="lstnumberx144.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx144.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx144.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">fail</span><span id="lstnumberx144.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx144.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">meet</span><span id="lstnumberx144.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx144.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">prime</span><span id="lstnumberx144.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">criteria</span><span id="lstnumberx144.20" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx144.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx144.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">program</span><span id="lstnumberx144.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">should</span><span id="lstnumberx144.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx144.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">then</span>
</span>
<span id="lstnumberx145" class="ltx_listingline"><span id="lstnumberx145.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">present</span><span id="lstnumberx145.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx145.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx145.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx145.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">value</span><span id="lstnumberx145.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx145.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">contained</span><span id="lstnumberx145.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx145.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">within</span><span id="lstnumberx145.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx145.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">variable</span><span id="lstnumberx145.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx145.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">y</span><span id="lstnumberx145.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx145.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">as</span><span id="lstnumberx145.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx145.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">its</span><span id="lstnumberx145.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx145.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">output</span><span id="lstnumberx145.20" class="ltx_text ltx_font_typewriter">.</span>
</span>
<span id="lstnumberx146" class="ltx_listingline">
</span>
<span id="lstnumberx147" class="ltx_listingline"><span id="lstnumberx147.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Examples</span><span id="lstnumberx147.2" class="ltx_text ltx_font_typewriter">:</span>
</span>
<span id="lstnumberx148" class="ltx_listingline"><span id="lstnumberx148.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Upon</span><span id="lstnumberx148.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx148.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">running</span><span id="lstnumberx148.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx148.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x_or_y</span><span id="lstnumberx148.6" class="ltx_text ltx_font_typewriter">(7,</span><span id="lstnumberx148.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx148.8" class="ltx_text ltx_font_typewriter">34,</span><span id="lstnumberx148.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx148.10" class="ltx_text ltx_font_typewriter">12)</span><span id="lstnumberx148.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx148.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx148.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx148.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">output</span><span id="lstnumberx148.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx148.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">will</span><span id="lstnumberx148.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx148.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">be</span><span id="lstnumberx148.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx148.20" class="ltx_text ltx_font_typewriter">34</span>
</span>
<span id="lstnumberx149" class="ltx_listingline"><span id="lstnumberx149.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">While</span><span id="lstnumberx149.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx149.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">executing</span><span id="lstnumberx149.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx149.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x_or_y</span><span id="lstnumberx149.6" class="ltx_text ltx_font_typewriter">(15,</span><span id="lstnumberx149.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx149.8" class="ltx_text ltx_font_typewriter">8,</span><span id="lstnumberx149.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx149.10" class="ltx_text ltx_font_typewriter">5)</span><span id="lstnumberx149.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx149.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx149.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx149.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">output</span><span id="lstnumberx149.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx149.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">will</span><span id="lstnumberx149.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx149.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">be</span><span id="lstnumberx149.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx149.20" class="ltx_text ltx_font_typewriter">5</span>
</span>
<span id="lstnumberx150" class="ltx_listingline">
</span>
<span id="lstnumberx151" class="ltx_listingline"><span id="lstnumberx151.1" class="ltx_text ltx_font_typewriter">*/</span>
</span>
<span id="lstnumberx152" class="ltx_listingline"><span id="lstnumberx152.1" class="ltx_text ltx_font_typewriter">#</span><span id="lstnumberx152.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">include</span><span id="lstnumberx152.3" class="ltx_text ltx_font_typewriter">&lt;</span><span id="lstnumberx152.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stdio</span><span id="lstnumberx152.5" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx152.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">h</span><span id="lstnumberx152.7" class="ltx_text ltx_font_typewriter">&gt;</span>
</span>
<span id="lstnumberx153" class="ltx_listingline"><span id="lstnumberx153.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">using</span><span id="lstnumberx153.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx153.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">namespace</span><span id="lstnumberx153.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx153.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">std</span><span id="lstnumberx153.6" class="ltx_text ltx_font_typewriter">;</span>
</span>
<span id="lstnumberx154" class="ltx_listingline"><span id="lstnumberx154.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">int</span><span id="lstnumberx154.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx154.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x_or_y</span><span id="lstnumberx154.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx154.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">int</span><span id="lstnumberx154.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx154.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx154.8" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx154.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">int</span><span id="lstnumberx154.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx154.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">x</span><span id="lstnumberx154.12" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx154.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">int</span><span id="lstnumberx154.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx154.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">y</span><span id="lstnumberx154.16" class="ltx_text ltx_font_typewriter">){</span>
</span>
</span></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="Ax1.T4" class="ltx_table ltx_figure_panel">
<table id="Ax1.T4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="Ax1.T4.1.1.1" class="ltx_tr">
<th id="Ax1.T4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="Ax1.T4.1.1.1.1.1" class="ltx_text ltx_font_bold">Model Name</span></th>
<th id="Ax1.T4.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="Ax1.T4.1.1.1.2.1" class="ltx_text ltx_font_bold">Family</span></th>
<th id="Ax1.T4.1.1.1.3" class="ltx_td ltx_nopad_r ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="Ax1.T4.1.1.1.3.1" class="ltx_text ltx_font_bold">pass@1</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="Ax1.T4.1.2.1" class="ltx_tr">
<td id="Ax1.T4.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t">Mistral-7b</td>
<td id="Ax1.T4.1.2.1.2" class="ltx_td ltx_align_left ltx_border_t">Mistral</td>
<td id="Ax1.T4.1.2.1.3" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.11</td>
</tr>
<tr id="Ax1.T4.1.3.2" class="ltx_tr">
<td id="Ax1.T4.1.3.2.1" class="ltx_td ltx_align_left">Mixtral 8x7B</td>
<td id="Ax1.T4.1.3.2.2" class="ltx_td ltx_align_left">Mistral</td>
<td id="Ax1.T4.1.3.2.3" class="ltx_td ltx_nopad_r ltx_align_left">0.17</td>
</tr>
<tr id="Ax1.T4.1.4.3" class="ltx_tr">
<td id="Ax1.T4.1.4.3.1" class="ltx_td ltx_align_left">Mixtral 8x22B</td>
<td id="Ax1.T4.1.4.3.2" class="ltx_td ltx_align_left">Mistral</td>
<td id="Ax1.T4.1.4.3.3" class="ltx_td ltx_nopad_r ltx_align_left">0.38</td>
</tr>
<tr id="Ax1.T4.1.5.4" class="ltx_tr">
<td id="Ax1.T4.1.5.4.1" class="ltx_td ltx_align_left">Codestral 22B</td>
<td id="Ax1.T4.1.5.4.2" class="ltx_td ltx_align_left">Mistral</td>
<td id="Ax1.T4.1.5.4.3" class="ltx_td ltx_nopad_r ltx_align_left">0.40</td>
</tr>
<tr id="Ax1.T4.1.6.5" class="ltx_tr">
<td id="Ax1.T4.1.6.5.1" class="ltx_td ltx_align_left">Codellama-7b-instruct</td>
<td id="Ax1.T4.1.6.5.2" class="ltx_td ltx_align_left">Meta</td>
<td id="Ax1.T4.1.6.5.3" class="ltx_td ltx_nopad_r ltx_align_left">0.14</td>
</tr>
<tr id="Ax1.T4.1.7.6" class="ltx_tr">
<td id="Ax1.T4.1.7.6.1" class="ltx_td ltx_align_left">Codellama-34b-instruct</td>
<td id="Ax1.T4.1.7.6.2" class="ltx_td ltx_align_left">Meta</td>
<td id="Ax1.T4.1.7.6.3" class="ltx_td ltx_nopad_r ltx_align_left">0.19</td>
</tr>
<tr id="Ax1.T4.1.8.7" class="ltx_tr">
<td id="Ax1.T4.1.8.7.1" class="ltx_td ltx_align_left">Llama2-7b-chat</td>
<td id="Ax1.T4.1.8.7.2" class="ltx_td ltx_align_left">Meta</td>
<td id="Ax1.T4.1.8.7.3" class="ltx_td ltx_nopad_r ltx_align_left">0.02</td>
</tr>
<tr id="Ax1.T4.1.9.8" class="ltx_tr">
<td id="Ax1.T4.1.9.8.1" class="ltx_td ltx_align_left">Llama2-70b-chat</td>
<td id="Ax1.T4.1.9.8.2" class="ltx_td ltx_align_left">Meta</td>
<td id="Ax1.T4.1.9.8.3" class="ltx_td ltx_nopad_r ltx_align_left">0.10</td>
</tr>
<tr id="Ax1.T4.1.10.9" class="ltx_tr">
<td id="Ax1.T4.1.10.9.1" class="ltx_td ltx_align_left">Llama3-8b-instruct</td>
<td id="Ax1.T4.1.10.9.2" class="ltx_td ltx_align_left">Meta</td>
<td id="Ax1.T4.1.10.9.3" class="ltx_td ltx_nopad_r ltx_align_left">0.27</td>
</tr>
<tr id="Ax1.T4.1.11.10" class="ltx_tr">
<td id="Ax1.T4.1.11.10.1" class="ltx_td ltx_align_left">Llama3-70b-instruct</td>
<td id="Ax1.T4.1.11.10.2" class="ltx_td ltx_align_left">Meta</td>
<td id="Ax1.T4.1.11.10.3" class="ltx_td ltx_nopad_r ltx_align_left">0.49</td>
</tr>
<tr id="Ax1.T4.1.12.11" class="ltx_tr">
<td id="Ax1.T4.1.12.11.1" class="ltx_td ltx_align_left">GPT-3.5-tb</td>
<td id="Ax1.T4.1.12.11.2" class="ltx_td ltx_align_left">OpenAI</td>
<td id="Ax1.T4.1.12.11.3" class="ltx_td ltx_nopad_r ltx_align_left">0.37</td>
</tr>
<tr id="Ax1.T4.1.13.12" class="ltx_tr">
<td id="Ax1.T4.1.13.12.1" class="ltx_td ltx_align_left">GPT-4</td>
<td id="Ax1.T4.1.13.12.2" class="ltx_td ltx_align_left">OpenAI</td>
<td id="Ax1.T4.1.13.12.3" class="ltx_td ltx_nopad_r ltx_align_left">0.53</td>
</tr>
<tr id="Ax1.T4.1.14.13" class="ltx_tr">
<td id="Ax1.T4.1.14.13.1" class="ltx_td ltx_align_left">GPT-4o</td>
<td id="Ax1.T4.1.14.13.2" class="ltx_td ltx_align_left">OpenAI</td>
<td id="Ax1.T4.1.14.13.3" class="ltx_td ltx_nopad_r ltx_align_left">0.63</td>
</tr>
<tr id="Ax1.T4.1.15.14" class="ltx_tr">
<td id="Ax1.T4.1.15.14.1" class="ltx_td ltx_align_left">Claude-2</td>
<td id="Ax1.T4.1.15.14.2" class="ltx_td ltx_align_left">Anthropic</td>
<td id="Ax1.T4.1.15.14.3" class="ltx_td ltx_nopad_r ltx_align_left">0.34</td>
</tr>
<tr id="Ax1.T4.1.16.15" class="ltx_tr">
<td id="Ax1.T4.1.16.15.1" class="ltx_td ltx_align_left">Claude-3-haiku</td>
<td id="Ax1.T4.1.16.15.2" class="ltx_td ltx_align_left">Anthropic</td>
<td id="Ax1.T4.1.16.15.3" class="ltx_td ltx_nopad_r ltx_align_left">0.34</td>
</tr>
<tr id="Ax1.T4.1.17.16" class="ltx_tr">
<td id="Ax1.T4.1.17.16.1" class="ltx_td ltx_align_left">Claude-3-sonnet</td>
<td id="Ax1.T4.1.17.16.2" class="ltx_td ltx_align_left">Anthropic</td>
<td id="Ax1.T4.1.17.16.3" class="ltx_td ltx_nopad_r ltx_align_left">0.40</td>
</tr>
<tr id="Ax1.T4.1.18.17" class="ltx_tr">
<td id="Ax1.T4.1.18.17.1" class="ltx_td ltx_align_left">Claude-3-opus</td>
<td id="Ax1.T4.1.18.17.2" class="ltx_td ltx_align_left">Anthropic</td>
<td id="Ax1.T4.1.18.17.3" class="ltx_td ltx_nopad_r ltx_align_left">0.54</td>
</tr>
<tr id="Ax1.T4.1.19.18" class="ltx_tr">
<td id="Ax1.T4.1.19.18.1" class="ltx_td ltx_align_left">Deepseek-coder-33b-instruct</td>
<td id="Ax1.T4.1.19.18.2" class="ltx_td ltx_align_left">DeepSeek</td>
<td id="Ax1.T4.1.19.18.3" class="ltx_td ltx_nopad_r ltx_align_left">0.39</td>
</tr>
<tr id="Ax1.T4.1.20.19" class="ltx_tr">
<td id="Ax1.T4.1.20.19.1" class="ltx_td ltx_align_left">DBRX Instruct</td>
<td id="Ax1.T4.1.20.19.2" class="ltx_td ltx_align_left">DBRX</td>
<td id="Ax1.T4.1.20.19.3" class="ltx_td ltx_nopad_r ltx_align_left">0.25</td>
</tr>
<tr id="Ax1.T4.1.21.20" class="ltx_tr">
<td id="Ax1.T4.1.21.20.1" class="ltx_td ltx_align_left">Qwen1.5-72B-Chat</td>
<td id="Ax1.T4.1.21.20.2" class="ltx_td ltx_align_left">Qwen</td>
<td id="Ax1.T4.1.21.20.3" class="ltx_td ltx_nopad_r ltx_align_left">0.20</td>
</tr>
<tr id="Ax1.T4.1.22.21" class="ltx_tr">
<td id="Ax1.T4.1.22.21.1" class="ltx_td ltx_align_left">Qwen1.5-110B-Chat</td>
<td id="Ax1.T4.1.22.21.2" class="ltx_td ltx_align_left">Qwen</td>
<td id="Ax1.T4.1.22.21.3" class="ltx_td ltx_nopad_r ltx_align_left">0.30</td>
</tr>
<tr id="Ax1.T4.1.23.22" class="ltx_tr">
<td id="Ax1.T4.1.23.22.1" class="ltx_td ltx_align_left">Qwen 2-72B Instruct</td>
<td id="Ax1.T4.1.23.22.2" class="ltx_td ltx_align_left">Qwen</td>
<td id="Ax1.T4.1.23.22.3" class="ltx_td ltx_nopad_r ltx_align_left">0.42</td>
</tr>
<tr id="Ax1.T4.1.24.23" class="ltx_tr">
<td id="Ax1.T4.1.24.23.1" class="ltx_td ltx_align_left">Command R+</td>
<td id="Ax1.T4.1.24.23.2" class="ltx_td ltx_align_left">Cohere</td>
<td id="Ax1.T4.1.24.23.3" class="ltx_td ltx_nopad_r ltx_align_left">0.22</td>
</tr>
<tr id="Ax1.T4.1.25.24" class="ltx_tr">
<td id="Ax1.T4.1.25.24.1" class="ltx_td ltx_align_left ltx_border_bb">Command R</td>
<td id="Ax1.T4.1.25.24.2" class="ltx_td ltx_align_left ltx_border_bb">Cohere</td>
<td id="Ax1.T4.1.25.24.3" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">0.12</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 4: </span>Results of pass@1 rate on the LBPP dataset for a selection of models.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<div class="ltx_pagination ltx_figure_panel ltx_role_newpage"></div>
</div>
</div>
</figure>
</div>
</div>
</figure>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.07564" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.07565" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.07565">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.07565" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.07566" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 15:54:47 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.07565
In this paper we consider contamination by code generation test sets, in particular in their use in modern large language models.
We discuss three possible sources of such contamination and show findings supporting each of them: (i)  direct data leakage, (ii)  indirect data leakage through the use of synthetic data and (iii)  overfitting to evaluation sets during model selection. Key to our findings is a new dataset of 161 prompts with their associated python solutions, dataset which is released athttps://huggingface.co/datasets/CohereForAI/lbpp.
abstract
On Leakage of Code Generation Evaluation Datasets
Abstract
2407.07565
1IntroductionCode generation has emerged as an important skill for large language models to master.
Measuring recent progress in code generation has relied on few, critical benchmarks to judge performance between model families and checkpoints. While many recent sophisticated evaluation datasets have been proposed, the community largely relies onHumanEvalandMBPPto judge a new model’s code capability. In fact, all major announcements in 2023-2024 claiming advanced code capabilities—from academic and industry labs—use at least one of these two datasets. Practically, reportingHumanEvalandMBPPis mandatory for a model to report competitive code generation.However, the importance of these benchmarks has led to a conflict between popularity and utility.
On one side, obtaining competitive numbers comes with significant scientific and economic reward—made increasingly easy with the proliferation of public replicas of these datasets.
However, this prevalence has led to data leakage beyond the original evaluation scope, i.e.,data contamination, and once this evaluation datacontaminatesmodel training, the validity of the metrics as a measure of generalization capability becomes unreliable.
If a model has been trained on the same data we use for out-of-distribution generalization (or is selected based on its performance on that data) , we break an implicit tenet of how model capability can be measured. We argue that understanding the effect of contamination is critical to accurately interpreting scores on these benchmarks.In this paper, we review the evidence that these two benchmarks have contaminated most large LLMs, which we define as any procedure that leaked those datasetsduringmodel training.
The most obvious method of contamination is presence inside training data, and we provide evidence that it is highly probably that this occurs at a scale too large to be avoidable.
A second possibility is that contamination happens indirectly through the use of synthetic data—a widespread paradigm used in particular to increase code capabilities by generating additional code training tokens.
Finally, we argue that final model selection might have been overly influenced by their performance on these datasets, overfitting to performance on these metrics over general-purpose code-oriented skills.To measure this contamination, we proposeLessBasicPythonProblems(LBPP) , a code generation benchmark similar toHumanEvalandMBPPin style and scale, but more difficult.
LBPP is similarly portable, but is produced in a manner to reduce any likelihood of leakage into present code training data. We contribute LBPP to act as a genuinely held-out test set to measurecurrentcode generation capability, and potential overfitting toHumanEvalandMBPP.
introduction
On Leakage of Code Generation Evaluation Datasets
1Introduction
2407.07565
2Related WorkHumanEvalandMBPPremain the most reported results on public leaderboards, but others similar datasets exist.
They consist of short and mostly simple (not programming competition level)  instructions with completions in Python.
Translation into other programming languages exist for those datasets, as well as versions with additional tests.proposed a continuously updated set of leetcode to improve dataset challenge by including harder and novel (unseen)  prompts.aims for challenging software engineering problems, that require understanding of full repositories.
In a similar vein, RepoQA111https://github.com/evalplus/repoqaand Bug In The Code Stack222https://github.com/HammingHQ/bug-in-the-code-stackfocus on understanding long contexts within code tasks.
One proposed solution is to use hidden evaluation sets, however, these do not allow inspection of failure cases and requires trusting the quality and correctness of an opaque ‘black-box’ evaluation setup.
Recently,analyzed data contamination in popular pretraining datasets: reporting thatofHumanEvalsamples are present in The Pile, andin The Stack.
Different from our analysis however, they conclude: “we do not find the performance rankings of the models to change with decontaminated results”.(a) HumanEval(b) MBPPFigure 1:Pass@1 rate of popular datasets and the 161 prompts in LBPP.
body
On Leakage of Code Generation Evaluation Datasets
2Related Work
2407.07565
3Possible sources of contaminationWe provide three hypotheses—with evidence for each one—on why existing models might be over-optimized towards existing leaked benchmarks.3.1Direct data leakageThe most obvious reason is the simplest: many of the test datasets are of widespread use and the simplest answer might be that modern LLMs are just trained on this evaluation data.
We note that intentional (i.e., tocheat)  or unintentional contamination has the same net effect: training on evaluation data limits the confidence and utility of the benchmark results.
For code tasks, it is very expensive to curate datasets of natural language to code instructions (one example generally costing several dozen US dollars) .
For any group aiming to minimize data cost to improve coding performance, the dollar value of creating new datasets can be very high.
This leads to a common practice of web scraping code-oriented resources (e.g., GitHub or Stackoverflow)  for data. However, these resources are also likely sources of contamination.
Since the release ofHumanEvalandMBPPin 2021 these datasets have been branched, re-used and copied all across the Internet.
The small data size and portability of such benchmarks encourages replication within code repositories.
For example, searching for the prompts fromHumanEvalon GitHub returns a hit in all cases—the median hits isand the minimum(see Fig.2) . In many cases, these hits are exact duplicates and indications of a fork of the original dataset.While decontamination of training sets is becoming more common, present decontamination filters designed for natural text adapts poorly to code.
To operate efficiently at scale, most filters rely on generic deduplication algorithms e.g., such as-gram matching or hashing functions.
Such surface-level matching does not adequately capture code similarity where a simple variable name change leaves program semantics unchanged, but changing a single keyword can have profound changes.333E.g., compare the instruction “return true if the string is a float” with “return true if the string is a verb”.The same shortcomings of decontamination efforts apply to the creation of large-scale synthetic datasets: for example the model-generated dataset of Starcoderis decontaminated only by removing exact docstrings or solutions that matchHumanEvalorMBPP.Figure 2:Histogram (excluding outliers)  of occurrences forHumanEvalprompts in public GitHub repositories. Every prompt occurs at leasttimes.The recent exploration ofaims to quantify the proportion of this data leakage in existing datasets using plagiarism tools specifically designed for code.
Even when static training datasets are cleaned, contamination may persist.
Entities who serve models through an API may encounter these benchmark tasks when evaluated by third party users. When a sample of real model usage is annotated for future training data, samples from benchmark evaluation can leak into future training corpora. Furthermore, these samples may include subtle phrasing variations and format changes that further complicate heuristic deduplication.
In this scenario, a model may easily memorize completions to purportedly novel prompts. As evidence of this phenomena, we prompted one popular commercial system with partial prompts fromHumanEvalthat were designed to keep the instruction under-specified.
TableAppendixin the Appendix shows the outcome and evidence that—despite the ambiguity of the prompt—the resulting completion matches exactly the gold solution from the test set ofHumanEval.3.2Data leakage through synthetic dataThe most capable of code language models rely heavily on the use of synthetic training data.A typical pipeline generally consists of: curating prompts related to code generation, inferring completions with a previously trained LLM, and synthesizing unit tests for relevant prompts using LLMs.
Completions that pass the respective unit tests are considered valid code solutions and can be used as future training examples.
Alternatively, if a sufficiently powerful model is used, completions might be used as-is.evol-instructfor example comprisescomplex query prompts coupled with completions from numerous closed and open-source models.444Per downloads, the most popular version is a ‘lightly decontaminated’ versionon HuggingFace here.It is widely used by many code LLMs such as WizardCoder.
Prior reports,discuss an apparent high similarity between some of those examples inevol-instructandHumanEval.
We extend this analysis by studying the similarity between embedded representations of theprompts555Embedded using Cohereembed v3.ofHumanEvalandMBPPwith nearest neighbors fromevol-instructandStarcoder-Instruct. Fig3highlights widespread similarity between evaluation data and synthetic training datasets ‘Starcoder V2 OSS Instruct’ and ‘evol instruct’. Even if unintentional, this contamination further damages the utility of these benchmarks as a held-out test of code generation capability.Figure 3:Histogram of cosine similarities for prompts inHumanEval,MBPPand LBPP relative to two popular synthetic code training datasets. We note the high similarity between mostHumanEvalprompts toevol-instruct, and how LBPP has reduced overall similarity to either training dataset.Strikingly, we saw that training on this dataset can increase performance of one of our models by 14 absolute points onHumanEval(from 0.52 pass@1 to 0.66) , while only by 1 point onMBPP(from 0.52 to 0.53) .
Inspecting closest neighbours for almost all examples inHumanEval(see examples in TableAppendixin the Appendix) —we identify a semantically equivalent version inevol-instruct.Even when the original evaluation datasets were not used as inspiration for the creation of the synthetic datasets the simple nature of the evaluation datasets might make duplication unavoidable.
There are only so many short natural language prompts describing typical interview-style programming questions that can be used.
While synthetically generated prompts may not be explicitly based on provided examples from a given test set in the prompt context, the massive scale of these datasets (instances for StarCoder-Instruct prior to deduplication)  runs the risk of exhausting the possible number of variations on questions.
Table1shows examples where generated prompts are extremely similar to questions in theMBPPtest set.
Note that despite that similarity training on this dataset did not improve substantially performance onMBPP.3.3Overfitting to test setsThe exaggerated importance of these benchmarks encourages an incentive structure where model selection prioritizes gain on a narrow suite of metrics. While it may be tempting to use such benchmarks as a deciding factor between similar checkpoints, there is weak evidence for these benchmarks correlating with ‘solving code generation’. While the meaning and measurement of this unscientific objective is subject to constant revision, selecting for optimalHumanEvalperformance may be akin to-hacking in other fields.
This practice can be justified by assuming that these benchmarks are the new dev sets, while the true test is the usage of users over time.
The risk remains however that some models overfit to those test sets more than others, distorting perception on relative performance.In order to measure this, we createdLess Basic Python Problems (LBPP) , a dataset of 161 code completion problems in the style ofHumanEval.
Human annotators were instructed to create totally fresh problems, which were not solvable by an internal model666We will update references to internal models in future revisions.they had access to.
Annotators had competitive programming experience and could use programming books as inspiration, but were instructed not to copy existing solutions on the Internet and not to use any LLMs.
All annotators were paid above minimum wage in their respective countries, and all final prompt-completion pairs were manually reviewed by the authors.
This adversarial collection resulted in more difficult problems, with most models solving less thanof the dataset.
Results on a selection of models are in Table4in the Appendix.Using this dataset we can correlate the performances of existing models on them against performance of the well-known benchmarks.
The two plots comparing them toHumanEvalandMBPPcan be seen in Fig.1.
There is a clear correlation on both data-sets, indicating that the public benchmarks are still a valuable target signal.
However, when zooming in — in particular for models from the same family — the correlation often becomes negative, which might indicate that the selected checkpoint to release performs better on those public datasets while under-performing on new datapoints.
Note in particular in Fig.1(a) the crowded space between 0.75 and 0.8 of the-axis (pass@1 ofHumanEval) .
Despite very similar performance on that public evaluation set, the numbers of LBPP vary wildly.
On the other side, in Fig.1(b) a vertical line can be traced at aroundof LBPP that crosses various points – indicating maybe an overfit toMBPP.
body
On Leakage of Code Generation Evaluation Datasets
3Possible sources of contamination
2407.07565
4ConclusionWe study the cause and effect of data contamination by two popular code generation benchmarks. Our analysis highlights that contamination is likely unavoidable at the LLM scale given the difficulty of filtering every potential permutation of a benchmark dataset.
This insight motivates our contribution of a novel code generation benchmark to evaluate contemporary LLMs in a contamination-free setting.
We are well aware that our decision to release this dataset will make future leakage impossible to control.
However, with the context of the fast-paced model development cycles that LLMs are currently undergoing we believe that releasing this increases trustworthiness and usefulness of this dataset.
It is conveniently designed to serve as drop-in replacement (or addition)  of current evaluation sets.
On top of its newness, the more challenging nature of this dataset also allows it to provide more signal for model comparison.
body
On Leakage of Code Generation Evaluation Datasets
4Conclusion
2407.07565
5LimitationsAll the model analysis was doneblack-box, without inspecting the model weights or the training set (excepting the work on synthetic data) .
There is no reason why this dataset will not follow the same path than the two studied here.
As mentioned in the Conclusion we believe there is more value in that than in an alternative solution (not releasing or keeping it behind an API access) .
conclusion
On Leakage of Code Generation Evaluation Datasets
5Limitations