<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2404.11018] Many-Shot In-Context Learning</title><meta property="og:description" content="Large language models (LLMs) excel at few-shot in-context learning (ICL) – learning from a few examples provided in context at inference, without any weight updates.
Newly expanded context windows allow us to investiga…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Many-Shot In-Context Learning">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Many-Shot In-Context Learning">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2404.11018">

<!--Generated on Sun May  5 20:28:29 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<div id="p1" class="ltx_para">
<span id="p1.1" class="ltx_ERROR undefined">\correspondingauthor</span>
<p id="p1.2" class="ltx_p">rishabhagarwal@google.com, singhavi@google.com
<span id="p1.2.1" class="ltx_ERROR undefined">\reportnumber</span> 














</p>
</div>
<h1 class="ltx_title ltx_title_document">Many-Shot In-Context Learning</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Rishabh Agarwal
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Contributed equally
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Avi Singh
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Contributed equally
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Lei M. Zhang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Core contribution
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Bernd Bohnet
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Core contribution
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Stephanie Chan
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Core contribution
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ankesh Anand
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Zaheer Abbas
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Azade Nova
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">John D. Co-Reyes
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Eric Chu
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Feryal Behbahani
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Aleksandra Faust
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Hugo Larochelle
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id1.id1" class="ltx_p">Large language models (LLMs) excel at few-shot in-context learning (ICL) – learning from a few examples provided in context at inference, without any weight updates.
Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples – the many-shot regime.
Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks.
While promising, many-shot ICL can be bottlenecked by the available amount of human-generated outputs.
To mitigate this limitation, we explore two new settings: “Reinforced ICL” and “Unsupervised ICL”.
Reinforced ICL uses model-generated chain-of-thought rationales in place of human rationales.
Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific inputs.
We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases and can learn high-dimensional functions with numerical inputs. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream performance.
</p>
</div>
<section id="S1" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<figure id="S1.F1" class="ltx_figure"><img src="/html/2404.11018/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="183" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span><span id="S1.F1.2.1" class="ltx_text ltx_font_bold">Many-shot vs Few-Shot In-Context Learning</span> (ICL) across several tasks. Many-shot learning exhibits consistent performance gains over few-shot ICL. This gain is especially dramatic for difficult non-natural language tasks like sequential parity prediction and linear classification. Number of best-performing shots for many-shot ICL are shown inside the bar for each task. For few-shot ICL, we either use typical number of shots used on a benchmark, for example, 4-shot for MATH, or the longest prompt among the ones we tested with less than the GPT-3 context length of 2048 tokens. Reasoning-oriented tasks, namely MATH, GSM8K, BBH, and GPQA uses human-generated chain-of-thought rationales. For translation, we report performance FLORES-MT result on English to Kurdish, summarization uses XLSum, MATH corresponds to the MATH500 test set, and sentiment analysis results are reported with semantically-unrelated labels. See §<a href="#S3" title="3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, §<a href="#S4" title="4 Many-shot Learning without Human-Written Rationales ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, and §<a href="#S5" title="5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> for more details.</figcaption>
</figure>
<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Large language models (LLMs) have demonstrated a remarkable ability to perform <em id="S1.p1.1.1" class="ltx_emph ltx_font_italic">in-context learning</em> (ICL): they can learn a new task just from input-output examples, also known as <em id="S1.p1.1.2" class="ltx_emph ltx_font_italic">shots</em>, which precede a test input presented within the LLM context. However, an LLM’s context window, i.e. the amount of tokenized inputs they can process for each forward inference, limits the number of shots that can be used for ICL, restricting prior research to the <em id="S1.p1.1.3" class="ltx_emph ltx_font_italic">few-shot</em> learning regime.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">While relatively under-explored, <em id="S1.p2.1.1" class="ltx_emph ltx_font_italic">many-shot</em> learning – ICL with a large number of shots, for example, hundreds or more – can potentially make LLMs more versatile, adaptable, and ultimately more useful. For example, many shots allow clearer specification of the task, whereas doing so through few shots may be more ambiguous. Exploring many-shot ICL is now feasible, given the recent increase in context windows of publicly available LLMs by at least <math id="S1.p2.1.m1.1" class="ltx_math_unparsed" alttext="100\times" display="inline"><semantics id="S1.p2.1.m1.1a"><mrow id="S1.p2.1.m1.1b"><mn id="S1.p2.1.m1.1.1">100</mn><mo lspace="0.222em" id="S1.p2.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S1.p2.1.m1.1c">100\times</annotation></semantics></math>: from only a few thousand tokens in GPT-3 <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib5" title="" class="ltx_ref">2020</a>)</cite> and Llama 2 <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite> to 1M tokens in Gemini 1.5 Pro <cite class="ltx_cite ltx_citemacro_citep">(Gemini Team, <a href="#bib.bib8" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this paper, we investigate how scaling the number of in-context examples (shots) effects LLM performance across diverse downstream tasks. To gain broad insights into many-shot learning, we analyze its effects on a variety of widely-used tasks: math problem solving using MATH <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite> and GSM8K <cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite>, question-answering <cite class="ltx_cite ltx_citemacro_citep">(GPQA, Rein et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite>, summarization using XSum <cite class="ltx_cite ltx_citemacro_citep">(Narayan et al., <a href="#bib.bib30" title="" class="ltx_ref">2018</a>)</cite> and XLSum <cite class="ltx_cite ltx_citemacro_citep">(Hasan et al., <a href="#bib.bib12" title="" class="ltx_ref">2021</a>)</cite>, algorithmic reasoning <cite class="ltx_cite ltx_citemacro_citep">(Big-Bench Hard, Suzgun et al., <a href="#bib.bib40" title="" class="ltx_ref">2022</a>)</cite>, outcome reward modeling <cite class="ltx_cite ltx_citemacro_citep">(Code verification, Ni et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>)</cite>, low-resource machine translation <cite class="ltx_cite ltx_citemacro_citep">(FLORES, Goyal et al., <a href="#bib.bib10" title="" class="ltx_ref">2022</a>)</cite>, planning <cite class="ltx_cite ltx_citemacro_citep">(Logistics, Seipp et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite>, and sentiment analysis <cite class="ltx_cite ltx_citemacro_citep">(FP, Malo et al., <a href="#bib.bib28" title="" class="ltx_ref">2014</a>)</cite>. Compared to few-shot learning, many-shot learning leads to significant performance gains across these tasks, using several hundreds or thousands of shots with Gemini 1.5 Pro (<a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 1</span></a>). Furthermore, maximum performance is often achieved only once the number of shots reaches up to <em id="S1.p3.1.1" class="ltx_emph ltx_font_italic">hundreds of thousands</em> of tokens (<a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 2</span></a>).</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">While many-shot learning holds significant promise, it can be constrained by the need for high-quality, human-generated outputs. This limitation is amplified in complex reasoning tasks, such as GPQA. To overcome this, we introduce <em id="S1.p4.1.1" class="ltx_emph ltx_font_italic">reinforced</em> ICL and <em id="S1.p4.1.2" class="ltx_emph ltx_font_italic">unsupervised</em> ICL.
Inspired by the efficacy of model-generated solutions for fine-tuning <cite class="ltx_cite ltx_citemacro_citep">(Singh et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite>, Reinforced ICL involves replacing human-written rationales with model-generated ones, filtered via answer correctness, for in-context learning. Inspired by task-recognition view of ICL <cite class="ltx_cite ltx_citemacro_citep">(Xie et al., <a href="#bib.bib46" title="" class="ltx_ref">2021</a>)</cite>, we also introduce Unsupervised ICL where we prompt the model with only problems instead of problem-solution pairs. On reasoning and problem-solving tasks, such as MATH and GPQA, we find that both reinforced and unsupervised ICL with many-shots can be more effective than few-shot ICL with human-generated rationales.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">Finally, we empirically study how the learning dynamics of in-context learning changes from few-shot to the many-shot regime. We find that with sufficient examples, ICL can overcome pre-training biases and solve high-dimensional prediction tasks with numerical inputs, namely sequential parity prediction and linear classification. This suggests the potential of many-shot learning to adapt to new tasks and domains that might be misaligned with an LLM’s training data. Surprisingly, the order of examples can significantly influence ICL performance, even in the many-shot setting. Furthermore, we discover that the widely-used next-token prediction loss may not reliably predict ICL performance on problem-solving and reasoning tasks.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">Our key contributions are as follows:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p"><span id="S1.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Scaling In-Context Learning (ICL)</span> (§<a href="#S3" title="3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>): We systematically evaluate LLM performance at different scales of in-context examples for a wide range of tasks. Our results indicate significant performance leaps for LLMs transitioning from few-shot to many-shot learning regime.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p"><span id="S1.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Reinforced and Unsupervised ICL</span> (§<a href="#S4" title="4 Many-shot Learning without Human-Written Rationales ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>): We find that either using model-generated rationales or only problems can reduce the dependence of many-shot learning on human-generated data.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p"><span id="S1.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Analysing ICL</span> (§<a href="#S5" title="5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>): We find that many-shot ICL can overcome pre-training biases and learn non-natural language prediction tasks, where few-shot ICL struggles.</p>
</div>
</li>
</ul>
</div>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2404.11018/assets/x2.png" id="S1.F2.g1" class="ltx_graphics ltx_img_landscape" width="461" height="155" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 2: </span><span id="S1.F2.2.1" class="ltx_text ltx_font_bold">Context Length</span> for best-performing and the maximum number of shots tested for each task. The horizontal dashed line shows the context length of GPT-3 (2048 tokens), which is representative of typical few-shot prompts tested in the LLM literature. For several tasks, we observed the best-performing shots correspond to the maximum number of shots we tested, which was often limited by the number of available examples for in-context learning. On some tasks (e.g., code verifier, planning), we did observe slight performance deterioration beyond a certain number of shots.</figcaption>
</figure>
</section>
<section id="S2" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Scaling in-context learning</h4>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p1.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">Brown et al. (<a href="#bib.bib5" title="" class="ltx_ref">2020</a>)</cite> reported improved performance as you increase the number of examples used for in-context learning in large language models, and later works corroborated this finding <cite class="ltx_cite ltx_citemacro_citep">(Lu et al., <a href="#bib.bib27" title="" class="ltx_ref">2022</a>)</cite>. However, very few works have explored using a large number of examples (1000 or above) in the prompt. This is likely due to the fact the context lengths in large language models have been quite limited until recently <cite class="ltx_cite ltx_citemacro_citep">(Gemini Team, <a href="#bib.bib8" title="" class="ltx_ref">2024</a>; Anthropic, <a href="#bib.bib3" title="" class="ltx_ref">2024</a>)</cite>. One closely related work to ours is from <cite class="ltx_cite ltx_citemacro_citet">Li et al. (<a href="#bib.bib22" title="" class="ltx_ref">2023</a>)</cite>, who scale the number of examples for in-context learning to 2000. However, <cite class="ltx_cite ltx_citemacro_citet">Li et al. (<a href="#bib.bib22" title="" class="ltx_ref">2023</a>)</cite> use a custom model architecture <cite class="ltx_cite ltx_citemacro_citep">(Zheng et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>)</cite> to achieve long context lengths, and only evaluate models of up to 1.3B parameters, which is several orders of magnitude smaller than state-of-the-art language models, and are ineffective for complex tasks, such as GPQA <cite class="ltx_cite ltx_citemacro_citep">(Rein et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S2.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p2.1" class="ltx_p">Concurrently to our work, <cite class="ltx_cite ltx_citemacro_citet">Anil et al. (<a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite> used many-shot prompting (upto 256 shots) to jailbreak language models. In our work, we focus on a much wider range of tasks, use a lot more examples (up to 8192 shots) and use models with much longer context lengths (up to 1M tokens). Also, we explore mitigations for needing many human-generated examples with many-shot ICL. Furthermore, while <cite class="ltx_cite ltx_citemacro_citet">Anil et al. (<a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite> use many-shot learning to override preferences learned during RLHF phase to elicit the biases stemming from pretraining, our results in §<a href="#S5.SS1" title="5.1 Overcoming Pre-training Biases with Many-Shot ICL ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.1</span></a> demonstrate that we can also override pretraining biases themselves.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Long-context scaling laws</h4>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px2.p1.1" class="ltx_p">Prior works <cite class="ltx_cite ltx_citemacro_citep">(Xiong et al., <a href="#bib.bib47" title="" class="ltx_ref">2023</a>; Anil et al., <a href="#bib.bib2" title="" class="ltx_ref">2024</a>; Kaplan et al., <a href="#bib.bib19" title="" class="ltx_ref">2020</a>; Gemini Team, <a href="#bib.bib8" title="" class="ltx_ref">2024</a>)</cite> have reported smaller next-token prediction loss with longer contexts, which <cite class="ltx_cite ltx_citemacro_citet">Jeon et al. (<a href="#bib.bib17" title="" class="ltx_ref">2024</a>)</cite> also show using theoretical analysis. Our findings confirm this trend for even longer context lengths, but our analysis reveals some of the limitations of using next-token prediction loss as a metric for evaluating long-context performance, as next-token prediction loss continues to go down even as overall performance plateaus.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Learning from self-generated data</h4>

<div id="S2.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px3.p1.1" class="ltx_p">Numerous recent works <cite class="ltx_cite ltx_citemacro_citep">(Gulcehre et al., <a href="#bib.bib11" title="" class="ltx_ref">2023</a>; Yuan et al., <a href="#bib.bib49" title="" class="ltx_ref">2023</a>; Singh et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite> propose fine-tuning language models on self-generated data to improve performance. Their approach consists of (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times. In this work, we extend this idea to the domain of in-context learning, and study the efficacy of Reinforced ICL in reasoning and problem-solving domains.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Self-generated data and in-context learning</h4>

<div id="S2.SS0.SSS0.Px4.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px4.p1.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">Kim et al. (<a href="#bib.bib20" title="" class="ltx_ref">2022</a>)</cite> explore performing in-context learning using self-generated data for classification problems. Their approach consists of generating demonstrations conditioned on the test input for each possible class label, and including these demonstrations in the context when performing the final prediction. In contrast, our approach generates data using the same procedure as <cite class="ltx_cite ltx_citemacro_citet">Singh et al. (<a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite>, does not require access to the test input for data generation, and can be applied to any problem for which we can obtain reliable reward signals (not just classification problems).</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Learning Input-Output Relationships with ICL</h4>

<div id="S2.SS0.SSS0.Px5.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px5.p1.1" class="ltx_p">Numerous works <cite class="ltx_cite ltx_citemacro_citep">(Min et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>; Kossen et al., <a href="#bib.bib21" title="" class="ltx_ref">2023</a>; Yoo et al., <a href="#bib.bib48" title="" class="ltx_ref">2022</a>; Lin and Lee, <a href="#bib.bib25" title="" class="ltx_ref">2024</a>)</cite> have investigated whether LLMs truly learn input-output relationships during in-context learning. <cite class="ltx_cite ltx_citemacro_citet">Min et al. (<a href="#bib.bib29" title="" class="ltx_ref">2022</a>)</cite> found that replacing the ground truth labels in in-context examples with random labels barely effected final performance. Further investigations by <cite class="ltx_cite ltx_citemacro_citet">Yoo et al. (<a href="#bib.bib48" title="" class="ltx_ref">2022</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Kossen et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite> found that this finding does not necessarily hold across tasks and model sizes. In particular, <cite class="ltx_cite ltx_citemacro_citet">Kossen et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>); Lin and Lee (<a href="#bib.bib25" title="" class="ltx_ref">2024</a>)</cite> showed that LLMs can indeed learn input-output relationships via in-context learning, but require more examples in order to do so well. In our work, we extrapolate the trend found in those works to much longer context lengths, showing that pre-training biases can be mostly overcome given enough training examples.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Scaling In-Context Learning</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">In-context learning (ICL) gives LLMs the ability to learn new tasks from examples provided only at inference time. During ICL, the LLM receives a prompt containing a set of input-output examples, also called <em id="S3.p1.1.1" class="ltx_emph ltx_font_italic">shots</em>, that illustrate the desired task. At the end of the prompt, we append a test input and allow the LM to make a prediction just by conditioning on the prompt and predicting the next tokens auto-regressively.
Contrary to task-specific fine-tuning, ICL does not require optimizing any model parameters, allowing LLMs to perform a variety of tasks at inference.</p>
</div>
<section id="S3.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Many-shot ICL</h4>

<div id="S3.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px1.p1.1" class="ltx_p">Recent increase in context windows of LLMs allow using many more shots for ICL than typically used. Compared to few-shot ICL, many-shot learning can use larger fractions of available fine-tuning datasets. This could make task-specific fine-tuning less essential or, in some cases, even unnecessary, allowing LLMs to tackle a wider range of tasks without specialization. Furthermore, exposure to diverse ICL examples can potentially lead to better generalization, the ability to handle more complex problems than what is possible with few-shot ICL, and greater control over model outputs, potentially reducing biases stemming from pre-training.</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluation details</h4>

<div id="S3.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px2.p1.3" class="ltx_p">We use the Gemini 1.5 Pro<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>This corresponds to original version in the Gemini 1.5 Tech Report, released in February 2024. We note that the Gemini 1.5 Pro API now serves a newer version starting from April 2024.</span></span></span> <cite class="ltx_cite ltx_citemacro_citep">(Gemini Team, <a href="#bib.bib8" title="" class="ltx_ref">2024</a>)</cite> MoE model with 1 million token context length, the largest publicly available so far. Unless specified otherwise, we use greedy decoding for evaluation. For reliable results, we randomly sample in-context examples for each <math id="S3.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.1.m1.1a"><mi id="S3.SS0.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.1.m1.1b"><ci id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.1.m1.1c">K</annotation></semantics></math>-shot prompt multiple times using multiple random seeds and report average performance across seeds, along with some visualization for performance on individual seeds. To ensure that using more shots provides additional information, any <math id="S3.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.2.m2.1a"><mi id="S3.SS0.SSS0.Px2.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.2.m2.1b"><ci id="S3.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.2.m2.1c">K</annotation></semantics></math>-shot prompt in our setup includes all in-context examples from prompts with less than <math id="S3.SS0.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.3.m3.1a"><mi id="S3.SS0.SSS0.Px2.p1.3.m3.1.1" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.3.m3.1b"><ci id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.3.m3.1c">K</annotation></semantics></math> examples.</p>
</div>
</section>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Machine Translation (MT)</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">To evaluate how performance scales as we increase the number of in-context examples, we consider machine translation from English to a low-resource target language, where knowledge learned during pre-training may not be sufficient. Following the analysis of <cite class="ltx_cite ltx_citemacro_citet">Robinson et al. (<a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite>, we use the target languages with the largest gap reported between LLMs and Google Translate, namely Tamil and Kurdish, from FLORES-200 MT benchmark <cite class="ltx_cite ltx_citemacro_citep">(NLLB Team, <a href="#bib.bib32" title="" class="ltx_ref">2022</a>)</cite>. We modify the default 1-shot MT prompt from <cite class="ltx_cite ltx_citemacro_citet">Gemini Team (<a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite> to include multiple translation pairs as shots from the FLORES dev split (containing 997 examples). We evaluate performance on the first 150 sentences from the test set using chRF, a standard MT metric based on character and word <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">n</annotation></semantics></math>-gram overlap between model-generated and reference translation. See <a href="#S3.F3" title="Figure 3 ‣ 3.1 Machine Translation (MT) ‣ 3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 3</span></a> for results.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p">Similar to <cite class="ltx_cite ltx_citemacro_citet">Robinson et al. (<a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite>, we observed marginal gains in the few-shot regime from 1-shot to 10-shot. However, when using the entire dev set for many-shot ICL, we observe improvements of 4.5% on Kurdish and 1.5% on Tamil, relative to the 1-shot Gemini prompt. Notably, these many-shot results establish the new-state-of-art for these language pairs, outperforming Google Translate.</p>
</div>
<figure id="S3.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2404.11018/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="221" height="140" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2404.11018/assets/x4.png" id="S3.F3.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="221" height="136" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span><span id="S3.F3.3.1" class="ltx_text ltx_font_bold">Machine Translation</span> (MT). Test Performance improves almost monotonically as we increase the number of MT pairs provided as in-context examples during inference. In both cases, Gemini 1.5 Pro <span id="S3.F3.4.2" class="ltx_text ltx_font_bold">outperforms Google Translate</span>, which obtains a chRF of 40% on Kurdish and 56% on Tamil <cite class="ltx_cite ltx_citemacro_citep">(Robinson et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite>. In terms of context length, 997-shot prompt corresponds to 85.3K and 95.3K tokens for translation to Kurdish and Tamil respectively. See an example prompt in <a href="#A1.F18" title="Figure A.18 ‣ A.5 Example Prompts ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure A.18</span></a>. These results complement the Kalamang translation with Gemini 1.5 <cite class="ltx_cite ltx_citemacro_citep">(Gemini Team, <a href="#bib.bib8" title="" class="ltx_ref">2024</a>)</cite> from a single book.
</figcaption>
</figure>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Summarization</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">We now consider abstractive summarization, which tests the comprehension ability of LLMs to capture essence of the text. Scaling examples for ICL reveals how effectively the model can generalize this ability across different kinds of topics, a realistic scenario where adaptability is crucial. Our evaluation leverages the XSum task from the GEM benchmark <cite class="ltx_cite ltx_citemacro_citep">(Akter et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite>. Using XSum development set examples (news articles and summaries) for many-shot ICL, we also investigate generalization capabilities to XLSum <cite class="ltx_cite ltx_citemacro_citep">(Hasan et al., <a href="#bib.bib12" title="" class="ltx_ref">2021</a>)</cite>. To assess performance, we report performance on the first 150 articles in the test set using ROUGE-L <cite class="ltx_cite ltx_citemacro_citep">(Lin, <a href="#bib.bib24" title="" class="ltx_ref">2004</a>)</cite>, which measures the longest common subsequence between reference and generated summaries.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">As depicted in <a href="#S3.F4" title="Figure 4 ‣ 3.2 Summarization ‣ 3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 4</span></a>, many-shot ICL achieves performance remarkably close to specialized summarization models fine-tuned on XSum (PEGASUS) and XLSum (mT5). Notably, XSum performance declines with more than 50 in-context examples. We surprisingly observed the many-shot prompted model occasionally generating summaries with fabricated dates and times (§<a href="#A1.SS2" title="A.2 Hallucinations with Many-Shot Prompting on Summarization ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.2</span></a>), despite the absence of such data in the in-context summaries. Nonetheless, performance on XLSum monotonically improves with more shots, demonstrating positive transfer from many-shot learning to a related task.</p>
</div>
<figure id="S3.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2404.11018/assets/x5.png" id="S3.F4.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="221" height="146" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2404.11018/assets/x6.png" id="S3.F4.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="221" height="146" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span><span id="S3.F4.2.1" class="ltx_text ltx_font_bold">Summarization</span>. As we increase the number of (news article, summary) pairs from XSum dev set as in-context examples, XSum performance improves up to 50 shots and then deteriorates. In contrast, XLSum performance typically improves with more shots from XSum. The 500-shot prompt corresponds to 205K tokens. PEGASUS <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib50" title="" class="ltx_ref">2020</a>)</cite> and mT5 <cite class="ltx_cite ltx_citemacro_citep">(Hasan et al., <a href="#bib.bib12" title="" class="ltx_ref">2021</a>)</cite> are specialized models fine-tuned for summarization. See an example prompt in <a href="#A1.F19" title="Figure A.19 ‣ A.5 Example Prompts ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure A.19</span></a>.
</figcaption>
</figure>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Planning: Logistics Domain</h3>

<figure id="S3.F5" class="ltx_figure ltx_align_floatright"><img src="/html/2404.11018/assets/x7.png" id="S3.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="438" height="243" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span><span id="S3.F5.2.1" class="ltx_text ltx_font_bold">In-context Planning.</span> Success rate quickly improves with up to 10 shots (37K tokens), followed by saturation up to 400 shots and a sudden performance jump at 800 shots.
As a reference, we report 1-shot GPT-4 results from <cite class="ltx_cite ltx_citemacro_citet">Valmeekam et al. (<a href="#bib.bib42" title="" class="ltx_ref">2024</a>)</cite>.</figcaption>
</figure>
<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">While LLMs have demonstrated remarkable reasoning abilities, their ability to perform commonsense planning seems to be limited <cite class="ltx_cite ltx_citemacro_citep">(Valmeekam et al., <a href="#bib.bib42" title="" class="ltx_ref">2024</a>)</cite>. To this end, we verify whether many-shot ICL can improve their ability to generate simple plans on the Logistics domain, a widely used benchmark. The objective in this domain is to transport packages within cities via trucks, and between cities via airplanes.
We generate a set of planning problems with 2-3 cities, 1-2 packages, 1 truck and airplane per city using a formal planning language (PDDL) generator <cite class="ltx_cite ltx_citemacro_citep">(Seipp et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite>, resulting in 1.3K problems for learning and 600 for evaluation.
To compute optimal solutions for each problem, we use the Fast-Downward planner <cite class="ltx_cite ltx_citemacro_citep">(Helmert, <a href="#bib.bib13" title="" class="ltx_ref">2006</a>)</cite>. See <a href="#A1.F20" title="Figure A.20 ‣ A.5 Example Prompts ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure A.20</span></a> for an example 1-shot prompt.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.1" class="ltx_p">As shown in <a href="#S3.F5" title="Figure 5 ‣ 3.3 Planning: Logistics Domain ‣ 3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 5</span></a>, in-context learning achieves significant improvement in success rate with increasing numbers of shots. While far from state-of-the-art planning approaches (e.g., Fast-Downward), our results demonstrate the potential of many-shot ICL to improve the commonsense planning abilities of LLMs.</p>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Reward Modelling: Learning Code Verifiers In-Context</h3>

<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.1" class="ltx_p">A standard approach to improve LLM reasoning is to use test-time verification <cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib6" title="" class="ltx_ref">2021</a>; Ni et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>; Hosseini et al., <a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite>. Specifically, an LLM generates multiple candidate solutions for a given problem and a learned verifier, also known as an <em id="S3.SS4.p1.1.1" class="ltx_emph ltx_font_italic">outcome reward</em> model, ranks these solutions and selects the best one. Such verifiers are typically obtained by fine-tuning LLMs to predict solution correctness via binary classification. Here, we focus on learning such verifiers in-context.</p>
</div>
<div id="S3.SS4.p2" class="ltx_para">
<p id="S3.SS4.p2.3" class="ltx_p">To create in-context examples for code verification, we utilize correct and incorrect code-based solutions sampled from Gemini 1.0 Pro <cite class="ltx_cite ltx_citemacro_citep">(Gemini Team, <a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite> on the GSM8K train set. In the prompt, each (problem, solution) pair is appended with the question “Is the solution correct?” followed by the <span id="S3.SS4.p2.3.1" class="ltx_text ltx_font_typewriter">Yes</span> or <span id="S3.SS4.p2.3.2" class="ltx_text ltx_font_typewriter">No</span> token according to ground truth correctness. At inference, we modify each test (problem, solution) pair in the same way and record the logit of the <span id="S3.SS4.p2.3.3" class="ltx_text ltx_font_typewriter">Yes</span> and <span id="S3.SS4.p2.3.4" class="ltx_text ltx_font_typewriter">No</span> tokens (denoted by <math id="S3.SS4.p2.1.m1.1" class="ltx_Math" alttext="L_{Yes}" display="inline"><semantics id="S3.SS4.p2.1.m1.1a"><msub id="S3.SS4.p2.1.m1.1.1" xref="S3.SS4.p2.1.m1.1.1.cmml"><mi id="S3.SS4.p2.1.m1.1.1.2" xref="S3.SS4.p2.1.m1.1.1.2.cmml">L</mi><mrow id="S3.SS4.p2.1.m1.1.1.3" xref="S3.SS4.p2.1.m1.1.1.3.cmml"><mi id="S3.SS4.p2.1.m1.1.1.3.2" xref="S3.SS4.p2.1.m1.1.1.3.2.cmml">Y</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.1.m1.1.1.3.1" xref="S3.SS4.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS4.p2.1.m1.1.1.3.3" xref="S3.SS4.p2.1.m1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.1.m1.1.1.3.1a" xref="S3.SS4.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS4.p2.1.m1.1.1.3.4" xref="S3.SS4.p2.1.m1.1.1.3.4.cmml">s</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.1.m1.1b"><apply id="S3.SS4.p2.1.m1.1.1.cmml" xref="S3.SS4.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.1.m1.1.1.1.cmml" xref="S3.SS4.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS4.p2.1.m1.1.1.2.cmml" xref="S3.SS4.p2.1.m1.1.1.2">𝐿</ci><apply id="S3.SS4.p2.1.m1.1.1.3.cmml" xref="S3.SS4.p2.1.m1.1.1.3"><times id="S3.SS4.p2.1.m1.1.1.3.1.cmml" xref="S3.SS4.p2.1.m1.1.1.3.1"></times><ci id="S3.SS4.p2.1.m1.1.1.3.2.cmml" xref="S3.SS4.p2.1.m1.1.1.3.2">𝑌</ci><ci id="S3.SS4.p2.1.m1.1.1.3.3.cmml" xref="S3.SS4.p2.1.m1.1.1.3.3">𝑒</ci><ci id="S3.SS4.p2.1.m1.1.1.3.4.cmml" xref="S3.SS4.p2.1.m1.1.1.3.4">𝑠</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.1.m1.1c">L_{Yes}</annotation></semantics></math>, <math id="S3.SS4.p2.2.m2.1" class="ltx_Math" alttext="L_{No}" display="inline"><semantics id="S3.SS4.p2.2.m2.1a"><msub id="S3.SS4.p2.2.m2.1.1" xref="S3.SS4.p2.2.m2.1.1.cmml"><mi id="S3.SS4.p2.2.m2.1.1.2" xref="S3.SS4.p2.2.m2.1.1.2.cmml">L</mi><mrow id="S3.SS4.p2.2.m2.1.1.3" xref="S3.SS4.p2.2.m2.1.1.3.cmml"><mi id="S3.SS4.p2.2.m2.1.1.3.2" xref="S3.SS4.p2.2.m2.1.1.3.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.2.m2.1.1.3.1" xref="S3.SS4.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="S3.SS4.p2.2.m2.1.1.3.3" xref="S3.SS4.p2.2.m2.1.1.3.3.cmml">o</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.2.m2.1b"><apply id="S3.SS4.p2.2.m2.1.1.cmml" xref="S3.SS4.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.2.m2.1.1.1.cmml" xref="S3.SS4.p2.2.m2.1.1">subscript</csymbol><ci id="S3.SS4.p2.2.m2.1.1.2.cmml" xref="S3.SS4.p2.2.m2.1.1.2">𝐿</ci><apply id="S3.SS4.p2.2.m2.1.1.3.cmml" xref="S3.SS4.p2.2.m2.1.1.3"><times id="S3.SS4.p2.2.m2.1.1.3.1.cmml" xref="S3.SS4.p2.2.m2.1.1.3.1"></times><ci id="S3.SS4.p2.2.m2.1.1.3.2.cmml" xref="S3.SS4.p2.2.m2.1.1.3.2">𝑁</ci><ci id="S3.SS4.p2.2.m2.1.1.3.3.cmml" xref="S3.SS4.p2.2.m2.1.1.3.3">𝑜</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.2.m2.1c">L_{No}</annotation></semantics></math>). To compute the verifier score, we use the <em id="S3.SS4.p2.3.5" class="ltx_emph ltx_font_italic">normalized</em> probability of the <span id="S3.SS4.p2.3.6" class="ltx_text ltx_font_typewriter">Yes</span> token: <math id="S3.SS4.p2.3.m3.6" class="ltx_Math" alttext="\text{I\kern-1.49994ptP}(Yes)=\exp(L_{Yes})/\big{(}\exp(L_{Yes})+\exp(L_{No})\big{)}" display="inline"><semantics id="S3.SS4.p2.3.m3.6a"><mrow id="S3.SS4.p2.3.m3.6.6" xref="S3.SS4.p2.3.m3.6.6.cmml"><mrow id="S3.SS4.p2.3.m3.4.4.1" xref="S3.SS4.p2.3.m3.4.4.1.cmml"><mtext id="S3.SS4.p2.3.m3.4.4.1.3" xref="S3.SS4.p2.3.m3.4.4.1.3a.cmml">IP</mtext><mo lspace="0em" rspace="0em" id="S3.SS4.p2.3.m3.4.4.1.2" xref="S3.SS4.p2.3.m3.4.4.1.2.cmml">​</mo><mrow id="S3.SS4.p2.3.m3.4.4.1.1.1" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS4.p2.3.m3.4.4.1.1.1.2" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.cmml">(</mo><mrow id="S3.SS4.p2.3.m3.4.4.1.1.1.1" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.cmml"><mi id="S3.SS4.p2.3.m3.4.4.1.1.1.1.2" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.2.cmml">Y</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.3.m3.4.4.1.1.1.1.1" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.1.cmml">​</mo><mi id="S3.SS4.p2.3.m3.4.4.1.1.1.1.3" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.3.m3.4.4.1.1.1.1.1a" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.1.cmml">​</mo><mi id="S3.SS4.p2.3.m3.4.4.1.1.1.1.4" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.4.cmml">s</mi></mrow><mo stretchy="false" id="S3.SS4.p2.3.m3.4.4.1.1.1.3" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS4.p2.3.m3.6.6.4" xref="S3.SS4.p2.3.m3.6.6.4.cmml">=</mo><mrow id="S3.SS4.p2.3.m3.6.6.3" xref="S3.SS4.p2.3.m3.6.6.3.cmml"><mrow id="S3.SS4.p2.3.m3.5.5.2.1.1" xref="S3.SS4.p2.3.m3.5.5.2.1.2.cmml"><mi id="S3.SS4.p2.3.m3.1.1" xref="S3.SS4.p2.3.m3.1.1.cmml">exp</mi><mo id="S3.SS4.p2.3.m3.5.5.2.1.1a" xref="S3.SS4.p2.3.m3.5.5.2.1.2.cmml">⁡</mo><mrow id="S3.SS4.p2.3.m3.5.5.2.1.1.1" xref="S3.SS4.p2.3.m3.5.5.2.1.2.cmml"><mo stretchy="false" id="S3.SS4.p2.3.m3.5.5.2.1.1.1.2" xref="S3.SS4.p2.3.m3.5.5.2.1.2.cmml">(</mo><msub id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.cmml"><mi id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.2" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.2.cmml">L</mi><mrow id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.cmml"><mi id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.2" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.2.cmml">Y</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.1" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.1.cmml">​</mo><mi id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.3" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.1a" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.1.cmml">​</mo><mi id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.4" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.4.cmml">s</mi></mrow></msub><mo stretchy="false" id="S3.SS4.p2.3.m3.5.5.2.1.1.1.3" xref="S3.SS4.p2.3.m3.5.5.2.1.2.cmml">)</mo></mrow></mrow><mo id="S3.SS4.p2.3.m3.6.6.3.3" xref="S3.SS4.p2.3.m3.6.6.3.3.cmml">/</mo><mrow id="S3.SS4.p2.3.m3.6.6.3.2.1" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.cmml"><mo maxsize="120%" minsize="120%" id="S3.SS4.p2.3.m3.6.6.3.2.1.2" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.cmml">(</mo><mrow id="S3.SS4.p2.3.m3.6.6.3.2.1.1" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.cmml"><mrow id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.2.cmml"><mi id="S3.SS4.p2.3.m3.2.2" xref="S3.SS4.p2.3.m3.2.2.cmml">exp</mi><mo id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1a" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.2.cmml">⁡</mo><mrow id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.2" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.2.cmml">(</mo><msub id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.cmml"><mi id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.2" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.2.cmml">L</mi><mrow id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.cmml"><mi id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.2" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.2.cmml">Y</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.1" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.3" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.1a" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.4" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.4.cmml">s</mi></mrow></msub><mo stretchy="false" id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.3" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S3.SS4.p2.3.m3.6.6.3.2.1.1.3" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.3.cmml">+</mo><mrow id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.2.cmml"><mi id="S3.SS4.p2.3.m3.3.3" xref="S3.SS4.p2.3.m3.3.3.cmml">exp</mi><mo id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1a" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.2.cmml">⁡</mo><mrow id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.2.cmml"><mo stretchy="false" id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.2" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.2.cmml">(</mo><msub id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.cmml"><mi id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.2" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.2.cmml">L</mi><mrow id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.cmml"><mi id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.2" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.1" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.1.cmml">​</mo><mi id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.3" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.3.cmml">o</mi></mrow></msub><mo stretchy="false" id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.3" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.2.cmml">)</mo></mrow></mrow></mrow><mo maxsize="120%" minsize="120%" id="S3.SS4.p2.3.m3.6.6.3.2.1.3" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.3.m3.6b"><apply id="S3.SS4.p2.3.m3.6.6.cmml" xref="S3.SS4.p2.3.m3.6.6"><eq id="S3.SS4.p2.3.m3.6.6.4.cmml" xref="S3.SS4.p2.3.m3.6.6.4"></eq><apply id="S3.SS4.p2.3.m3.4.4.1.cmml" xref="S3.SS4.p2.3.m3.4.4.1"><times id="S3.SS4.p2.3.m3.4.4.1.2.cmml" xref="S3.SS4.p2.3.m3.4.4.1.2"></times><ci id="S3.SS4.p2.3.m3.4.4.1.3a.cmml" xref="S3.SS4.p2.3.m3.4.4.1.3"><mtext id="S3.SS4.p2.3.m3.4.4.1.3.cmml" xref="S3.SS4.p2.3.m3.4.4.1.3">IP</mtext></ci><apply id="S3.SS4.p2.3.m3.4.4.1.1.1.1.cmml" xref="S3.SS4.p2.3.m3.4.4.1.1.1"><times id="S3.SS4.p2.3.m3.4.4.1.1.1.1.1.cmml" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.1"></times><ci id="S3.SS4.p2.3.m3.4.4.1.1.1.1.2.cmml" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.2">𝑌</ci><ci id="S3.SS4.p2.3.m3.4.4.1.1.1.1.3.cmml" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.3">𝑒</ci><ci id="S3.SS4.p2.3.m3.4.4.1.1.1.1.4.cmml" xref="S3.SS4.p2.3.m3.4.4.1.1.1.1.4">𝑠</ci></apply></apply><apply id="S3.SS4.p2.3.m3.6.6.3.cmml" xref="S3.SS4.p2.3.m3.6.6.3"><divide id="S3.SS4.p2.3.m3.6.6.3.3.cmml" xref="S3.SS4.p2.3.m3.6.6.3.3"></divide><apply id="S3.SS4.p2.3.m3.5.5.2.1.2.cmml" xref="S3.SS4.p2.3.m3.5.5.2.1.1"><exp id="S3.SS4.p2.3.m3.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1"></exp><apply id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.cmml" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.1.cmml" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1">subscript</csymbol><ci id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.2.cmml" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.2">𝐿</ci><apply id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.cmml" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3"><times id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.1.cmml" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.1"></times><ci id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.2.cmml" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.2">𝑌</ci><ci id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.3.cmml" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.3">𝑒</ci><ci id="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.4.cmml" xref="S3.SS4.p2.3.m3.5.5.2.1.1.1.1.3.4">𝑠</ci></apply></apply></apply><apply id="S3.SS4.p2.3.m3.6.6.3.2.1.1.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1"><plus id="S3.SS4.p2.3.m3.6.6.3.2.1.1.3.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.3"></plus><apply id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.2.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1"><exp id="S3.SS4.p2.3.m3.2.2.cmml" xref="S3.SS4.p2.3.m3.2.2"></exp><apply id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.2.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.2">𝐿</ci><apply id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3"><times id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.1.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.1"></times><ci id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.2.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.2">𝑌</ci><ci id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.3.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.3">𝑒</ci><ci id="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.4.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.1.1.1.1.3.4">𝑠</ci></apply></apply></apply><apply id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.2.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1"><exp id="S3.SS4.p2.3.m3.3.3.cmml" xref="S3.SS4.p2.3.m3.3.3"></exp><apply id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.1.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1">subscript</csymbol><ci id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.2.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.2">𝐿</ci><apply id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3"><times id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.1.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.1"></times><ci id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.2.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.2">𝑁</ci><ci id="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.3.cmml" xref="S3.SS4.p2.3.m3.6.6.3.2.1.1.2.1.1.1.3.3">𝑜</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.3.m3.6c">\text{I\kern-1.49994ptP}(Yes)=\exp(L_{Yes})/\big{(}\exp(L_{Yes})+\exp(L_{No})\big{)}</annotation></semantics></math>. To evaluate verifier performance, we perform best-of-4 selection using the verifier score on 200 problems from GSM8K test set with Gemini 1.0 solutions.</p>
</div>
<figure id="S3.F6" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S3.F6.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_top" style="width:212.5pt;">
<img src="/html/2404.11018/assets/x8.png" id="S3.F6.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="280" alt="Refer to caption">
</div>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S3.F6.2" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_top" style="width:212.5pt;">
<img src="/html/2404.11018/assets/x9.png" id="S3.F6.2.g1" class="ltx_graphics ltx_img_landscape" width="461" height="280" alt="Refer to caption">
</div>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 6: </span><span id="S3.F6.9.1" class="ltx_text ltx_font_bold">Learning Verifiers In-Context </span> for checking correctness of GSM8K code solutions. Error bars denotes standard error of mean over 3 seeds. See Figure <a href="#A1.F22" title="Figure A.22 ‣ A.5 Example Prompts ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.22</span></a> for a 2-shot prompt example. <span id="S3.F6.10.2" class="ltx_text ltx_font_bold">Best-of-N accuracy</span>. (Left) Average accuracy of top-ranked code solutions (among 4 solutions) based on the verifier score on 200 GSM8K test problems. Best-of-4 selection with 128-shot bridges the gap between Pass@1 accuracy of 77.25% and Pass@4 accuracy of 90% with Gemini 1.0 Pro model. <span id="S3.F6.11.3" class="ltx_text ltx_font_bold">Verifier Confidence</span>. (Right) Conditional Probabilities of the <span id="S3.F6.12.4" class="ltx_text ltx_font_typewriter">Yes</span> token <math id="S3.F6.4.m1.1" class="ltx_Math" alttext="\text{I\kern-1.49994ptP}(Yes)" display="inline"><semantics id="S3.F6.4.m1.1b"><mrow id="S3.F6.4.m1.1.1" xref="S3.F6.4.m1.1.1.cmml"><mtext id="S3.F6.4.m1.1.1.3" xref="S3.F6.4.m1.1.1.3a.cmml">IP</mtext><mo lspace="0em" rspace="0em" id="S3.F6.4.m1.1.1.2" xref="S3.F6.4.m1.1.1.2.cmml">​</mo><mrow id="S3.F6.4.m1.1.1.1.1" xref="S3.F6.4.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.F6.4.m1.1.1.1.1.2" xref="S3.F6.4.m1.1.1.1.1.1.cmml">(</mo><mrow id="S3.F6.4.m1.1.1.1.1.1" xref="S3.F6.4.m1.1.1.1.1.1.cmml"><mi id="S3.F6.4.m1.1.1.1.1.1.2" xref="S3.F6.4.m1.1.1.1.1.1.2.cmml">Y</mi><mo lspace="0em" rspace="0em" id="S3.F6.4.m1.1.1.1.1.1.1" xref="S3.F6.4.m1.1.1.1.1.1.1.cmml">​</mo><mi id="S3.F6.4.m1.1.1.1.1.1.3" xref="S3.F6.4.m1.1.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.F6.4.m1.1.1.1.1.1.1b" xref="S3.F6.4.m1.1.1.1.1.1.1.cmml">​</mo><mi id="S3.F6.4.m1.1.1.1.1.1.4" xref="S3.F6.4.m1.1.1.1.1.1.4.cmml">s</mi></mrow><mo stretchy="false" id="S3.F6.4.m1.1.1.1.1.3" xref="S3.F6.4.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.F6.4.m1.1c"><apply id="S3.F6.4.m1.1.1.cmml" xref="S3.F6.4.m1.1.1"><times id="S3.F6.4.m1.1.1.2.cmml" xref="S3.F6.4.m1.1.1.2"></times><ci id="S3.F6.4.m1.1.1.3a.cmml" xref="S3.F6.4.m1.1.1.3"><mtext id="S3.F6.4.m1.1.1.3.cmml" xref="S3.F6.4.m1.1.1.3">IP</mtext></ci><apply id="S3.F6.4.m1.1.1.1.1.1.cmml" xref="S3.F6.4.m1.1.1.1.1"><times id="S3.F6.4.m1.1.1.1.1.1.1.cmml" xref="S3.F6.4.m1.1.1.1.1.1.1"></times><ci id="S3.F6.4.m1.1.1.1.1.1.2.cmml" xref="S3.F6.4.m1.1.1.1.1.1.2">𝑌</ci><ci id="S3.F6.4.m1.1.1.1.1.1.3.cmml" xref="S3.F6.4.m1.1.1.1.1.1.3">𝑒</ci><ci id="S3.F6.4.m1.1.1.1.1.1.4.cmml" xref="S3.F6.4.m1.1.1.1.1.1.4">𝑠</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F6.4.m1.1d">\text{I\kern-1.49994ptP}(Yes)</annotation></semantics></math> from the verifier, averaged over all correct and incorrect solutions on test problems.
</figcaption>
</figure>
<div id="S3.SS4.p3" class="ltx_para">
<p id="S3.SS4.p3.1" class="ltx_p">As shown in <a href="#S3.F6" title="Figure 6 ‣ 3.4 Reward Modelling: Learning Code Verifiers In-Context ‣ 3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 6</span></a> (left), best-of-4 accuracy with the few-shot prompted verifier significantly improves above pass@1 accuracy with 16 or more in-context examples. Along with an accuracy improvement, the probabilities of the <span id="S3.SS4.p3.1.1" class="ltx_text ltx_font_typewriter">Yes</span> token conditioned on ground-truth correct and incorrect solutions separate with increasing the number of shots up to 256, as shown in <a href="#S3.F6" title="Figure 6 ‣ 3.4 Reward Modelling: Learning Code Verifiers In-Context ‣ 3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 6</span></a> (right). Overall, these results show a proof-of-concept that the Gemini model becomes better at verifying correctness of solutions with many-shot ICL.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Many-shot Learning without Human-Written Rationales</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">Many-shot ICL could potentially be limited by the availability of high-quality human-generated rationales or demonstrations. This is particularly challenging for complex reasoning tasks, such as GPQA <cite class="ltx_cite ltx_citemacro_citep">(Rein et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite>, where human-generated rationales require significant resources and expert knowledge. In this work, we explore two approaches for addressing this issue.</p>
</div>
<section id="S4.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Reinforced ICL</h4>

<div id="S4.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS0.SSS0.Px1.p1.1" class="ltx_p">Recent work <cite class="ltx_cite ltx_citemacro_citep">(Singh et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>; Gulcehre et al., <a href="#bib.bib11" title="" class="ltx_ref">2023</a>)</cite> has introduced Reinforced Self-Training (ReST). In particular, <cite class="ltx_cite ltx_citemacro_citet">Singh et al. (<a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite> demonstrated that fine-tuning using model-generated rationales can be more effective than human-generated rationales for problem-solving tasks, and can be viewed as applying expectation-maximization for reinforcement
learning.
We introduce Reinforced ICL, where we use model-generated rationales for in-context learning.
To obtain model-generated rationales, we use a few-shot or zero-shot chain-of-thought prompt as a starting point to sample multiple rationales for each training problem.
Then, we select rationales that obtain the correct final answer (we assume access to ground truth final answers), and arrange them into in-context examples containing (problem, rationale) pairs.
</p>
</div>
<div id="S4.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S4.SS0.SSS0.Px1.p2.1" class="ltx_p">One potential issue with model-generated rationales is that of false positives: it is possible for an incorrect reasoning chain to lead to the correct final answer, and fine-tuning or prompting using such a reasoning chain would typically harm performance.
Nevertheless, as we discuss in later sections, we often find model-generated rationales to be at least as effective human-written rationales.</p>
</div>
</section>
<section id="S4.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Unsupervised ICL</h4>

<div id="S4.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS0.SSS0.Px2.p1.1" class="ltx_p">Motivated by encouraging results with Reinforced ICL, we go one step further: what if we removed rationales from the many-shot prompt altogether, and prompt the model only with inputs (e.g., problems)?
Specifically, the Unsupervised ICL prompt consists of: 1) a preamble, such as, “You will be provided questions similar to the ones below:”, 2) a list of unsolved inputs or problems, and 3) a zero-shot instruction or a few-shot prompt with outputs for the desired output format. See Appendix <a href="#A1.SS6" title="A.6 Prompts for Unsupervised ICL ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.6</span></a> for the exact prompts we use.</p>
</div>
<div id="S4.SS0.SSS0.Px2.p2" class="ltx_para">
<p id="S4.SS0.SSS0.Px2.p2.1" class="ltx_p">One hypothesis for how many-shot unsupervised ICL might surpass few-shot learning with human demonstrations is that, when the LLM already possesses the required knowledge to solve a task, any information inserted in the prompt that can narrow down what knowledge is needed for the task becomes helpful. This would be consistent with the view that ICL simply “locates” latent concepts (e.g., math problem-solving) the LLM acquired during pre-training <cite class="ltx_cite ltx_citemacro_citep">(Xie et al., <a href="#bib.bib46" title="" class="ltx_ref">2021</a>; Hendel et al., <a href="#bib.bib14" title="" class="ltx_ref">2023</a>; Wang et al., <a href="#bib.bib43" title="" class="ltx_ref">2024</a>)</cite>. As such, any of the prompt components – inputs, outputs, and their mapping – can help locate such concepts. While Unsupervised ICL is broadly applicable, it may not perform well when the outputs are critical for specifying the task, for example, see <a href="#A1.F17" title="Figure A.17 ‣ A.3 Additional Results for Reinforced and Unsupervised ICL ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure A.17</span></a> for negative results low-resource machine translation.</p>
</div>
</section>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Problem-solving: Hendrycks MATH &amp; GSM8K</h3>

<figure id="S4.F7" class="ltx_figure"><img src="/html/2404.11018/assets/x10.png" id="S4.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="160" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span><span id="S4.F7.4.1" class="ltx_text ltx_font_bold">Many-shot Reinforced and Unsupervised ICL for problem-solving</span> generally outperform ICL with ground-truth MATH solutions. <span id="S4.F7.5.2" class="ltx_text ltx_font_bold">MATH</span>. (Left) The bar plots depict the average performance across five random seeds on the MATH500 test set. Each random seed (denoted by the dots) corresponds to a different subset of problems along with ground truth or model-generated solutions (if any) in the prompt. <span id="S4.F7.6.3" class="ltx_text ltx_font_bold">Transfer to GSM8K</span>. (Right) We see that the prompt obtained from MATH transfers well to the GSM8K test split containing 500 problems. Our results with many-shot ICL outperform the 4-shot Minerva prompt, which obtains a test accuracy of 55.7% on MATH500 and 90.6% on GSM8K.</figcaption>
</figure>
<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We evaluate Reinforced and Unsupervised ICL on the Hendrycks MATH dataset <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite>, which consists of challenging high school competition-level mathematics problems. We use the MATH500 test set from <cite class="ltx_cite ltx_citemacro_citet">Lightman et al. (<a href="#bib.bib23" title="" class="ltx_ref">2023</a>)</cite> to report performance, and our 4-shot MATH prompt for data generation can be found in the Appendix (Figure <a href="#A1.F23" title="Figure A.23 ‣ A.5 Example Prompts ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.23</span></a>). For Unsupervised ICL, we append this 4-shot prompt after the unsolved problems (see Figure <a href="#A1.F25" title="Figure A.25 ‣ A.6 Prompts for Unsupervised ICL ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.25</span></a> for complete prompt). For comparison, we also evaluate ICL with human-written solutions (ground-truth) from the MATH training set.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p">Our results are shown in the Figure <a href="#S4.F7" title="Figure 7 ‣ 4.1 Problem-solving: Hendrycks MATH &amp; GSM8K ‣ 4 Many-shot Learning without Human-Written Rationales ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> (left). On MATH500, both Reinforced and Unsupervised ICL outperforms ICL with ground-truth solutions in both the few-shot and many-shot regime. For ICL, we observe that the performance improves with more examples in the prompt up to a point, and then declines (with the peak being at about 125 examples). Performance for Reinforced ICL also improves with the number of examples, and reaches a plateau at around 25 examples (while being about 5% higher than ICL), and unlike ICL, we don’t see a significant drop in performance even for a very large number of examples in the context. Notably, many-shot ICL achieves comparable or superior performance when using only problems compared to using problems with solutions. This suggests solutions may be redundant for eliciting problem-solving via in-context learning on this domain, potentially due to the math-related data seen during pretraining.</p>
</div>
<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Can many-shot ICL enable out-of-distribution generalization?</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">Singh et al. (<a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite> found that fine-tuning a model on model-generated solutions from MATH resulted in improved test performance on GSM8K <cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite>, which has a different distribution of problems than MATH. Here, we investigate whether many-shot ICL also improves transfer performance on GSM8K, indicating an improvement in general problem-solving abilities from in-context learning. Our results in Figure <a href="#S4.F7" title="Figure 7 ‣ 4.1 Problem-solving: Hendrycks MATH &amp; GSM8K ‣ 4 Many-shot Learning without Human-Written Rationales ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> (right) show that this is indeed the case – Reinforced ICL with MATH prompts excels on GSM8K, outperforming ICL with ground truth MATH solutions as well as Unsupervised ICL in the many-shot setting with at least 25 shots. This indicates that model-generated solutions <em id="S4.SS1.SSS0.Px1.p1.1.1" class="ltx_emph ltx_font_italic">can</em> enable better generalization than just using problems or combining them with ground-truth solutions for ICL.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Question Answering: Google-Proof QA (GPQA)</h3>

<figure id="S4.F8" class="ltx_figure"><img src="/html/2404.11018/assets/x11.png" id="S4.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="452" height="127" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span><span id="S4.F8.2.1" class="ltx_text ltx_font_bold">Many-shot Reinforced and Unsupervised ICL for GPQA</span>. The baseline zero-shot prompt, which is used for generating rationales for Reinforced ICL and appended to the prompt for Unsupervised ICL, obtains a performance of 38.8%.
The average test accuracy with 125-shot prompt with both ground-truth or model-generated rationales surpass the 40.4% obtained by Claude-3 Sonnet. As we vary the number of shots, while Unsupervised ICL matches or outperforms the zero-shot prompt, Reinforced ICL consistently outperforms it.
</figcaption>
</figure>
<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">GPQA <cite class="ltx_cite ltx_citemacro_citep">(Rein et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite> is a multiple-choice QA benchmark, with difficult questions focused on graduate-level reasoning in biology, physics, and chemistry. Following Claude-3 <cite class="ltx_cite ltx_citemacro_citep">(Anthropic, <a href="#bib.bib3" title="" class="ltx_ref">2024</a>)</cite>, we use the diamond split (198 problems) for evaluation. This split focuses on questions where domain experts agree but experts in other domains struggle despite extended effort and internet access. Remaining 250 questions in non-diamond split are used for many-shot ICL with and without human-written rationales. For Reinforced ICL, we use a zero-shot prompt (<a href="#A1.F21" title="Figure A.21 ‣ A.5 Example Prompts ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure A.21</span></a>) to generate multiple rationales on the non-diamond split, solving 129 problems. We also append this zero-shot prompt after the GPQA problems for specifying output format for Unsupervised ICL.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p">As shown in <a href="#S4.F8" title="Figure 8 ‣ 4.2 Question Answering: Google-Proof QA (GPQA) ‣ 4 Many-shot Learning without Human-Written Rationales ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 8</span></a>, average test accuracy with ground-truth rationales improves substantially from 5 shots to 125 shots, with the best-performing 125-shot prompt nearly matching the accuracy of the state-of-the-art Claude-3 Opus. However, we do observe a performance degradation with 250 shots. Moreover, Reinforced ICL results indicate that model-generated rationales on GPQA seem to be better than ground-truth rationales up to 25 shots, while resulting in similar performance with more shots. Additionally, Unsupervised ICL does not follow any systematic trend: it sometimes performs better ICL with ground-truth rationales depending on the number of shots, but generally underperforms Reinforced ICL. As noted in <cite class="ltx_cite ltx_citemacro_citet">Anthropic (<a href="#bib.bib3" title="" class="ltx_ref">2024</a>)</cite>, GPQA is a small evaluation dataset and has an inherent higher variance across different runs, which might explain the non-systematic trends.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Algorithmic and Symbolic Reasoning: Big-Bench Hard</h3>

<figure id="S4.F9" class="ltx_figure"><img src="/html/2404.11018/assets/x12.png" id="S4.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="452" height="160" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span><span id="S4.F9.2.1" class="ltx_text ltx_font_bold">BIG-Bench Hard</span>. Reinforced ICL’s performance over number of shots, averaged across five random seeds. The error bars denote standard deviation. Reinforced ICL outperforms the human-written chain-of-thought prompt for all tasks. Averaged across tasks, chain-of-thought prompting using human-written prompts gets a success rate of 72.1%, while Reinforced ICL gets 83%.</figcaption>
</figure>
<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">We now evaluate Reinforced ICL on BIG-Bench Hard <cite class="ltx_cite ltx_citemacro_citep">(Suzgun et al., <a href="#bib.bib40" title="" class="ltx_ref">2022</a>)</cite>, a suite of challenging algorithmic reasoning tasks. To reduce the impact of false positives, we select 8 tasks out of 23 in BIG-Bench Hard for which the likelihood of getting a false positive is low: either the answer string is long, or the number of options for each question is large (at least 6). We use the standard 3-shot CoT prompt from <cite class="ltx_cite ltx_citemacro_citet">Suzgun et al. (<a href="#bib.bib40" title="" class="ltx_ref">2022</a>)</cite> to sample 10 rationales per problem from a training set of 150 problem at a temperature of 1.0. We filter the rationales based on final answer correctness and arrange them into prompts containing 3 to 100 (problem, rationale) pairs. We evaluate test performance on a held-out set of 100 problems.</p>
</div>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.1" class="ltx_p">As shown in Figure <a href="#S4.F9" title="Figure 9 ‣ 4.3 Algorithmic and Symbolic Reasoning: Big-Bench Hard ‣ 4 Many-shot Learning without Human-Written Rationales ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>, Reinforced ICL strongly outperforms the standard 3-shot CoT prompt for almost all tasks, and the performance generally improves monotonically with the number of prompts for 7 out of 8 tasks.
For a few tasks, Reinforced ICL outperforms the human-written 3-shot prompt even in the 3-shot setting. This result suggests that model-generated rationales can <em id="S4.SS3.p2.1.1" class="ltx_emph ltx_font_italic">s</em>ometimes outperform human-written rationales even when controlling for the amount of data, mirroring the results reported by <cite class="ltx_cite ltx_citemacro_citet">Singh et al. (<a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite> for fine-tuning.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Analyzing Many-Shot In-Context Learning</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">In this section, we study how ICL behavior changes from few-shot to the many-shot regime.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Overcoming Pre-training Biases with Many-Shot ICL</h3>

<figure id="S5.F10" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2404.11018/assets/x13.png" id="S5.F10.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="221" height="127" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2404.11018/assets/x14.png" id="S5.F10.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="221" height="134" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span><span id="S5.F10.4.1" class="ltx_text ltx_font_bold">Overcoming Pre-Training Bias with Many-Shot ICL.</span>
(Left) <span id="S5.F10.5.2" class="ltx_text ltx_font_bold">Many-shot ICL overcomes label flips</span>: Test accuracy for sentiment analysis typically improves with more training shots. Flipped and abstract labels eventually approaching the performance of default labels. (Right) <span id="S5.F10.6.3" class="ltx_text ltx_font_bold">Confidence shift in overcoming bias</span>. For flipped and abstract labels, model confidence in its predicted sentiment labels initially drops, then sharply increases with more training shots to similar value, suggesting a period of overcoming pre-training bias. See §<a href="#S5.SS1" title="5.1 Overcoming Pre-training Biases with Many-Shot ICL ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.1</span></a> for more details.
</figcaption>
</figure>
<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">While LLMs demonstrate in-context learning of novel tasks, <cite class="ltx_cite ltx_citemacro_citet">Kossen et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite> suggest that ICL may have difficulty unlearning biases derived from pre-training data. Their experiments, however, focused mainly on few-shot ICL due to LLM context length limitations.
Here, we revisit their study using many-shot ICL on the Financial PhraseBank (FP) sentiment analysis dataset <cite class="ltx_cite ltx_citemacro_citep">(Malo et al., <a href="#bib.bib28" title="" class="ltx_ref">2014</a>)</cite>. Like <cite class="ltx_cite ltx_citemacro_citet">Kossen et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite>, we study label relationships that challenge pre-training preferences:</p>
<ul id="S5.I1" class="ltx_itemize">
<li id="S5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i1.p1" class="ltx_para">
<p id="S5.I1.i1.p1.1" class="ltx_p">Flipped Labels: Default labels are rotated, that is, [‘negative’, ‘neutral’, ‘positive’] becomes [ ‘neutral’, ‘positive’, ‘negative’]. This conflicts with sentiment biases an LLM might have learned.</p>
</div>
</li>
<li id="S5.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i2.p1" class="ltx_para">
<p id="S5.I1.i2.p1.1" class="ltx_p">Abstract Labels: Semantically-Unrelated labels [‘A’, ‘B’, ‘C’] are used,
removing any pre-existing sentiment association <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
</li>
</ul>
<p id="S5.SS1.p1.2" class="ltx_p">For ICL shots, we sample examples from the validation set (with replaced labels) to exhibit the input-label relationship. We report the results in <a href="#S5.F10" title="Figure 10 ‣ 5.1 Overcoming Pre-training Biases with Many-Shot ICL ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 10</span></a> (left). With few shots, test accuracy with replacement labels is much lower than with default labels. This suggests that with few-shot ICL, the model struggles to overcome its pre-existing biases from pre-training. However, as the number of shots increases, performance on flipped and abstract labels dramatically improves, approaching that of default labels.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p">We also report model confidence on predicted labels in <a href="#S5.F10" title="Figure 10 ‣ 5.1 Overcoming Pre-training Biases with Many-Shot ICL ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 10</span></a> (right). For default labels, confidence in predicted labels steadily increases with more shots. In contrast, for flipped labels, confidence initially drops then sharply increases before reaching a plateau, suggesting a period of overcoming pre-training bias. We posit that the initial drop in performance and confidence in the few-shot regime from 4 to 8 shots may be attributed to the “early ascent” phenomenon <cite class="ltx_cite ltx_citemacro_citep">(Pan, <a href="#bib.bib33" title="" class="ltx_ref">2023</a>; Lin and Lee, <a href="#bib.bib25" title="" class="ltx_ref">2024</a>)</cite>: a small number of shots may lead to the retrieval of an incorrect skill, which eventually diminishes as task learning takes effect with more shots. Overall, these results indicates that with sufficient in-context examples, LLMs <em id="S5.SS1.p2.1.1" class="ltx_emph ltx_font_italic">can</em> overcome pre-training biases.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Learning Non-Natural Language Tasks</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">We now test many-shot ICL’s ability to learn abstract mathematical functions with numerical inputs.
We focus on parity functions and high-dimensional linear classification; these tasks are appealing because their synthetic nature let us test the generality of ICL and determine its applicability beyond the linguistic domain.</p>
</div>
<figure id="S5.F11" class="ltx_figure"><img src="/html/2404.11018/assets/x15.png" id="S5.F11.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="138" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11: </span><span id="S5.F11.6.1" class="ltx_text ltx_font_bold">In-Context Linear Classification</span>. Test accuracy for 16, 32 and 64 dimensional linear classification problems, averaged across 5 randomly-generated datasets with 25 points per class for each dataset (250 evaluation points total). As we increase the number of shots, the accuracy improves and approximately tracks the performance of the nearest-neighbor baseline trained from scratch on the same data. We use the default implementation of <math id="S5.F11.3.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.F11.3.m1.1b"><mi id="S5.F11.3.m1.1.1" xref="S5.F11.3.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.F11.3.m1.1c"><ci id="S5.F11.3.m1.1.1.cmml" xref="S5.F11.3.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F11.3.m1.1d">k</annotation></semantics></math>-nearest neighbours (with <math id="S5.F11.4.m2.1" class="ltx_Math" alttext="k=5" display="inline"><semantics id="S5.F11.4.m2.1b"><mrow id="S5.F11.4.m2.1.1" xref="S5.F11.4.m2.1.1.cmml"><mi id="S5.F11.4.m2.1.1.2" xref="S5.F11.4.m2.1.1.2.cmml">k</mi><mo id="S5.F11.4.m2.1.1.1" xref="S5.F11.4.m2.1.1.1.cmml">=</mo><mn id="S5.F11.4.m2.1.1.3" xref="S5.F11.4.m2.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.F11.4.m2.1c"><apply id="S5.F11.4.m2.1.1.cmml" xref="S5.F11.4.m2.1.1"><eq id="S5.F11.4.m2.1.1.1.cmml" xref="S5.F11.4.m2.1.1.1"></eq><ci id="S5.F11.4.m2.1.1.2.cmml" xref="S5.F11.4.m2.1.1.2">𝑘</ci><cn type="integer" id="S5.F11.4.m2.1.1.3.cmml" xref="S5.F11.4.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F11.4.m2.1d">k=5</annotation></semantics></math>) from scikit-learn <cite class="ltx_cite ltx_citemacro_citep">(Pedregosa et al., <a href="#bib.bib34" title="" class="ltx_ref">2011</a>)</cite>. See <a href="#A1.F24" title="Figure A.24 ‣ A.5 Example Prompts ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure A.24</span></a> for an example prompt.
</figcaption>
</figure>
<section id="S5.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Linear Classification in High Dimensions</h4>

<div id="S5.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px1.p1.10" class="ltx_p">We first consider linear classification in high dimensions to test the limits of many-shot ICL. Following the setup from <cite class="ltx_cite ltx_citemacro_citet">Wei et al. (<a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite>, we create datasets with <math id="S5.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S5.SS2.SSS0.Px1.p1.1.m1.1a"><mi id="S5.SS2.SSS0.Px1.p1.1.m1.1.1" xref="S5.SS2.SSS0.Px1.p1.1.m1.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px1.p1.1.m1.1b"><ci id="S5.SS2.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.1.m1.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px1.p1.1.m1.1c">N</annotation></semantics></math>-dimensional inputs vectors and their corresponding labels, where each dimension is a random integer in <math id="S5.SS2.SSS0.Px1.p1.2.m2.2" class="ltx_Math" alttext="[1,1000]" display="inline"><semantics id="S5.SS2.SSS0.Px1.p1.2.m2.2a"><mrow id="S5.SS2.SSS0.Px1.p1.2.m2.2.3.2" xref="S5.SS2.SSS0.Px1.p1.2.m2.2.3.1.cmml"><mo stretchy="false" id="S5.SS2.SSS0.Px1.p1.2.m2.2.3.2.1" xref="S5.SS2.SSS0.Px1.p1.2.m2.2.3.1.cmml">[</mo><mn id="S5.SS2.SSS0.Px1.p1.2.m2.1.1" xref="S5.SS2.SSS0.Px1.p1.2.m2.1.1.cmml">1</mn><mo id="S5.SS2.SSS0.Px1.p1.2.m2.2.3.2.2" xref="S5.SS2.SSS0.Px1.p1.2.m2.2.3.1.cmml">,</mo><mn id="S5.SS2.SSS0.Px1.p1.2.m2.2.2" xref="S5.SS2.SSS0.Px1.p1.2.m2.2.2.cmml">1000</mn><mo stretchy="false" id="S5.SS2.SSS0.Px1.p1.2.m2.2.3.2.3" xref="S5.SS2.SSS0.Px1.p1.2.m2.2.3.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px1.p1.2.m2.2b"><interval closure="closed" id="S5.SS2.SSS0.Px1.p1.2.m2.2.3.1.cmml" xref="S5.SS2.SSS0.Px1.p1.2.m2.2.3.2"><cn type="integer" id="S5.SS2.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.2.m2.1.1">1</cn><cn type="integer" id="S5.SS2.SSS0.Px1.p1.2.m2.2.2.cmml" xref="S5.SS2.SSS0.Px1.p1.2.m2.2.2">1000</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px1.p1.2.m2.2c">[1,1000]</annotation></semantics></math>. For each dataset, we randomly sample another <math id="S5.SS2.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S5.SS2.SSS0.Px1.p1.3.m3.1a"><mi id="S5.SS2.SSS0.Px1.p1.3.m3.1.1" xref="S5.SS2.SSS0.Px1.p1.3.m3.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px1.p1.3.m3.1b"><ci id="S5.SS2.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.3.m3.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px1.p1.3.m3.1c">N</annotation></semantics></math>-dimensional vector as the decision boundary and a decision threshold (§<a href="#A1.SS4" title="A.4 Linear Classification: Data Generation ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.4</span></a>). We then provide <math id="S5.SS2.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S5.SS2.SSS0.Px1.p1.4.m4.1a"><mi id="S5.SS2.SSS0.Px1.p1.4.m4.1.1" xref="S5.SS2.SSS0.Px1.p1.4.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px1.p1.4.m4.1b"><ci id="S5.SS2.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.4.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px1.p1.4.m4.1c">K</annotation></semantics></math> N-dimensional points above this threshold and <math id="S5.SS2.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S5.SS2.SSS0.Px1.p1.5.m5.1a"><mi id="S5.SS2.SSS0.Px1.p1.5.m5.1.1" xref="S5.SS2.SSS0.Px1.p1.5.m5.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px1.p1.5.m5.1b"><ci id="S5.SS2.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.5.m5.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px1.p1.5.m5.1c">K</annotation></semantics></math> points below that same
threshold as in-context exemplars, and the model must determine whether unseen N-dimensional points are above or below the threshold (we do not tell the model the equation or the threshold). While <cite class="ltx_cite ltx_citemacro_citet">Wei et al. (<a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite> used only 16 shots per class, we scale ICL up to 2048 shots per class. As shown in Figure <a href="#S5.F11" title="Figure 11 ‣ 5.2 Learning Non-Natural Language Tasks ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>, many-shot ICL substantially outperforms random-chance accuracy and nearly matches the performance of a strong baseline, namely <math id="S5.SS2.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS2.SSS0.Px1.p1.6.m6.1a"><mi id="S5.SS2.SSS0.Px1.p1.6.m6.1.1" xref="S5.SS2.SSS0.Px1.p1.6.m6.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px1.p1.6.m6.1b"><ci id="S5.SS2.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.6.m6.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px1.p1.6.m6.1c">k</annotation></semantics></math>-nearest neighbors. While <math id="S5.SS2.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S5.SS2.SSS0.Px1.p1.7.m7.1a"><mn id="S5.SS2.SSS0.Px1.p1.7.m7.1.1" xref="S5.SS2.SSS0.Px1.p1.7.m7.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px1.p1.7.m7.1b"><cn type="integer" id="S5.SS2.SSS0.Px1.p1.7.m7.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.7.m7.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px1.p1.7.m7.1c">2048</annotation></semantics></math> shots per class perform best when <math id="S5.SS2.SSS0.Px1.p1.8.m8.1" class="ltx_Math" alttext="N=16" display="inline"><semantics id="S5.SS2.SSS0.Px1.p1.8.m8.1a"><mrow id="S5.SS2.SSS0.Px1.p1.8.m8.1.1" xref="S5.SS2.SSS0.Px1.p1.8.m8.1.1.cmml"><mi id="S5.SS2.SSS0.Px1.p1.8.m8.1.1.2" xref="S5.SS2.SSS0.Px1.p1.8.m8.1.1.2.cmml">N</mi><mo id="S5.SS2.SSS0.Px1.p1.8.m8.1.1.1" xref="S5.SS2.SSS0.Px1.p1.8.m8.1.1.1.cmml">=</mo><mn id="S5.SS2.SSS0.Px1.p1.8.m8.1.1.3" xref="S5.SS2.SSS0.Px1.p1.8.m8.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px1.p1.8.m8.1b"><apply id="S5.SS2.SSS0.Px1.p1.8.m8.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.8.m8.1.1"><eq id="S5.SS2.SSS0.Px1.p1.8.m8.1.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.8.m8.1.1.1"></eq><ci id="S5.SS2.SSS0.Px1.p1.8.m8.1.1.2.cmml" xref="S5.SS2.SSS0.Px1.p1.8.m8.1.1.2">𝑁</ci><cn type="integer" id="S5.SS2.SSS0.Px1.p1.8.m8.1.1.3.cmml" xref="S5.SS2.SSS0.Px1.p1.8.m8.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px1.p1.8.m8.1c">N=16</annotation></semantics></math>, we observe slight accuracy decrease beyond <math id="S5.SS2.SSS0.Px1.p1.9.m9.1" class="ltx_Math" alttext="512" display="inline"><semantics id="S5.SS2.SSS0.Px1.p1.9.m9.1a"><mn id="S5.SS2.SSS0.Px1.p1.9.m9.1.1" xref="S5.SS2.SSS0.Px1.p1.9.m9.1.1.cmml">512</mn><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px1.p1.9.m9.1b"><cn type="integer" id="S5.SS2.SSS0.Px1.p1.9.m9.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.9.m9.1.1">512</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px1.p1.9.m9.1c">512</annotation></semantics></math> shots for higher values of <math id="S5.SS2.SSS0.Px1.p1.10.m10.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S5.SS2.SSS0.Px1.p1.10.m10.1a"><mi id="S5.SS2.SSS0.Px1.p1.10.m10.1.1" xref="S5.SS2.SSS0.Px1.p1.10.m10.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px1.p1.10.m10.1b"><ci id="S5.SS2.SSS0.Px1.p1.10.m10.1.1.cmml" xref="S5.SS2.SSS0.Px1.p1.10.m10.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px1.p1.10.m10.1c">N</annotation></semantics></math> (<a href="#S5.F11" title="Figure 11 ‣ 5.2 Learning Non-Natural Language Tasks ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 11</span></a> C, R).</p>
</div>
</section>
<section id="S5.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Sequential Parity</h4>

<div id="S5.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px2.p1.5" class="ltx_p">Parity is a fundamental Boolean function that determines if a binary input sequence contains an even or odd number of 1s. It’s computed by applying the XOR (<math id="S5.SS2.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="\oplus" display="inline"><semantics id="S5.SS2.SSS0.Px2.p1.1.m1.1a"><mo id="S5.SS2.SSS0.Px2.p1.1.m1.1.1" xref="S5.SS2.SSS0.Px2.p1.1.m1.1.1.cmml">⊕</mo><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.p1.1.m1.1b"><csymbol cd="latexml" id="S5.SS2.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.1.m1.1.1">direct-sum</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.p1.1.m1.1c">\oplus</annotation></semantics></math>) operation to all bits in the sequence. Despite their power, transformers trained specifically for in-context learning, struggle to in-context learn the Parity function over 20-digit sequences, achieving near chance-level accuracy <cite class="ltx_cite ltx_citemacro_citep">(Bhattamishra et al., <a href="#bib.bib4" title="" class="ltx_ref">2023</a>)</cite>. In this work, we evaluate how well ICL performs with a pretrained LLM to learn the sequential parity function <math id="S5.SS2.SSS0.Px2.p1.2.m2.8" class="ltx_Math" alttext="f(x)=[f_{1}(x),f_{2}(x),\cdots,f_{n}(x)]" display="inline"><semantics id="S5.SS2.SSS0.Px2.p1.2.m2.8a"><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.8.8" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.cmml"><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.cmml"><mi id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.1.cmml">​</mo><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.3.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.cmml"><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.3.2.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.cmml">(</mo><mi id="S5.SS2.SSS0.Px2.p1.2.m2.1.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.cmml">x</mi><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.3.2.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.cmml">)</mo></mrow></mrow><mo id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.4" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.4.cmml">=</mo><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.4.cmml"><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.4" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.4.cmml">[</mo><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.cmml"><msub id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.cmml"><mi id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.2.cmml">f</mi><mn id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.3" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.1.cmml">​</mo><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.3.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.cmml"><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.3.2.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.cmml">(</mo><mi id="S5.SS2.SSS0.Px2.p1.2.m2.2.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.2.2.cmml">x</mi><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.3.2.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.cmml">)</mo></mrow></mrow><mo id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.5" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.4.cmml">,</mo><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.cmml"><msub id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.cmml"><mi id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.2.cmml">f</mi><mn id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.3" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.1.cmml">​</mo><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.3.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.cmml"><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.3.2.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.cmml">(</mo><mi id="S5.SS2.SSS0.Px2.p1.2.m2.3.3" xref="S5.SS2.SSS0.Px2.p1.2.m2.3.3.cmml">x</mi><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.3.2.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.cmml">)</mo></mrow></mrow><mo id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.6" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.4.cmml">,</mo><mi mathvariant="normal" id="S5.SS2.SSS0.Px2.p1.2.m2.5.5" xref="S5.SS2.SSS0.Px2.p1.2.m2.5.5.cmml">⋯</mi><mo id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.7" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.4.cmml">,</mo><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.cmml"><msub id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.cmml"><mi id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.2.cmml">f</mi><mi id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.3" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.3.cmml">n</mi></msub><mo lspace="0em" rspace="0em" id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.1.cmml">​</mo><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.3.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.cmml"><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.3.2.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.cmml">(</mo><mi id="S5.SS2.SSS0.Px2.p1.2.m2.4.4" xref="S5.SS2.SSS0.Px2.p1.2.m2.4.4.cmml">x</mi><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.3.2.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.8" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.4.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.p1.2.m2.8b"><apply id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8"><eq id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.4.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.4"></eq><apply id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5"><times id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.1"></times><ci id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.5.2">𝑓</ci><ci id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1">𝑥</ci></apply><list id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.4.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3"><apply id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1"><times id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.1"></times><apply id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2"><csymbol cd="ambiguous" id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2">subscript</csymbol><ci id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.2">𝑓</ci><cn type="integer" id="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.3.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.6.6.1.1.1.2.3">1</cn></apply><ci id="S5.SS2.SSS0.Px2.p1.2.m2.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.2.2">𝑥</ci></apply><apply id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2"><times id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.1"></times><apply id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2"><csymbol cd="ambiguous" id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2">subscript</csymbol><ci id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.2">𝑓</ci><cn type="integer" id="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.3.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.7.7.2.2.2.2.3">2</cn></apply><ci id="S5.SS2.SSS0.Px2.p1.2.m2.3.3.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.3.3">𝑥</ci></apply><ci id="S5.SS2.SSS0.Px2.p1.2.m2.5.5.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.5.5">⋯</ci><apply id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3"><times id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.1"></times><apply id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2"><csymbol cd="ambiguous" id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2">subscript</csymbol><ci id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.2">𝑓</ci><ci id="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.3.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.8.8.3.3.3.2.3">𝑛</ci></apply><ci id="S5.SS2.SSS0.Px2.p1.2.m2.4.4.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.4.4">𝑥</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.p1.2.m2.8c">f(x)=[f_{1}(x),f_{2}(x),\cdots,f_{n}(x)]</annotation></semantics></math>, where <math id="S5.SS2.SSS0.Px2.p1.3.m3.2" class="ltx_Math" alttext="x\in\{0,1\}^{n}" display="inline"><semantics id="S5.SS2.SSS0.Px2.p1.3.m3.2a"><mrow id="S5.SS2.SSS0.Px2.p1.3.m3.2.3" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.cmml"><mi id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.2" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.2.cmml">x</mi><mo id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.1" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.1.cmml">∈</mo><msup id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.cmml"><mrow id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.2.2" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.2.1.cmml"><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.2.2.1" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.2.1.cmml">{</mo><mn id="S5.SS2.SSS0.Px2.p1.3.m3.1.1" xref="S5.SS2.SSS0.Px2.p1.3.m3.1.1.cmml">0</mn><mo id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.2.2.2" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.2.1.cmml">,</mo><mn id="S5.SS2.SSS0.Px2.p1.3.m3.2.2" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.2.cmml">1</mn><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.2.2.3" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.2.1.cmml">}</mo></mrow><mi id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.3" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.3.cmml">n</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.p1.3.m3.2b"><apply id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3"><in id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.1.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.1"></in><ci id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.2.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.2">𝑥</ci><apply id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3"><csymbol cd="ambiguous" id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.1.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3">superscript</csymbol><set id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.2.1.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.2.2"><cn type="integer" id="S5.SS2.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.1.1">0</cn><cn type="integer" id="S5.SS2.SSS0.Px2.p1.3.m3.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.2">1</cn></set><ci id="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.3.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.2.3.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.p1.3.m3.2c">x\in\{0,1\}^{n}</annotation></semantics></math> and <math id="S5.SS2.SSS0.Px2.p1.4.m4.3" class="ltx_Math" alttext="f_{i}(x)=x_{1}\oplus x_{2}\cdots\oplus x_{i}\ \forall\ i\in[1,n]" display="inline"><semantics id="S5.SS2.SSS0.Px2.p1.4.m4.3a"><mrow id="S5.SS2.SSS0.Px2.p1.4.m4.3.4" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.cmml"><mrow id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.cmml"><msub id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.cmml"><mi id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.2.cmml">f</mi><mi id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.3" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.1" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.1.cmml">​</mo><mrow id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.3.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.cmml"><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.3.2.1" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.cmml">(</mo><mi id="S5.SS2.SSS0.Px2.p1.4.m4.1.1" xref="S5.SS2.SSS0.Px2.p1.4.m4.1.1.cmml">x</mi><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.3.2.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.cmml">)</mo></mrow></mrow><mo id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.3" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.3.cmml">=</mo><mrow id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.cmml"><msub id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.cmml"><mi id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.2.cmml">x</mi><mn id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.3" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.3.cmml">1</mn></msub><mo id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.1" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.1.cmml">⊕</mo><mrow id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.cmml"><msub id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.cmml"><mi id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.2.cmml">x</mi><mn id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.3" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.1" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.1.cmml">​</mo><mi mathvariant="normal" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.3" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.3.cmml">⋯</mi></mrow><mo id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.1a" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.1.cmml">⊕</mo><mrow id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.cmml"><msub id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.cmml"><mi id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.2.cmml">x</mi><mi id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.3" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.3.cmml">i</mi></msub><mo lspace="0.167em" rspace="0em" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.1" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.1.cmml">​</mo><mrow id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3.cmml"><mo rspace="0.667em" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3.1" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3.1.cmml">∀</mo><mi id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3.2.cmml">i</mi></mrow></mrow></mrow><mo id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.5" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.5.cmml">∈</mo><mrow id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.6.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.6.1.cmml"><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.6.2.1" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.6.1.cmml">[</mo><mn id="S5.SS2.SSS0.Px2.p1.4.m4.2.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.2.2.cmml">1</mn><mo id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.6.2.2" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.6.1.cmml">,</mo><mi id="S5.SS2.SSS0.Px2.p1.4.m4.3.3" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.3.cmml">n</mi><mo stretchy="false" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.6.2.3" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.6.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.p1.4.m4.3b"><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4"><and id="S5.SS2.SSS0.Px2.p1.4.m4.3.4a.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4"></and><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4b.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4"><eq id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.3.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.3"></eq><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2"><times id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.1"></times><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2"><csymbol cd="ambiguous" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2">subscript</csymbol><ci id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.2">𝑓</ci><ci id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.3.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.2.2.3">𝑖</ci></apply><ci id="S5.SS2.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.1.1">𝑥</ci></apply><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4"><csymbol cd="latexml" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.1">direct-sum</csymbol><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2"><csymbol cd="ambiguous" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2">subscript</csymbol><ci id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.2">𝑥</ci><cn type="integer" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.3.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.2.3">1</cn></apply><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3"><times id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.1"></times><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2"><csymbol cd="ambiguous" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2">subscript</csymbol><ci id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.2">𝑥</ci><cn type="integer" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.3.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.2.3">2</cn></apply><ci id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.3.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.3.3">⋯</ci></apply><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4"><times id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.1"></times><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2"><csymbol cd="ambiguous" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2">subscript</csymbol><ci id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.2">𝑥</ci><ci id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.3.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.2.3">𝑖</ci></apply><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3"><csymbol cd="latexml" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3.1">for-all</csymbol><ci id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.4.3.2">𝑖</ci></apply></apply></apply></apply><apply id="S5.SS2.SSS0.Px2.p1.4.m4.3.4c.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4"><in id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.5.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.5"></in><share href="#S5.SS2.SSS0.Px2.p1.4.m4.3.4.4.cmml" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4d.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4"></share><interval closure="closed" id="S5.SS2.SSS0.Px2.p1.4.m4.3.4.6.1.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.4.6.2"><cn type="integer" id="S5.SS2.SSS0.Px2.p1.4.m4.2.2.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.2.2">1</cn><ci id="S5.SS2.SSS0.Px2.p1.4.m4.3.3.cmml" xref="S5.SS2.SSS0.Px2.p1.4.m4.3.3">𝑛</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.p1.4.m4.3c">f_{i}(x)=x_{1}\oplus x_{2}\cdots\oplus x_{i}\ \forall\ i\in[1,n]</annotation></semantics></math>. We report the results in <a href="#S5.F12" title="Figure 12 ‣ Sequential Parity ‣ 5.2 Learning Non-Natural Language Tasks ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 12</span></a>. We see consistent improvement in test accuracy as we increase the number of shots 8192. Performance surpasses a GPT-2 Medium sized transformer <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib35" title="" class="ltx_ref">2019</a>)</cite> trained from scratch on 20<math id="S5.SS2.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.SS2.SSS0.Px2.p1.5.m5.1a"><mo id="S5.SS2.SSS0.Px2.p1.5.m5.1.1" xref="S5.SS2.SSS0.Px2.p1.5.m5.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.p1.5.m5.1b"><times id="S5.SS2.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.5.m5.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.p1.5.m5.1c">\times</annotation></semantics></math> more input-output examples (with no repeated examples; details in Appendix <a href="#A1.SS7" title="A.7 Training GPT-2 from scratch on the sequential parity task ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.7</span></a>).</p>
</div>
<figure id="S5.F12" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S5.F12.2.2" class="ltx_p ltx_figure_panel ltx_minipage ltx_align_middle" style="width:190.8pt;">

<span id="S5.F12.2.2.2.2" class="ltx_tabular ltx_align_middle ltx_framed ltx_framed_rectangle">
<span class="ltx_tbody">
<span id="S5.F12.2.2.2.2.3.1" class="ltx_tr">
<span id="S5.F12.2.2.2.2.3.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S5.F12.2.2.2.2.3.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span><span id="S5.F12.2.2.2.2.3.1.1.2" class="ltx_text" style="font-size:80%;">: 1 0 1 1 0 0 0 1 1 1 0 0 0 0 1 0 0 1 1 1</span></span></span>
<span id="S5.F12.2.2.2.2.4.2" class="ltx_tr">
<span id="S5.F12.2.2.2.2.4.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S5.F12.2.2.2.2.4.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Label</span><span id="S5.F12.2.2.2.2.4.2.1.2" class="ltx_text" style="font-size:80%;">: Odd Odd Even Odd Odd Odd Odd Even</span></span></span>
<span id="S5.F12.2.2.2.2.5.3" class="ltx_tr">
<span id="S5.F12.2.2.2.2.5.3.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S5.F12.2.2.2.2.5.3.1.1" class="ltx_text" style="font-size:80%;">Odd Even Even Even Even Even Odd Odd Odd</span></span></span>
<span id="S5.F12.2.2.2.2.6.4" class="ltx_tr">
<span id="S5.F12.2.2.2.2.6.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S5.F12.2.2.2.2.6.4.1.1" class="ltx_text" style="font-size:80%;">Even Odd Even</span></span></span>
<span id="S5.F12.1.1.1.1.1" class="ltx_tr">
<span id="S5.F12.1.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><math id="S5.F12.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="S5.F12.1.1.1.1.1.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="S5.F12.1.1.1.1.1.1.m1.1.1" xref="S5.F12.1.1.1.1.1.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="S5.F12.1.1.1.1.1.1.m1.1b"><ci id="S5.F12.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F12.1.1.1.1.1.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F12.1.1.1.1.1.1.m1.1c">\cdots</annotation></semantics></math></span></span>
<span id="S5.F12.2.2.2.2.2" class="ltx_tr">
<span id="S5.F12.2.2.2.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><math id="S5.F12.2.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="S5.F12.2.2.2.2.2.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="S5.F12.2.2.2.2.2.1.m1.1.1" xref="S5.F12.2.2.2.2.2.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="S5.F12.2.2.2.2.2.1.m1.1b"><ci id="S5.F12.2.2.2.2.2.1.m1.1.1.cmml" xref="S5.F12.2.2.2.2.2.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F12.2.2.2.2.2.1.m1.1c">\cdots</annotation></semantics></math></span></span>
<span id="S5.F12.2.2.2.2.7.5" class="ltx_tr">
<span id="S5.F12.2.2.2.2.7.5.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S5.F12.2.2.2.2.7.5.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span><span id="S5.F12.2.2.2.2.7.5.1.2" class="ltx_text" style="font-size:80%;">: 0 1 1 0 0 1 1 0 1 1 0 0 1 1 0 0 0 1 1 1</span></span></span>
<span id="S5.F12.2.2.2.2.8.6" class="ltx_tr">
<span id="S5.F12.2.2.2.2.8.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S5.F12.2.2.2.2.8.6.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Label</span><span id="S5.F12.2.2.2.2.8.6.1.2" class="ltx_text" style="font-size:80%;">:</span></span></span>
</span>
</span><span id="S5.F12.2.2.3" class="ltx_text" style="font-size:80%;"></span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S5.F12.3" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_middle" style="width:238.5pt;">
<img src="/html/2404.11018/assets/x16.png" id="S5.F12.3.g1" class="ltx_graphics ltx_img_landscape" width="461" height="246" alt="Refer to caption">
</div>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 12: </span><span id="S5.F12.9.1" class="ltx_text ltx_font_bold">Learning Sequential Parity Function In-context</span>. We report test accuracy over 200 unseen inputs, averaged across 3 seeds. Error bars denote standard error of the mean.
<span id="S5.F12.10.2" class="ltx_text ltx_font_bold">Task Prompt</span>. (Left) Example prompt with input and output labels of the 20-digit Sequential Parity Function. <span id="S5.F12.11.3" class="ltx_text ltx_font_bold">Test accuracy</span> (Right) Many-shot ICL performance improves almost monotonically with the number of shots, surpassing performance of GPT-2 Medium sized transformer trained from scratch for 1 epoch on 20<math id="S5.F12.5.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.F12.5.m1.1b"><mo id="S5.F12.5.m1.1.1" xref="S5.F12.5.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.F12.5.m1.1c"><times id="S5.F12.5.m1.1.1.cmml" xref="S5.F12.5.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.F12.5.m1.1d">\times</annotation></semantics></math> more data.</figcaption>
</figure>
</section>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Is Many-Shot ICL Sensitive to Example Ordering?</h3>

<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">In few-shot in-context learning (ICL), the order of examples within the prompt can significantly impact model performance <cite class="ltx_cite ltx_citemacro_citep">(Lu et al., <a href="#bib.bib26" title="" class="ltx_ref">2021</a>; Xiang et al., <a href="#bib.bib45" title="" class="ltx_ref">2024</a>)</cite>.
Here, we investigate whether such sensitivity to prompt ordering observed in few-shot ICL persists in many-shot scenarios, which remains</p>
</div>
<figure id="S5.F13" class="ltx_figure ltx_align_floatright"><img src="/html/2404.11018/assets/x17.png" id="S5.F13.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="456" height="278" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 13: </span><span id="S5.F13.2.1" class="ltx_text ltx_font_bold">Many-Shot Sensitivity To Example Ordering</span>.
Each colored data point represents a different random ordering of 50 in-context examples provided to Gemini 1.5 Pro.
</figcaption>
</figure>
<div id="S5.SS3.p2" class="ltx_para">
<p id="S5.SS3.p2.1" class="ltx_p">largely unexplored. Specifically, we evaluate ten different random orderings of fixed 50 in-context examples from MATH training split and evaluate performance on the held-out MATH500 test set <cite class="ltx_cite ltx_citemacro_citep">(Lightman et al., <a href="#bib.bib23" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S5.SS3.p3" class="ltx_para">
<p id="S5.SS3.p3.1" class="ltx_p">As <a href="#S5.F13" title="Figure 13 ‣ 5.3 Is Many-Shot ICL Sensitive to Example Ordering? ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 13</span></a> reveals, performance varies significantly across different subareas in MATH500. Strikingly, an ordering that that excels in one subarea may perform poorly in another, for example, the best Geometry ordering yields weak results on Number Theory. This fluctuation results in a smaller variation in average performance compared to individual subareas. Overall, these findings highlight a key challenge in ensuring reliable results with many-shot ICL for long-context models.</p>
</div>
</section>
<section id="S5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.4 </span>NLL May Not Be Predictive of ICL Performance</h3>

<figure id="S5.F14" class="ltx_figure"><img src="/html/2404.11018/assets/x18.png" id="S5.F14.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="452" height="134" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 14: </span><span id="S5.F14.2.1" class="ltx_text ltx_font_bold">Negative Log-Likelihood</span> as a function of number of shots We plot negative log-likelihood (NLL) on ground truth test set solutions for GPQA, MATH and GSM8K. For GPQA and MATH, questions for Reinforced ICL and Unsupervised ICL comes from the training splits of those datasets. We study GSM8K in the transfer setting, i.e. questions for Reinforced ICL and Unsupervised ICL come from MATH. Note that the absolute NLL numbers for ICL and Reinforced ICL are not directly comparable to Unsupervised ICL, since they employ different prompt formats.</figcaption>
</figure>
<div id="S5.SS4.p1" class="ltx_para">
<p id="S5.SS4.p1.1" class="ltx_p">Prior works <cite class="ltx_cite ltx_citemacro_citep">(Xiong et al., <a href="#bib.bib47" title="" class="ltx_ref">2023</a>; Anil et al., <a href="#bib.bib2" title="" class="ltx_ref">2024</a>; Kaplan et al., <a href="#bib.bib19" title="" class="ltx_ref">2020</a>)</cite> have found that the negative log-likelihood (NLL) for ground-truth test outputs, which corresponds to next-token prediction loss, decreases predictably as the context length increases. We confirm this finding for GPQA, Hendrycks MATH and GSM8K with many-shot in-context learning, and report our results in Figure <a href="#S5.F14" title="Figure 14 ‣ 5.4 NLL May Not Be Predictive of ICL Performance ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>. However, we note that NLL trends are not a strong predictor for downstream task performance. For example, the success rate for both MATH and GPQA with ICL decreases after 125 shots (Figure <a href="#S4.F7" title="Figure 7 ‣ 4.1 Problem-solving: Hendrycks MATH &amp; GSM8K ‣ 4 Many-shot Learning without Human-Written Rationales ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>,<a href="#S4.F8" title="Figure 8 ‣ 4.2 Question Answering: Google-Proof QA (GPQA) ‣ 4 Many-shot Learning without Human-Written Rationales ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>), but we do not observe a corresponding increase in the negative log-likelihood in Figure <a href="#S5.F14" title="Figure 14 ‣ 5.4 NLL May Not Be Predictive of ICL Performance ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>.</p>
</div>
<div id="S5.SS4.p2" class="ltx_para">
<p id="S5.SS4.p2.1" class="ltx_p">We also plot negative log-likelihood curves for Reinforced and Unsupervised ICL, and find them to generally have a smaller slope when compared to supervised ICL. Furthermore, we observe that NLL for ICL with ground-truth outputs is much lower compared to model-generated outputs, depsite model-generated outputs often resulting in better performance than ground-truth outputs. These results imply that NLL may not be predictive of downstream ICL performance when using prompts that are out-of-distribution with respect to the test set.</p>
</div>
<div id="S5.SS4.p3" class="ltx_para">
<p id="S5.SS4.p3.1" class="ltx_p">In the GSM8K transfer setting (using MATH problems and solutions to score GSM8K solutions), we see that the change in NLL is close to nil. Once again, these trends do not necessarily reflect the final performance observed in Figure <a href="#S4.F7" title="Figure 7 ‣ 4.1 Problem-solving: Hendrycks MATH &amp; GSM8K ‣ 4 Many-shot Learning without Human-Written Rationales ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, as GSM8K performance continues to increase with more examples in the transfer setting. Finally, while MATH performance for Reinforced and Unsupervised ICL saturates around 25-shot prompts, the NLL continues to decrease with more shots.</p>
</div>
<div id="S5.SS4.p4" class="ltx_para">
<p id="S5.SS4.p4.1" class="ltx_p">Our results lead us to conclude that negative log-likelihood is not a reliable proxy when attempting to predict final performance for problem-solving domains. This makes intuitive sense: for any given problem, there are a large number of potentially correct chain-of-thought paths that the model can take, and calculating the log-likelihood on only one such path may not provide a clear picture for overall model capability. We also explore computing negative log-likelihood on a diverse set of model-generated outputs on MATH, and our findings are presented in Appendix <a href="#A1.SS1" title="A.1 Negative Log-Likelihood on Model-Generated Data ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.1</span></a>.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">We investigated many-shot in-context learning.
We found significant gains in performance when going from few-shot to many-shot in a wide range of tasks, including translation, summarization, planning, reward modeling, mathematical problem solving, scientific question-answering, and algorithmic reasoning.
To overcome the challenges of obtaining a large number of high-quality human-written rationales for many-shot learning, we introduced two new in-context learning regimes: Reinforced ICL and Unsupervised ICL.
We found that, for problem-solving domains where human-generated rationales are expensive to obtain, Reinforced and Unsupervised ICL can obtain strong performance when compared to ICL with human data. We concluded with a set of analysis experiments showing that many-shot learning can overcome pre-training biases, allows learning non-natural language tasks typically difficult for LLMs with few-shot learning, and explored negative log-likelihood trends with respect to context length in the many-shot setting.</p>
</div>
<section id="S6.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Limitations</h4>

<div id="S6.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S6.SS0.SSS0.Px1.p1.1" class="ltx_p">An important limitation of our work is that it uses a single model for all its experiments, the Gemini 1.5 Pro <cite class="ltx_cite ltx_citemacro_citep">(Gemini Team, <a href="#bib.bib8" title="" class="ltx_ref">2024</a>)</cite>.
Future work should focus on evaluating the many-shot in-context learning abilities of a wide range of long context models, as they become available.
Many-shot performance can likely serve as a valuable metric for evaluating the quality of long-context models, going beyond the needle-in-a-haystack test <cite class="ltx_cite ltx_citemacro_citep">(Kamradt, <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite> that is currently popular for this purpose.</p>
</div>
<div id="S6.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S6.SS0.SSS0.Px1.p2.1" class="ltx_p">Another limitation of our work is that we don’t completely understand why performance can sometimes degrades with more examples in the prompt (for example, for MATH).
Our analysis found that negative log-likelihood trends are insufficient to explain this degradation, and future work should focus on investigating new research directions to shed light on the matter.</p>
</div>
</section>
</section>
<section id="Sx1" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">Acknowledgements</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">We would like to thank Gheorghe Comanici for reviewing an early draft of this work. We are also grateful to Doina Precup, Aviral Kumar, Dale Schuurmans, Ankit Anand, Ross Goroshin, Urvashi Singh, and Daniel Toyoma for helpful discussions.</p>
</div>
</section>
<section id="Sx2" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">Contribution Statement</h2>

<div id="Sx2.p1" class="ltx_para">
<p id="Sx2.p1.1" class="ltx_p">RA initiated the project, ran majority of the many-shot experiments and analysis, came up with reinforced ICL, on-boarded collaborators, wrote the initial draft. AS contributed initial infra for experiments on MATH and GSM8K, ran BBH experiments, conducted NLL analysis on problem-solving tasks, and wrote several sections of the paper.</p>
</div>
<div id="Sx2.p2" class="ltx_para">
<p id="Sx2.p2.1" class="ltx_p">LZ contributed results for in-context verifier. BB contributed the planning logistics task. AA helped with GPQA, SC contributed the baseline for parity task and both helped edit the paper. AF and HL provided feedback on an early draft. HL also suggested the unsupervised ICL experiments. Others were involved in project discussions and minor edits to the paper.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography" lang="en">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Akter et al. (2023)</span>
<span class="ltx_bibblock">
S. N. Akter, Z. Yu, A. Muhamed, T. Ou, A. Bäuerle, Á. A. Cabrera,
K. Dholakia, C. Xiong, and G. Neubig.

</span>
<span class="ltx_bibblock">An in-depth look at gemini’s language abilities.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.11444</em>, 2023.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Anil et al. (2024)</span>
<span class="ltx_bibblock">
C. Anil, E. Durmus, M. Sharma, J. Benton, S. Kundu, J. Batson, N. Rimsky,
M. Tong, J. Mu, D. Ford, F. Mosconi, R. Agrawal, R. Schaeffer, N. Bashkansky,
S. Svenningsen, M. Lambert, A. Radhakrishnan, C. Denison, E. J. Hubinger,
Y. Bai, T. Bricken, T. Maxwell, N. Schiefer, J. Sully, A. Tamkin, T. Lanham,
K. Nguyen, T. Korbak, J. Kaplan, D. Ganguli, S. R. Bowman, E. Perez,
R. Grosse, and D. Duvenaud.

</span>
<span class="ltx_bibblock">Many-shot jailbreaking.

</span>
<span class="ltx_bibblock">Technical report, Anthropic, 2024.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Anthropic (2024)</span>
<span class="ltx_bibblock">
Anthropic.

</span>
<span class="ltx_bibblock">The claude 3 model family: Opus, sonnet, haiku.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Technical Report</em>, 2024.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bhattamishra et al. (2023)</span>
<span class="ltx_bibblock">
S. Bhattamishra, A. Patel, P. Blunsom, and V. Kanade.

</span>
<span class="ltx_bibblock">Understanding in-context learning in transformers and llms by
learning to learn discrete functions.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.03016</em>, 2023.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal,
A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal,
A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M.
Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray,
B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and
D. Amodei.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock">In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin,
editors, <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems 33: Annual
Conference on Neural Information Processing Systems 2020, NeurIPS 2020,
December 6-12, 2020, virtual</em>, 2020.

</span>
<span class="ltx_bibblock">URL
<a target="_blank" href="https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. (2021)</span>
<span class="ltx_bibblock">
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert,
J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman.

</span>
<span class="ltx_bibblock">Training verifiers to solve math word problems.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2110.14168</em>, 2021.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gemini Team (2023)</span>
<span class="ltx_bibblock">
G. Gemini Team.

</span>
<span class="ltx_bibblock">Gemini: A family of highly capable multimodal models.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.11805</em>, 2023.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gemini Team (2024)</span>
<span class="ltx_bibblock">
G. Gemini Team.

</span>
<span class="ltx_bibblock">Gemini 1.5: Unlocking multimodal understanding across millions of
tokens of context.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arxiv:2403.05530</em>, 2024.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ghallab et al. (1998)</span>
<span class="ltx_bibblock">
M. Ghallab, A. Howe, C. Knoblock, D. Mcdermott, A. Ram, M. Veloso, D. Weld, and
D. Wilkins.

</span>
<span class="ltx_bibblock">PDDL—The Planning Domain Definition Language, 1998.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goyal et al. (2022)</span>
<span class="ltx_bibblock">
N. Goyal, C. Gao, V. Chaudhary, P. Chen, G. Wenzek, D. Ju, S. Krishnan,
M. Ranzato, F. Guzmán, and A. Fan.

</span>
<span class="ltx_bibblock">The flores-101 evaluation benchmark for low-resource and multilingual
machine translation.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Trans. Assoc. Comput. Linguistics</em>, 10:522–538,
2022.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gulcehre et al. (2023)</span>
<span class="ltx_bibblock">
C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma,
A. Siddhant, A. Ahern, M. Wang, C. Gu, et al.

</span>
<span class="ltx_bibblock">Reinforced self-training (rest) for language modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.08998</em>, 2023.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hasan et al. (2021)</span>
<span class="ltx_bibblock">
T. Hasan, A. Bhattacharjee, M. S. Islam, K. S. Mubasshir, Y. Li, Y. Kang, M. S.
Rahman, and R. Shahriyar.

</span>
<span class="ltx_bibblock">Xl-sum: Large-scale multilingual abstractive summarization for 44
languages.

</span>
<span class="ltx_bibblock">In C. Zong, F. Xia, W. Li, and R. Navigli, editors, <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Findings of
the Association for Computational Linguistics: ACL/IJCNLP 2021, Online
Event, August 1-6, 2021</em>, volume ACL/IJCNLP 2021 of <em id="bib.bib12.2.2" class="ltx_emph ltx_font_italic">Findings of
ACL</em>, pages 4693–4703. Association for Computational Linguistics, 2021.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Helmert (2006)</span>
<span class="ltx_bibblock">
M. Helmert.

</span>
<span class="ltx_bibblock">The fast downward planning system.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Journal of Artificial Intelligence Research</em>, 26:191–246, July 2006.

</span>
<span class="ltx_bibblock">ISSN 1076-9757.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1613/jair.1705" title="" class="ltx_ref">10.1613/jair.1705</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://dx.doi.org/10.1613/jair.1705" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://dx.doi.org/10.1613/jair.1705</a>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendel et al. (2023)</span>
<span class="ltx_bibblock">
R. Hendel, M. Geva, and A. Globerson.

</span>
<span class="ltx_bibblock">In-context learning creates task vectors.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.15916</em>, 2023.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021)</span>
<span class="ltx_bibblock">
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and
J. Steinhardt.

</span>
<span class="ltx_bibblock">Measuring mathematical problem solving with the math dataset.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2103.03874</em>, 2021.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hosseini et al. (2024)</span>
<span class="ltx_bibblock">
A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal.

</span>
<span class="ltx_bibblock">V-star: Training verifiers for self-taught reasoners.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.06457</em>, 2024.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jeon et al. (2024)</span>
<span class="ltx_bibblock">
H. J. Jeon, J. D. Lee, Q. Lei, and B. Van Roy.

</span>
<span class="ltx_bibblock">An information-theoretic analysis of in-context learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.15530</em>, 2024.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kamradt (2023)</span>
<span class="ltx_bibblock">
G. Kamradt.

</span>
<span class="ltx_bibblock">LLMTest_NeedleInAHaystack.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/gkamradt/LLMTest_NeedleInAHaystack" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/gkamradt/LLMTest_NeedleInAHaystack</a>, 2023.

</span>
<span class="ltx_bibblock">Accessed: 2024-04-16.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaplan et al. (2020)</span>
<span class="ltx_bibblock">
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child,
S. Gray, A. Radford, J. Wu, and D. Amodei.

</span>
<span class="ltx_bibblock">Scaling laws for neural language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2001.08361</em>, 2020.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kim et al. (2022)</span>
<span class="ltx_bibblock">
H. J. Kim, H. Cho, J. Kim, T. Kim, K. M. Yoo, and S. Lee.

</span>
<span class="ltx_bibblock">Self-generated in-context learning: Leveraging auto-regressive
language models as a demonstration generator.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2206.08082, 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2206.08082" title="" class="ltx_ref">10.48550/ARXIV.2206.08082</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2206.08082" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2206.08082</a>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kossen et al. (2023)</span>
<span class="ltx_bibblock">
J. Kossen, Y. Gal, and T. Rainforth.

</span>
<span class="ltx_bibblock">In-context learning learns label relationships but is not
conventional learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">The Twelfth International Conference on Learning
Representations</em>, 2023.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023)</span>
<span class="ltx_bibblock">
M. Li, S. Gong, J. Feng, Y. Xu, J. Zhang, Z. Wu, and L. Kong.

</span>
<span class="ltx_bibblock">In-context learning with many demonstration examples.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2302.04931, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2302.04931" title="" class="ltx_ref">10.48550/ARXIV.2302.04931</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2302.04931" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2302.04931</a>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lightman et al. (2023)</span>
<span class="ltx_bibblock">
H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike,
J. Schulman, I. Sutskever, and K. Cobbe.

</span>
<span class="ltx_bibblock">Let’s verify step by step.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2305.20050, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2305.20050" title="" class="ltx_ref">10.48550/ARXIV.2305.20050</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2305.20050" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2305.20050</a>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin (2004)</span>
<span class="ltx_bibblock">
C.-Y. Lin.

</span>
<span class="ltx_bibblock">ROUGE: A package for automatic evaluation of summaries.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Text Summarization Branches Out</em>, pages 74–81, Barcelona,
Spain, July 2004. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/W04-1013" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/W04-1013</a>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin and Lee (2024)</span>
<span class="ltx_bibblock">
Z. Lin and K. Lee.

</span>
<span class="ltx_bibblock">Dual operating modes of in-context learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.18819</em>, 2024.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. (2021)</span>
<span class="ltx_bibblock">
Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp.

</span>
<span class="ltx_bibblock">Fantastically ordered prompts and where to find them: Overcoming
few-shot prompt order sensitivity.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2104.08786</em>, 2021.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. (2022)</span>
<span class="ltx_bibblock">
Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp.

</span>
<span class="ltx_bibblock">Fantastically ordered prompts and where to find them: Overcoming
few-shot prompt order sensitivity.

</span>
<span class="ltx_bibblock">In S. Muresan, P. Nakov, and A. Villavicencio, editors,
<em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin,
Ireland, May 22-27, 2022</em>, pages 8086–8098. Association for Computational
Linguistics, 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.18653/V1/2022.ACL-LONG.556" title="" class="ltx_ref">10.18653/V1/2022.ACL-LONG.556</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2022.acl-long.556" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2022.acl-long.556</a>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Malo et al. (2014)</span>
<span class="ltx_bibblock">
P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala.

</span>
<span class="ltx_bibblock">Good debt or bad debt: Detecting semantic orientations in economic
texts.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Journal of the Association for Information Science and
Technology</em>, 65(4):782–796, 2014.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Min et al. (2022)</span>
<span class="ltx_bibblock">
S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and
L. Zettlemoyer.

</span>
<span class="ltx_bibblock">Rethinking the role of demonstrations: What makes in-context learning
work?

</span>
<span class="ltx_bibblock">In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Proceedings
of the 2022 Conference on Empirical Methods in Natural Language Processing,
EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022</em>, pages
11048–11064. Association for Computational Linguistics, 2022.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Narayan et al. (2018)</span>
<span class="ltx_bibblock">
S. Narayan, S. B. Cohen, and M. Lapata.

</span>
<span class="ltx_bibblock">Don’t give me the details, just the summary! topic-aware
convolutional neural networks for extreme summarization.

</span>
<span class="ltx_bibblock">In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors,
<em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in Natural
Language Processing, Brussels, Belgium, October 31 - November 4, 2018</em>, pages
1797–1807. Association for Computational Linguistics, 2018.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ni et al. (2023)</span>
<span class="ltx_bibblock">
A. Ni, S. Iyer, D. Radev, V. Stoyanov, W.-t. Yih, S. Wang, and X. V. Lin.

</span>
<span class="ltx_bibblock">Lever: Learning to verify language-to-code generation with execution.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pages
26106–26128. PMLR, 2023.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">NLLB Team (2022)</span>
<span class="ltx_bibblock">
M. A. NLLB Team.

</span>
<span class="ltx_bibblock">No language left behind: Scaling human-centered machine translation.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint</em>, 2022.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pan (2023)</span>
<span class="ltx_bibblock">
J. Pan.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">What in-context learning “learns” in-context: Disentangling
task recognition and task learning</em>.

</span>
<span class="ltx_bibblock">PhD thesis, Princeton University, 2023.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pedregosa et al. (2011)</span>
<span class="ltx_bibblock">
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel,
M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al.

</span>
<span class="ltx_bibblock">Scikit-learn: Machine learning in python.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">the Journal of machine Learning research</em>, 12:2825–2830, 2011.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2019)</span>
<span class="ltx_bibblock">
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">OpenAI blog</em>, 1(8):9, 2019.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rein et al. (2023)</span>
<span class="ltx_bibblock">
D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani,
J. Michael, and S. R. Bowman.

</span>
<span class="ltx_bibblock">Gpqa: A graduate-level google-proof q&amp;a benchmark.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.12022</em>, 2023.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Robinson et al. (2023)</span>
<span class="ltx_bibblock">
N. R. Robinson, P. Ogayo, D. R. Mortensen, and G. Neubig.

</span>
<span class="ltx_bibblock">Chatgpt mt: Competitive for high-(but not low-) resource languages.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.07423</em>, 2023.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Seipp et al. (2022)</span>
<span class="ltx_bibblock">
J. Seipp, Á. Torralba, and J. Hoffmann.

</span>
<span class="ltx_bibblock">PDDL generators.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.5281/zenodo.6382173" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.5281/zenodo.6382173</a>, 2022.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Singh et al. (2023)</span>
<span class="ltx_bibblock">
A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, P. J. Liu,
J. Harrison, J. Lee, K. Xu, A. Parisi, et al.

</span>
<span class="ltx_bibblock">Beyond human data: Scaling self-training for problem-solving with
language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.06585</em>, 2023.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Suzgun et al. (2022)</span>
<span class="ltx_bibblock">
M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung,
A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al.

</span>
<span class="ltx_bibblock">Challenging big-bench tasks and whether chain-of-thought can solve
them.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.09261</em>, 2022.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei,
N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.09288</em>, 2023.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Valmeekam et al. (2024)</span>
<span class="ltx_bibblock">
K. Valmeekam, M. Marquez, S. Sreedharan, and S. Kambhampati.

</span>
<span class="ltx_bibblock">On the planning abilities of large language models-a critical
investigation.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2024)</span>
<span class="ltx_bibblock">
X. Wang, W. Zhu, M. Saxon, M. Steyvers, and W. Y. Wang.

</span>
<span class="ltx_bibblock">Large language models are latent variable models: Explaining and
finding good demonstrations for in-context learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2023)</span>
<span class="ltx_bibblock">
J. Wei, J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang,
D. Zhou, et al.

</span>
<span class="ltx_bibblock">Larger language models do in-context learning differently.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.03846</em>, 2023.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiang et al. (2024)</span>
<span class="ltx_bibblock">
Y. Xiang, H. Yan, L. Gui, and Y. He.

</span>
<span class="ltx_bibblock">Addressing order sensitivity of in-context demonstration examples in
causal language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.15637</em>, 2024.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xie et al. (2021)</span>
<span class="ltx_bibblock">
S. M. Xie, A. Raghunathan, P. Liang, and T. Ma.

</span>
<span class="ltx_bibblock">An explanation of in-context learning as implicit bayesian inference.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2111.02080</em>, 2021.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiong et al. (2023)</span>
<span class="ltx_bibblock">
W. Xiong, J. Liu, I. Molybog, H. Zhang, P. Bhargava, R. Hou, L. Martin,
R. Rungta, K. A. Sankararaman, B. Oguz, M. Khabsa, H. Fang, Y. Mehdad,
S. Narang, K. Malik, A. Fan, S. Bhosale, S. Edunov, M. Lewis, S. Wang, and
H. Ma.

</span>
<span class="ltx_bibblock">Effective long-context scaling of foundation models.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2309.16039, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2309.16039" title="" class="ltx_ref">10.48550/ARXIV.2309.16039</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2309.16039" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2309.16039</a>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yoo et al. (2022)</span>
<span class="ltx_bibblock">
K. M. Yoo, J. Kim, H. J. Kim, H. Cho, H. Jo, S. Lee, S. Lee, and T. Kim.

</span>
<span class="ltx_bibblock">Ground-truth labels matter: A deeper look into input-label
demonstrations.

</span>
<span class="ltx_bibblock">In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, <em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Proceedings
of the 2022 Conference on Empirical Methods in Natural Language Processing,
EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022</em>, pages
2422–2437. Association for Computational Linguistics, 2022.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yuan et al. (2023)</span>
<span class="ltx_bibblock">
Z. Yuan, H. Yuan, C. Li, G. Dong, C. Tan, and C. Zhou.

</span>
<span class="ltx_bibblock">Scaling relationship on learning mathematical reasoning with large
language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.01825</em>, 2023.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2020)</span>
<span class="ltx_bibblock">
J. Zhang, Y. Zhao, M. Saleh, and P. Liu.

</span>
<span class="ltx_bibblock">Pegasus: Pre-training with extracted gap-sentences for abstractive
summarization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages
11328–11339. PMLR, 2020.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. (2023)</span>
<span class="ltx_bibblock">
L. Zheng, J. Yuan, C. Wang, and L. Kong.

</span>
<span class="ltx_bibblock">Efficient attention via control variates.

</span>
<span class="ltx_bibblock">In <em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning
Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023</em>. OpenReview.net,
2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/pdf?id=G-uNfHKrj46" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/pdf?id=G-uNfHKrj46</a>.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Appendix</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Negative Log-Likelihood on Model-Generated Data</h3>

<figure id="A1.F15" class="ltx_figure"><img src="/html/2404.11018/assets/x19.png" id="A1.F15.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="452" height="163" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure A.15: </span><span id="A1.F15.2.1" class="ltx_text ltx_font_bold">NLL For Model-Generated Data</span>. We compute negative log-likelihood on model-generated solutions for Hendrycks MATH. We generate solutions using prompts of varying lengths, starting from (model-generated) 4-shot prompts, and going up to 500-shot prompts. We then score the solutions with (model-generated) prompts of different lengths, from 4-shot to 500-shot prompts. We see that, for the solutions generated using the 4-shot prompt, the NLL is lowest for 4-shot and 10-shot scoring prompts, and increases as the length of the scoring prompt increases. Note that the generation and scoring prompts contain different subsets of examples from the training sets (and are randomized for each sample). Generally speaking, we see that the NLL is lowest when the generation and the scoring prompts are of similar lengths, and higher when they are further apart. This reveals to us that prompts of similar lengths induce similar distributions over the outputs, even when the actual contents of the prompt are different. Further, we see that correct and incorrect solutions follow similar curves, and the gap between NLL for correct and incorrect solutions does not follow any specific pattern.</figcaption>
</figure>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Hallucinations with Many-Shot Prompting on Summarization</h3>

<div id="A1.SS2.p1" class="ltx_para">
<p id="A1.SS2.p1.1" class="ltx_p">Here is an example on XSum showing summaries with halluncinated date and time when increasing the number of shots for many-shot learning.
<br class="ltx_break">
<br class="ltx_break"><span id="A1.SS2.p1.1.1" class="ltx_text ltx_font_bold">Reference Summary</span>: Russia’s leading independent polling agency has been labelled a "foreign agent" by the justice ministry and says it cannot now work. 
<br class="ltx_break"><span id="A1.SS2.p1.1.2" class="ltx_text ltx_font_bold">1-shot</span>: The Levada Centre, a Russian organization that conducts political opinion surveys, has been designated as a "foreign agent" by the government. This label restricts the organization’s activities and effectively prevents it from conducting independent polls, particularly in relation to the upcoming parliamentary elections. Critics view this move as political censorship and an attempt to suppress dissent.

<br class="ltx_break"><span id="A1.SS2.p1.1.3" class="ltx_text ltx_font_bold">5-shot</span>: Russia’s leading independent pollster has been labelled a "foreign agent" by the justice ministry, two weeks before parliamentary elections.

<br class="ltx_break"><span id="A1.SS2.p1.1.4" class="ltx_text ltx_font_bold">10-shot</span>: A leading Russian opinion pollster has been labelled a "foreign agent" by the authorities, effectively barring it from conducting any work linked to the upcoming parliamentary elections.

<br class="ltx_break"><span id="A1.SS2.p1.1.5" class="ltx_text ltx_font_bold">25-shot</span>: Russia’s leading independent pollster has been labelled a "foreign agent" by the justice ministry, effectively barring it from conducting election-related work.

<br class="ltx_break"><span id="A1.SS2.p1.1.6" class="ltx_text ltx_font_bold">50-shot</span>: 12 September 2016 Last updated at 14:32 GMT
One of Russia’s last independent pollsters has been designated a "foreign agent" by the justice ministry.

<br class="ltx_break"><span id="A1.SS2.p1.1.7" class="ltx_text ltx_font_bold">125-shot</span>: 12 September 2016 Last updated at 14:43 GMT
One of Russia’s last independent pollsters has been branded a "foreign agent" by the justice ministry.

<br class="ltx_break"><span id="A1.SS2.p1.1.8" class="ltx_text ltx_font_bold">250-shot</span>: 2016-09-06 Last updated at 14:33 GMT
Russia’s only independent pollster has been branded a "foreign agent" by the justice ministry.

<br class="ltx_break"><span id="A1.SS2.p1.1.9" class="ltx_text ltx_font_bold">500-shot</span>: 12 September 2016 Last updated at 14:22 GMT
One of Russia’s last independent pollsters has been branded a "foreign agent" by the government.
<br class="ltx_break"></p>
</div>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Additional Results for Reinforced and Unsupervised ICL</h3>

<figure id="A1.F17" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A1.F17.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_middle" style="width:212.5pt;"><img src="/html/2404.11018/assets/x20.png" id="A1.F17.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="298" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure A.16: </span><span id="A1.F17.1.2.1" class="ltx_text ltx_font_bold">Hendrycks MATH and Minvera Prompt</span>. We find the performance of model-generated rationales with 4-shot Minerva prompt is generally better or comparable to the ones generated by 4-shot InnerMono prompt. Furthermore, another iteration of Reinforced ICL – generating rationales from the best performing 25-shot prompt (with model-generated rationales) on the MATH training set and using the problems which were not solved in first iteration – seem to further improve many-shot performance.
</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A1.F17.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_middle" style="width:212.5pt;"><img src="/html/2404.11018/assets/x21.png" id="A1.F17.2.g1" class="ltx_graphics ltx_img_landscape" width="461" height="291" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure A.17: </span><span id="A1.F17.2.2.1" class="ltx_text ltx_font_bold">Unsupervised ICL does not work for low-resource machine translation.</span> This is expected as providing only source sentences for translation task doesn’t improve the task specification. See <a href="#A1.F27" title="Figure A.27 ‣ A.6 Prompts for Unsupervised ICL ‣ Appendix A Appendix ‣ Many-Shot In-Context Learning" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure A.27</span></a> for the prompt used for unsupervised ICL for this experiment.</figcaption>
</figure>
</div>
</div>
</figure>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.4 </span>Linear Classification: Data Generation</h3>

<figure id="LST1" class="ltx_float ltx_lstlisting">
<div id="LST1.1" class="ltx_listing ltx_lst_language_Python ltx_lstlisting ltx_listing">
<div class="ltx_listing_data"><a href="data:text/plain;base64,CmltcG9ydCBudW1weSBhcyBucAoKZGVmIF9nZW5lcmF0ZV9kYXRhc2V0KG1pbnYsIG1heHYsIE4sIGssIGEsIHQpOgogICAgeHRyYWluLCB5dHJhaW4gPSBbXSwgW10KICAgIGNvdW50X3BvcywgY291bnRfbmVnID0gMCwgMAoKICAgIHdoaWxlIChjb3VudF9wb3MgPCBrKSBvciAoY291bnRfbmVnIDwgayk6CiAgICAgIHhfZXggPSBucC5yYW5kb20ucmFuZGludChtaW52LCBtYXh2LCBzaXplPU4pCiAgICAgIGxhYmVsID0gMQogICAgICBpZiBucC5kb3QoeF9leCwgYSkgPiB0OgogICAgICAgIGlmIGNvdW50X3BvcyA+PSBrOgogICAgICAgICAgY29udGludWUKICAgICAgICBjb3VudF9wb3MgKz0gMQogICAgICBlbHNlOgogICAgICAgIGlmIGNvdW50X25lZyA+PSBrOgogICAgICAgICAgY29udGludWUKICAgICAgICBjb3VudF9uZWcgKz0gMQogICAgICAgIGxhYmVsID0gLTEKICAgICAgeHRyYWluLmFwcGVuZCh4X2V4KQogICAgICB5dHJhaW4uYXBwZW5kKGxhYmVsKQogICAgcmV0dXJuIG5wLmFycmF5KHh0cmFpbikuYXN0eXBlKHN0ciksIG5wLmFycmF5KHl0cmFpbikKCmRlZiBHRU5FUkFURUVWQUwoTiwgaywgc2VlZCk6CiAgICAiIiJHZW5lcmF0ZXMgb25lIGV2YWx1YXRpb24gZXhhbXBsZSBmb3IgTi1kaW1lbnNpb25hbCBsaW5lYXIgY2xhc3NpZmljYXRpb24uCgogICAgQXJnczoKICAgICAgICBOOiBEaW1lbnNpb25hbGl0eSBvZiB0aGUgZGF0YS4KICAgICAgICBrOiBOdW1iZXIgb2YgaW4tY29udGV4dCBleGVtcGxhcnMgcGVyIGNsYXNzLgoKICAgIFJldHVybnM6CiAgICAgICAgeHRyYWluOiBBIGxpc3Qgb2YgMmsgdHJhaW5pbmcgZXhhbXBsZXMgKGsgcG9zaXRpdmUsIGsgbmVnYXRpdmUpLgogICAgICAgIHl0cmFpbjogQSBsaXN0IG9mIGNvcnJlc3BvbmRpbmcgbGFiZWxzIGZvciB0cmFpbmluZyBleGFtcGxlcy4KICAgICAgICB4ZXZhbDogIEEgbGlzdCBvZiBldmFsdWF0aW9uIGV4YW1wbGVzICgyNSBwb3NpdGl2ZSwgMjUgbmVnYXRpdmUpCiAgICAgICAgeWV2YWw6ICBHcm91bmQtdHJ1dGggbGFiZWxzIGZvciBldmFsdWF0aW9uIGV4YW1wbGVzLgogICAgIiIiCgogICAgIyBTdGVwIDI6IEdlbmVyYXRlIGdyb3VuZC10cnV0aCBjb2VmZmljaWVudHMKICAgIG5wLnJhbmRvbS5zZWVkKHNlZWQpCiAgICBtaW52LCBtYXh2ID0gMSwgMTAwMAogICAgYSA9IG5wLnJhbmRvbS5yYW5kaW50KG1pbnYsIG1heHYsIHNpemU9TikgICMgUmFuZG9tIGludGVnZXIgY29lZmZpY2llbnRzCgogICAgIyBTdGVwIDM6IEdlbmVyYXRlIGEgcGl2b3QgcG9pbnQKICAgIHAgPSBucC5yYW5kb20ucmFuZGludChtaW52LCBtYXh2LCBzaXplPU4pCgogICAgIyBTdGVwIDQ6IENhbGN1bGF0ZSB0aGUgY2xhc3NpZmljYXRpb24gdGhyZXNob2xkCiAgICB0ID0gbnAuZG90KGEsIHApCgogICAgIyBTdGVwcyA1OiBHZW5lcmF0ZSB0cmFpbmluZyBleGFtcGxlcwogICAgeHRyYWluLCB5dHJhaW4gPSBfZ2VuZXJhdGVfZGF0YXNldChtaW52LCBtYXh2LCBOLCBrLCBhLCB0KQoKICAgICMgU3RlcHMgNjogR2VuZXJhdGUgdGhlIGV2YWx1YXRpb24gZXhhbXBsZQogICAgeGV2YWwsIHlldmFsID0gX2dlbmVyYXRlX2RhdGFzZXQobWludiwgbWF4diwgTiwgMjUsIGEsIHQpCgogICAgcmV0dXJuIHh0cmFpbiwgeXRyYWluLCAoeGV2YWwsIHlldmFsKQ==" download="">⬇</a></div>
<div id="lstnumberx1" class="ltx_listingline">
</div>
<div id="lstnumberx2" class="ltx_listingline">
<span id="lstnumberx2.1" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">import</span><span id="lstnumberx2.2" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx2.3" class="ltx_text ltx_lst_identifier" style="font-size:50%;">numpy</span><span id="lstnumberx2.4" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx2.5" class="ltx_text ltx_lst_identifier" style="font-size:50%;">as</span><span id="lstnumberx2.6" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx2.7" class="ltx_text ltx_lst_identifier" style="font-size:50%;">np</span>
</div>
<div id="lstnumberx3" class="ltx_listingline">
</div>
<div id="lstnumberx4" class="ltx_listingline">
<span id="lstnumberx4.1" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">def</span><span id="lstnumberx4.2" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx4.3" class="ltx_text ltx_lst_identifier" style="font-size:50%;">_generate_dataset</span><span id="lstnumberx4.4" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx4.5" class="ltx_text ltx_lst_identifier" style="font-size:50%;">minv</span><span id="lstnumberx4.6" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx4.7" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx4.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">maxv</span><span id="lstnumberx4.9" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx4.10" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx4.11" class="ltx_text ltx_lst_identifier" style="font-size:50%;">N</span><span id="lstnumberx4.12" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx4.13" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx4.14" class="ltx_text ltx_lst_identifier" style="font-size:50%;">k</span><span id="lstnumberx4.15" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx4.16" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx4.17" class="ltx_text ltx_lst_identifier" style="font-size:50%;">a</span><span id="lstnumberx4.18" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx4.19" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx4.20" class="ltx_text ltx_lst_identifier" style="font-size:50%;">t</span><span id="lstnumberx4.21" class="ltx_text" style="font-size:50%;">):</span>
</div>
<div id="lstnumberx5" class="ltx_listingline">
<span id="lstnumberx5.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx5.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">xtrain</span><span id="lstnumberx5.3" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx5.4" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx5.5" class="ltx_text ltx_lst_identifier" style="font-size:50%;">ytrain</span><span id="lstnumberx5.6" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx5.7" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx5.8" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx5.9" class="ltx_text" style="font-size:50%;">[],</span><span id="lstnumberx5.10" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx5.11" class="ltx_text" style="font-size:50%;">[]</span>
</div>
<div id="lstnumberx6" class="ltx_listingline">
<span id="lstnumberx6.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx6.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">count_pos</span><span id="lstnumberx6.3" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx6.4" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx6.5" class="ltx_text ltx_lst_identifier" style="font-size:50%;">count_neg</span><span id="lstnumberx6.6" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx6.7" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx6.8" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx6.9" class="ltx_text" style="font-size:50%;">0,</span><span id="lstnumberx6.10" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx6.11" class="ltx_text" style="font-size:50%;">0</span>
</div>
<div id="lstnumberx7" class="ltx_listingline">
</div>
<div id="lstnumberx8" class="ltx_listingline">
<span id="lstnumberx8.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx8.2" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">while</span><span id="lstnumberx8.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx8.4" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx8.5" class="ltx_text ltx_lst_identifier" style="font-size:50%;">count_pos</span><span id="lstnumberx8.6" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx8.7" class="ltx_text" style="font-size:50%;">&lt;</span><span id="lstnumberx8.8" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx8.9" class="ltx_text ltx_lst_identifier" style="font-size:50%;">k</span><span id="lstnumberx8.10" class="ltx_text" style="font-size:50%;">)</span><span id="lstnumberx8.11" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx8.12" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">or</span><span id="lstnumberx8.13" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx8.14" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx8.15" class="ltx_text ltx_lst_identifier" style="font-size:50%;">count_neg</span><span id="lstnumberx8.16" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx8.17" class="ltx_text" style="font-size:50%;">&lt;</span><span id="lstnumberx8.18" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx8.19" class="ltx_text ltx_lst_identifier" style="font-size:50%;">k</span><span id="lstnumberx8.20" class="ltx_text" style="font-size:50%;">):</span>
</div>
<div id="lstnumberx9" class="ltx_listingline">
<span id="lstnumberx9.1" class="ltx_text ltx_lst_space" style="font-size:50%;">      </span><span id="lstnumberx9.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">x_ex</span><span id="lstnumberx9.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx9.4" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx9.5" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx9.6" class="ltx_text ltx_lst_identifier" style="font-size:50%;">np</span><span id="lstnumberx9.7" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx9.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">random</span><span id="lstnumberx9.9" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx9.10" class="ltx_text ltx_lst_identifier" style="font-size:50%;">randint</span><span id="lstnumberx9.11" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx9.12" class="ltx_text ltx_lst_identifier" style="font-size:50%;">minv</span><span id="lstnumberx9.13" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx9.14" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx9.15" class="ltx_text ltx_lst_identifier" style="font-size:50%;">maxv</span><span id="lstnumberx9.16" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx9.17" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx9.18" class="ltx_text ltx_lst_identifier" style="font-size:50%;">size</span><span id="lstnumberx9.19" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx9.20" class="ltx_text ltx_lst_identifier" style="font-size:50%;">N</span><span id="lstnumberx9.21" class="ltx_text" style="font-size:50%;">)</span>
</div>
<div id="lstnumberx10" class="ltx_listingline">
<span id="lstnumberx10.1" class="ltx_text ltx_lst_space" style="font-size:50%;">      </span><span id="lstnumberx10.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">label</span><span id="lstnumberx10.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx10.4" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx10.5" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx10.6" class="ltx_text" style="font-size:50%;">1</span>
</div>
<div id="lstnumberx11" class="ltx_listingline">
<span id="lstnumberx11.1" class="ltx_text ltx_lst_space" style="font-size:50%;">      </span><span id="lstnumberx11.2" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">if</span><span id="lstnumberx11.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx11.4" class="ltx_text ltx_lst_identifier" style="font-size:50%;">np</span><span id="lstnumberx11.5" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx11.6" class="ltx_text ltx_lst_identifier" style="font-size:50%;">dot</span><span id="lstnumberx11.7" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx11.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">x_ex</span><span id="lstnumberx11.9" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx11.10" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx11.11" class="ltx_text ltx_lst_identifier" style="font-size:50%;">a</span><span id="lstnumberx11.12" class="ltx_text" style="font-size:50%;">)</span><span id="lstnumberx11.13" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx11.14" class="ltx_text" style="font-size:50%;">&gt;</span><span id="lstnumberx11.15" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx11.16" class="ltx_text ltx_lst_identifier" style="font-size:50%;">t</span><span id="lstnumberx11.17" class="ltx_text" style="font-size:50%;">:</span>
</div>
<div id="lstnumberx12" class="ltx_listingline">
<span id="lstnumberx12.1" class="ltx_text ltx_lst_space" style="font-size:50%;">        </span><span id="lstnumberx12.2" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">if</span><span id="lstnumberx12.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx12.4" class="ltx_text ltx_lst_identifier" style="font-size:50%;">count_pos</span><span id="lstnumberx12.5" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx12.6" class="ltx_text" style="font-size:50%;">&gt;=</span><span id="lstnumberx12.7" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx12.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">k</span><span id="lstnumberx12.9" class="ltx_text" style="font-size:50%;">:</span>
</div>
<div id="lstnumberx13" class="ltx_listingline">
<span id="lstnumberx13.1" class="ltx_text ltx_lst_space" style="font-size:50%;">          </span><span id="lstnumberx13.2" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">continue</span>
</div>
<div id="lstnumberx14" class="ltx_listingline">
<span id="lstnumberx14.1" class="ltx_text ltx_lst_space" style="font-size:50%;">        </span><span id="lstnumberx14.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">count_pos</span><span id="lstnumberx14.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx14.4" class="ltx_text" style="font-size:50%;">+=</span><span id="lstnumberx14.5" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx14.6" class="ltx_text" style="font-size:50%;">1</span>
</div>
<div id="lstnumberx15" class="ltx_listingline">
<span id="lstnumberx15.1" class="ltx_text ltx_lst_space" style="font-size:50%;">      </span><span id="lstnumberx15.2" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">else</span><span id="lstnumberx15.3" class="ltx_text" style="font-size:50%;">:</span>
</div>
<div id="lstnumberx16" class="ltx_listingline">
<span id="lstnumberx16.1" class="ltx_text ltx_lst_space" style="font-size:50%;">        </span><span id="lstnumberx16.2" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">if</span><span id="lstnumberx16.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx16.4" class="ltx_text ltx_lst_identifier" style="font-size:50%;">count_neg</span><span id="lstnumberx16.5" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx16.6" class="ltx_text" style="font-size:50%;">&gt;=</span><span id="lstnumberx16.7" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx16.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">k</span><span id="lstnumberx16.9" class="ltx_text" style="font-size:50%;">:</span>
</div>
<div id="lstnumberx17" class="ltx_listingline">
<span id="lstnumberx17.1" class="ltx_text ltx_lst_space" style="font-size:50%;">          </span><span id="lstnumberx17.2" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">continue</span>
</div>
<div id="lstnumberx18" class="ltx_listingline">
<span id="lstnumberx18.1" class="ltx_text ltx_lst_space" style="font-size:50%;">        </span><span id="lstnumberx18.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">count_neg</span><span id="lstnumberx18.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx18.4" class="ltx_text" style="font-size:50%;">+=</span><span id="lstnumberx18.5" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx18.6" class="ltx_text" style="font-size:50%;">1</span>
</div>
<div id="lstnumberx19" class="ltx_listingline">
<span id="lstnumberx19.1" class="ltx_text ltx_lst_space" style="font-size:50%;">        </span><span id="lstnumberx19.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">label</span><span id="lstnumberx19.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx19.4" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx19.5" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx19.6" class="ltx_text" style="font-size:50%;">-1</span>
</div>
<div id="lstnumberx20" class="ltx_listingline">
<span id="lstnumberx20.1" class="ltx_text ltx_lst_space" style="font-size:50%;">      </span><span id="lstnumberx20.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">xtrain</span><span id="lstnumberx20.3" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx20.4" class="ltx_text ltx_lst_identifier" style="font-size:50%;">append</span><span id="lstnumberx20.5" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx20.6" class="ltx_text ltx_lst_identifier" style="font-size:50%;">x_ex</span><span id="lstnumberx20.7" class="ltx_text" style="font-size:50%;">)</span>
</div>
<div id="lstnumberx21" class="ltx_listingline">
<span id="lstnumberx21.1" class="ltx_text ltx_lst_space" style="font-size:50%;">      </span><span id="lstnumberx21.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">ytrain</span><span id="lstnumberx21.3" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx21.4" class="ltx_text ltx_lst_identifier" style="font-size:50%;">append</span><span id="lstnumberx21.5" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx21.6" class="ltx_text ltx_lst_identifier" style="font-size:50%;">label</span><span id="lstnumberx21.7" class="ltx_text" style="font-size:50%;">)</span>
</div>
<div id="lstnumberx22" class="ltx_listingline">
<span id="lstnumberx22.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx22.2" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">return</span><span id="lstnumberx22.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx22.4" class="ltx_text ltx_lst_identifier" style="font-size:50%;">np</span><span id="lstnumberx22.5" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx22.6" class="ltx_text ltx_lst_identifier" style="font-size:50%;">array</span><span id="lstnumberx22.7" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx22.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">xtrain</span><span id="lstnumberx22.9" class="ltx_text" style="font-size:50%;">).</span><span id="lstnumberx22.10" class="ltx_text ltx_lst_identifier" style="font-size:50%;">astype</span><span id="lstnumberx22.11" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx22.12" class="ltx_text ltx_lst_keyword ltx_lst_keywords2 ltx_font_bold" style="font-size:50%;">str</span><span id="lstnumberx22.13" class="ltx_text" style="font-size:50%;">),</span><span id="lstnumberx22.14" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx22.15" class="ltx_text ltx_lst_identifier" style="font-size:50%;">np</span><span id="lstnumberx22.16" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx22.17" class="ltx_text ltx_lst_identifier" style="font-size:50%;">array</span><span id="lstnumberx22.18" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx22.19" class="ltx_text ltx_lst_identifier" style="font-size:50%;">ytrain</span><span id="lstnumberx22.20" class="ltx_text" style="font-size:50%;">)</span>
</div>
<div id="lstnumberx23" class="ltx_listingline">
</div>
<div id="lstnumberx24" class="ltx_listingline">
<span id="lstnumberx24.1" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">def</span><span id="lstnumberx24.2" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx24.3" class="ltx_text ltx_lst_identifier" style="font-size:50%;">GENERATEEVAL</span><span id="lstnumberx24.4" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx24.5" class="ltx_text ltx_lst_identifier" style="font-size:50%;">N</span><span id="lstnumberx24.6" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx24.7" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx24.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">k</span><span id="lstnumberx24.9" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx24.10" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx24.11" class="ltx_text ltx_lst_identifier" style="font-size:50%;">seed</span><span id="lstnumberx24.12" class="ltx_text" style="font-size:50%;">):</span>
</div>
<div id="lstnumberx25" class="ltx_listingline">
<span id="lstnumberx25.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx25.2" class="ltx_text ltx_lst_string" style="font-size:50%;">""</span><span id="lstnumberx25.3" class="ltx_text ltx_lst_string" style="font-size:50%;">"Generates<span id="lstnumberx25.3.1" class="ltx_text ltx_lst_space">␣</span>one<span id="lstnumberx25.3.2" class="ltx_text ltx_lst_space">␣</span>evaluation<span id="lstnumberx25.3.3" class="ltx_text ltx_lst_space">␣</span>example<span id="lstnumberx25.3.4" class="ltx_text ltx_lst_space">␣</span>for<span id="lstnumberx25.3.5" class="ltx_text ltx_lst_space">␣</span>N-dimensional<span id="lstnumberx25.3.6" class="ltx_text ltx_lst_space">␣</span>linear<span id="lstnumberx25.3.7" class="ltx_text ltx_lst_space">␣</span>classification.</span>
</div>
<div id="lstnumberx26" class="ltx_listingline">
</div>
<div id="lstnumberx27" class="ltx_listingline">
<span id="lstnumberx27.1" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣␣␣</span><span id="lstnumberx27.2" class="ltx_text" style="font-size:50%;">Args:</span>
</div>
<div id="lstnumberx28" class="ltx_listingline">
<span id="lstnumberx28.1" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣␣␣␣␣␣␣</span><span id="lstnumberx28.2" class="ltx_text" style="font-size:50%;">N:</span><span id="lstnumberx28.3" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx28.4" class="ltx_text" style="font-size:50%;">Dimensionality</span><span id="lstnumberx28.5" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx28.6" class="ltx_text" style="font-size:50%;">of</span><span id="lstnumberx28.7" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx28.8" class="ltx_text" style="font-size:50%;">the</span><span id="lstnumberx28.9" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx28.10" class="ltx_text" style="font-size:50%;">data.</span>
</div>
<div id="lstnumberx29" class="ltx_listingline">
<span id="lstnumberx29.1" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣␣␣␣␣␣␣</span><span id="lstnumberx29.2" class="ltx_text" style="font-size:50%;">k:</span><span id="lstnumberx29.3" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx29.4" class="ltx_text" style="font-size:50%;">Number</span><span id="lstnumberx29.5" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx29.6" class="ltx_text" style="font-size:50%;">of</span><span id="lstnumberx29.7" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx29.8" class="ltx_text" style="font-size:50%;">in-context</span><span id="lstnumberx29.9" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx29.10" class="ltx_text" style="font-size:50%;">exemplars</span><span id="lstnumberx29.11" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx29.12" class="ltx_text" style="font-size:50%;">per</span><span id="lstnumberx29.13" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx29.14" class="ltx_text" style="font-size:50%;">class.</span>
</div>
<div id="lstnumberx30" class="ltx_listingline">
</div>
<div id="lstnumberx31" class="ltx_listingline">
<span id="lstnumberx31.1" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣␣␣</span><span id="lstnumberx31.2" class="ltx_text" style="font-size:50%;">Returns:</span>
</div>
<div id="lstnumberx32" class="ltx_listingline">
<span id="lstnumberx32.1" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣␣␣␣␣␣␣</span><span id="lstnumberx32.2" class="ltx_text" style="font-size:50%;">xtrain:</span><span id="lstnumberx32.3" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx32.4" class="ltx_text" style="font-size:50%;">A</span><span id="lstnumberx32.5" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx32.6" class="ltx_text" style="font-size:50%;">list</span><span id="lstnumberx32.7" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx32.8" class="ltx_text" style="font-size:50%;">of</span><span id="lstnumberx32.9" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx32.10" class="ltx_text" style="font-size:50%;">2k</span><span id="lstnumberx32.11" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx32.12" class="ltx_text" style="font-size:50%;">training</span><span id="lstnumberx32.13" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx32.14" class="ltx_text" style="font-size:50%;">examples</span><span id="lstnumberx32.15" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx32.16" class="ltx_text" style="font-size:50%;">(k</span><span id="lstnumberx32.17" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx32.18" class="ltx_text" style="font-size:50%;">positive,</span><span id="lstnumberx32.19" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx32.20" class="ltx_text" style="font-size:50%;">k</span><span id="lstnumberx32.21" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx32.22" class="ltx_text" style="font-size:50%;">negative).</span>
</div>
<div id="lstnumberx33" class="ltx_listingline">
<span id="lstnumberx33.1" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣␣␣␣␣␣␣</span><span id="lstnumberx33.2" class="ltx_text" style="font-size:50%;">ytrain:</span><span id="lstnumberx33.3" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx33.4" class="ltx_text" style="font-size:50%;">A</span><span id="lstnumberx33.5" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx33.6" class="ltx_text" style="font-size:50%;">list</span><span id="lstnumberx33.7" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx33.8" class="ltx_text" style="font-size:50%;">of</span><span id="lstnumberx33.9" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx33.10" class="ltx_text" style="font-size:50%;">corresponding</span><span id="lstnumberx33.11" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx33.12" class="ltx_text" style="font-size:50%;">labels</span><span id="lstnumberx33.13" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx33.14" class="ltx_text" style="font-size:50%;">for</span><span id="lstnumberx33.15" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx33.16" class="ltx_text" style="font-size:50%;">training</span><span id="lstnumberx33.17" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx33.18" class="ltx_text" style="font-size:50%;">examples.</span>
</div>
<div id="lstnumberx34" class="ltx_listingline">
<span id="lstnumberx34.1" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣␣␣␣␣␣␣</span><span id="lstnumberx34.2" class="ltx_text" style="font-size:50%;">xeval:</span><span id="lstnumberx34.3" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣</span><span id="lstnumberx34.4" class="ltx_text" style="font-size:50%;">A</span><span id="lstnumberx34.5" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx34.6" class="ltx_text" style="font-size:50%;">list</span><span id="lstnumberx34.7" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx34.8" class="ltx_text" style="font-size:50%;">of</span><span id="lstnumberx34.9" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx34.10" class="ltx_text" style="font-size:50%;">evaluation</span><span id="lstnumberx34.11" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx34.12" class="ltx_text" style="font-size:50%;">examples</span><span id="lstnumberx34.13" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx34.14" class="ltx_text" style="font-size:50%;">(25</span><span id="lstnumberx34.15" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx34.16" class="ltx_text" style="font-size:50%;">positive,</span><span id="lstnumberx34.17" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx34.18" class="ltx_text" style="font-size:50%;">25</span><span id="lstnumberx34.19" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx34.20" class="ltx_text" style="font-size:50%;">negative)</span>
</div>
<div id="lstnumberx35" class="ltx_listingline">
<span id="lstnumberx35.1" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣␣␣␣␣␣␣</span><span id="lstnumberx35.2" class="ltx_text" style="font-size:50%;">yeval:</span><span id="lstnumberx35.3" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣</span><span id="lstnumberx35.4" class="ltx_text" style="font-size:50%;">Ground-truth</span><span id="lstnumberx35.5" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx35.6" class="ltx_text" style="font-size:50%;">labels</span><span id="lstnumberx35.7" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx35.8" class="ltx_text" style="font-size:50%;">for</span><span id="lstnumberx35.9" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx35.10" class="ltx_text" style="font-size:50%;">evaluation</span><span id="lstnumberx35.11" class="ltx_text ltx_lst_space" style="font-size:50%;">␣</span><span id="lstnumberx35.12" class="ltx_text" style="font-size:50%;">examples.</span>
</div>
<div id="lstnumberx36" class="ltx_listingline">
<span id="lstnumberx36.1" class="ltx_text ltx_lst_space" style="font-size:50%;">␣␣␣␣</span><span id="lstnumberx36.2" class="ltx_text" style="font-size:50%;">"</span><span id="lstnumberx36.3" class="ltx_text ltx_lst_string" style="font-size:50%;">""</span>
</div>
<div id="lstnumberx37" class="ltx_listingline">
</div>
<div id="lstnumberx38" class="ltx_listingline">
<span id="lstnumberx38.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx38.2" class="ltx_text ltx_lst_comment ltx_font_italic" style="font-size:50%;">#<span id="lstnumberx38.2.1" class="ltx_text ltx_lst_space"> </span>Step<span id="lstnumberx38.2.2" class="ltx_text ltx_lst_space"> </span>2:<span id="lstnumberx38.2.3" class="ltx_text ltx_lst_space"> </span>Generate<span id="lstnumberx38.2.4" class="ltx_text ltx_lst_space"> </span>ground-truth<span id="lstnumberx38.2.5" class="ltx_text ltx_lst_space"> </span>coefficients</span>
</div>
<div id="lstnumberx39" class="ltx_listingline">
<span id="lstnumberx39.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx39.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">np</span><span id="lstnumberx39.3" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx39.4" class="ltx_text ltx_lst_identifier" style="font-size:50%;">random</span><span id="lstnumberx39.5" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx39.6" class="ltx_text ltx_lst_identifier" style="font-size:50%;">seed</span><span id="lstnumberx39.7" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx39.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">seed</span><span id="lstnumberx39.9" class="ltx_text" style="font-size:50%;">)</span>
</div>
<div id="lstnumberx40" class="ltx_listingline">
<span id="lstnumberx40.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx40.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">minv</span><span id="lstnumberx40.3" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx40.4" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx40.5" class="ltx_text ltx_lst_identifier" style="font-size:50%;">maxv</span><span id="lstnumberx40.6" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx40.7" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx40.8" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx40.9" class="ltx_text" style="font-size:50%;">1,</span><span id="lstnumberx40.10" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx40.11" class="ltx_text" style="font-size:50%;">1000</span>
</div>
<div id="lstnumberx41" class="ltx_listingline">
<span id="lstnumberx41.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx41.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">a</span><span id="lstnumberx41.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx41.4" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx41.5" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx41.6" class="ltx_text ltx_lst_identifier" style="font-size:50%;">np</span><span id="lstnumberx41.7" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx41.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">random</span><span id="lstnumberx41.9" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx41.10" class="ltx_text ltx_lst_identifier" style="font-size:50%;">randint</span><span id="lstnumberx41.11" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx41.12" class="ltx_text ltx_lst_identifier" style="font-size:50%;">minv</span><span id="lstnumberx41.13" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx41.14" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx41.15" class="ltx_text ltx_lst_identifier" style="font-size:50%;">maxv</span><span id="lstnumberx41.16" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx41.17" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx41.18" class="ltx_text ltx_lst_identifier" style="font-size:50%;">size</span><span id="lstnumberx41.19" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx41.20" class="ltx_text ltx_lst_identifier" style="font-size:50%;">N</span><span id="lstnumberx41.21" class="ltx_text" style="font-size:50%;">)</span><span id="lstnumberx41.22" class="ltx_text ltx_lst_space" style="font-size:50%;">  </span><span id="lstnumberx41.23" class="ltx_text ltx_lst_comment ltx_font_italic" style="font-size:50%;">#<span id="lstnumberx41.23.1" class="ltx_text ltx_lst_space"> </span>Random<span id="lstnumberx41.23.2" class="ltx_text ltx_lst_space"> </span>integer<span id="lstnumberx41.23.3" class="ltx_text ltx_lst_space"> </span>coefficients</span>
</div>
<div id="lstnumberx42" class="ltx_listingline">
</div>
<div id="lstnumberx43" class="ltx_listingline">
<span id="lstnumberx43.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx43.2" class="ltx_text ltx_lst_comment ltx_font_italic" style="font-size:50%;">#<span id="lstnumberx43.2.1" class="ltx_text ltx_lst_space"> </span>Step<span id="lstnumberx43.2.2" class="ltx_text ltx_lst_space"> </span>3:<span id="lstnumberx43.2.3" class="ltx_text ltx_lst_space"> </span>Generate<span id="lstnumberx43.2.4" class="ltx_text ltx_lst_space"> </span>a<span id="lstnumberx43.2.5" class="ltx_text ltx_lst_space"> </span>pivot<span id="lstnumberx43.2.6" class="ltx_text ltx_lst_space"> </span>point</span>
</div>
<div id="lstnumberx44" class="ltx_listingline">
<span id="lstnumberx44.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx44.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">p</span><span id="lstnumberx44.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx44.4" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx44.5" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx44.6" class="ltx_text ltx_lst_identifier" style="font-size:50%;">np</span><span id="lstnumberx44.7" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx44.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">random</span><span id="lstnumberx44.9" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx44.10" class="ltx_text ltx_lst_identifier" style="font-size:50%;">randint</span><span id="lstnumberx44.11" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx44.12" class="ltx_text ltx_lst_identifier" style="font-size:50%;">minv</span><span id="lstnumberx44.13" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx44.14" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx44.15" class="ltx_text ltx_lst_identifier" style="font-size:50%;">maxv</span><span id="lstnumberx44.16" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx44.17" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx44.18" class="ltx_text ltx_lst_identifier" style="font-size:50%;">size</span><span id="lstnumberx44.19" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx44.20" class="ltx_text ltx_lst_identifier" style="font-size:50%;">N</span><span id="lstnumberx44.21" class="ltx_text" style="font-size:50%;">)</span>
</div>
<div id="lstnumberx45" class="ltx_listingline">
</div>
<div id="lstnumberx46" class="ltx_listingline">
<span id="lstnumberx46.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx46.2" class="ltx_text ltx_lst_comment ltx_font_italic" style="font-size:50%;">#<span id="lstnumberx46.2.1" class="ltx_text ltx_lst_space"> </span>Step<span id="lstnumberx46.2.2" class="ltx_text ltx_lst_space"> </span>4:<span id="lstnumberx46.2.3" class="ltx_text ltx_lst_space"> </span>Calculate<span id="lstnumberx46.2.4" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx46.2.5" class="ltx_text ltx_lst_space"> </span>classification<span id="lstnumberx46.2.6" class="ltx_text ltx_lst_space"> </span>threshold</span>
</div>
<div id="lstnumberx47" class="ltx_listingline">
<span id="lstnumberx47.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx47.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">t</span><span id="lstnumberx47.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx47.4" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx47.5" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx47.6" class="ltx_text ltx_lst_identifier" style="font-size:50%;">np</span><span id="lstnumberx47.7" class="ltx_text" style="font-size:50%;">.</span><span id="lstnumberx47.8" class="ltx_text ltx_lst_identifier" style="font-size:50%;">dot</span><span id="lstnumberx47.9" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx47.10" class="ltx_text ltx_lst_identifier" style="font-size:50%;">a</span><span id="lstnumberx47.11" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx47.12" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx47.13" class="ltx_text ltx_lst_identifier" style="font-size:50%;">p</span><span id="lstnumberx47.14" class="ltx_text" style="font-size:50%;">)</span>
</div>
<div id="lstnumberx48" class="ltx_listingline">
</div>
<div id="lstnumberx49" class="ltx_listingline">
<span id="lstnumberx49.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx49.2" class="ltx_text ltx_lst_comment ltx_font_italic" style="font-size:50%;">#<span id="lstnumberx49.2.1" class="ltx_text ltx_lst_space"> </span>Steps<span id="lstnumberx49.2.2" class="ltx_text ltx_lst_space"> </span>5:<span id="lstnumberx49.2.3" class="ltx_text ltx_lst_space"> </span>Generate<span id="lstnumberx49.2.4" class="ltx_text ltx_lst_space"> </span>training<span id="lstnumberx49.2.5" class="ltx_text ltx_lst_space"> </span>examples</span>
</div>
<div id="lstnumberx50" class="ltx_listingline">
<span id="lstnumberx50.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx50.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">xtrain</span><span id="lstnumberx50.3" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx50.4" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx50.5" class="ltx_text ltx_lst_identifier" style="font-size:50%;">ytrain</span><span id="lstnumberx50.6" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx50.7" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx50.8" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx50.9" class="ltx_text ltx_lst_identifier" style="font-size:50%;">_generate_dataset</span><span id="lstnumberx50.10" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx50.11" class="ltx_text ltx_lst_identifier" style="font-size:50%;">minv</span><span id="lstnumberx50.12" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx50.13" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx50.14" class="ltx_text ltx_lst_identifier" style="font-size:50%;">maxv</span><span id="lstnumberx50.15" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx50.16" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx50.17" class="ltx_text ltx_lst_identifier" style="font-size:50%;">N</span><span id="lstnumberx50.18" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx50.19" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx50.20" class="ltx_text ltx_lst_identifier" style="font-size:50%;">k</span><span id="lstnumberx50.21" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx50.22" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx50.23" class="ltx_text ltx_lst_identifier" style="font-size:50%;">a</span><span id="lstnumberx50.24" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx50.25" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx50.26" class="ltx_text ltx_lst_identifier" style="font-size:50%;">t</span><span id="lstnumberx50.27" class="ltx_text" style="font-size:50%;">)</span>
</div>
<div id="lstnumberx51" class="ltx_listingline">
</div>
<div id="lstnumberx52" class="ltx_listingline">
<span id="lstnumberx52.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx52.2" class="ltx_text ltx_lst_comment ltx_font_italic" style="font-size:50%;">#<span id="lstnumberx52.2.1" class="ltx_text ltx_lst_space"> </span>Steps<span id="lstnumberx52.2.2" class="ltx_text ltx_lst_space"> </span>6:<span id="lstnumberx52.2.3" class="ltx_text ltx_lst_space"> </span>Generate<span id="lstnumberx52.2.4" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx52.2.5" class="ltx_text ltx_lst_space"> </span>evaluation<span id="lstnumberx52.2.6" class="ltx_text ltx_lst_space"> </span>example</span>
</div>
<div id="lstnumberx53" class="ltx_listingline">
<span id="lstnumberx53.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx53.2" class="ltx_text ltx_lst_identifier" style="font-size:50%;">xeval</span><span id="lstnumberx53.3" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx53.4" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx53.5" class="ltx_text ltx_lst_identifier" style="font-size:50%;">yeval</span><span id="lstnumberx53.6" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx53.7" class="ltx_text" style="font-size:50%;">=</span><span id="lstnumberx53.8" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx53.9" class="ltx_text ltx_lst_identifier" style="font-size:50%;">_generate_dataset</span><span id="lstnumberx53.10" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx53.11" class="ltx_text ltx_lst_identifier" style="font-size:50%;">minv</span><span id="lstnumberx53.12" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx53.13" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx53.14" class="ltx_text ltx_lst_identifier" style="font-size:50%;">maxv</span><span id="lstnumberx53.15" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx53.16" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx53.17" class="ltx_text ltx_lst_identifier" style="font-size:50%;">N</span><span id="lstnumberx53.18" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx53.19" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx53.20" class="ltx_text" style="font-size:50%;">25,</span><span id="lstnumberx53.21" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx53.22" class="ltx_text ltx_lst_identifier" style="font-size:50%;">a</span><span id="lstnumberx53.23" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx53.24" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx53.25" class="ltx_text ltx_lst_identifier" style="font-size:50%;">t</span><span id="lstnumberx53.26" class="ltx_text" style="font-size:50%;">)</span>
</div>
<div id="lstnumberx54" class="ltx_listingline">
</div>
<div id="lstnumberx55" class="ltx_listingline">
<span id="lstnumberx55.1" class="ltx_text ltx_lst_space" style="font-size:50%;">    </span><span id="lstnumberx55.2" class="ltx_text ltx_lst_keyword ltx_font_bold" style="font-size:50%;">return</span><span id="lstnumberx55.3" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx55.4" class="ltx_text ltx_lst_identifier" style="font-size:50%;">xtrain</span><span id="lstnumberx55.5" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx55.6" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx55.7" class="ltx_text ltx_lst_identifier" style="font-size:50%;">ytrain</span><span id="lstnumberx55.8" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx55.9" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx55.10" class="ltx_text" style="font-size:50%;">(</span><span id="lstnumberx55.11" class="ltx_text ltx_lst_identifier" style="font-size:50%;">xeval</span><span id="lstnumberx55.12" class="ltx_text" style="font-size:50%;">,</span><span id="lstnumberx55.13" class="ltx_text ltx_lst_space" style="font-size:50%;"> </span><span id="lstnumberx55.14" class="ltx_text ltx_lst_identifier" style="font-size:50%;">yeval</span><span id="lstnumberx55.15" class="ltx_text" style="font-size:50%;">)</span>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float">Listing 1: </span>Code for Generating Sythetic datasets for Linear Classification in High Dimensions.</figcaption>
</figure>
</section>
<section id="A1.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.5 </span>Example Prompts</h3>

<figure id="A1.F18" class="ltx_figure">
<p id="A1.F18.2" class="ltx_p ltx_align_center">

<span id="A1.F18.2.2.2" class="ltx_tabular ltx_align_middle ltx_framed ltx_framed_rectangle">
<span class="ltx_tbody">
<span id="A1.F18.2.2.2.3.1" class="ltx_tr">
<span id="A1.F18.2.2.2.3.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.3.1.1.1" class="ltx_text" style="font-size:80%;">You are an expert translator. I am going to give you one or more example pairs of text snippets where the</span></span></span>
<span id="A1.F18.2.2.2.4.2" class="ltx_tr">
<span id="A1.F18.2.2.2.4.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.4.2.1.1" class="ltx_text" style="font-size:80%;">first is in English and the second is a translation of the first snippet into Kurdish. The sentences will be</span></span></span>
<span id="A1.F18.2.2.2.5.3" class="ltx_tr">
<span id="A1.F18.2.2.2.5.3.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.5.3.1.1" class="ltx_text" style="font-size:80%;">written</span></span></span>
<span id="A1.F18.2.2.2.6.4" class="ltx_tr">
<span id="A1.F18.2.2.2.6.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.6.4.1.1" class="ltx_text" style="font-size:80%;">English: &lt;first sentence&gt;</span></span></span>
<span id="A1.F18.2.2.2.7.5" class="ltx_tr">
<span id="A1.F18.2.2.2.7.5.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.7.5.1.1" class="ltx_text" style="font-size:80%;">Kurdish: &lt;translated first sentence&gt;</span></span></span>
<span id="A1.F18.2.2.2.8.6" class="ltx_tr">
<span id="A1.F18.2.2.2.8.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.8.6.1.1" class="ltx_text" style="font-size:80%;">After the example pairs, I am going to provide another sentence in English and I want you to translate it</span></span></span>
<span id="A1.F18.2.2.2.9.7" class="ltx_tr">
<span id="A1.F18.2.2.2.9.7.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.9.7.1.1" class="ltx_text" style="font-size:80%;">into Kurdish. Give only the translation, and no extra commentary, formatting, or chattiness. Translate the</span></span></span>
<span id="A1.F18.2.2.2.10.8" class="ltx_tr">
<span id="A1.F18.2.2.2.10.8.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.10.8.1.1" class="ltx_text" style="font-size:80%;">text from English to Kurdish.</span></span></span>
<span id="A1.F18.2.2.2.11.9" class="ltx_tr">
<span id="A1.F18.2.2.2.11.9.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.11.9.1.1" class="ltx_text" style="font-size:80%;">English: Its remnants produced showers across most of the islands, though as of yet, no damage or flooding</span></span></span>
<span id="A1.F18.2.2.2.12.10" class="ltx_tr">
<span id="A1.F18.2.2.2.12.10.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.12.10.1.1" class="ltx_text" style="font-size:80%;">has been reported.</span></span></span>
<span id="A1.F18.2.2.2.13.11" class="ltx_tr">
<span id="A1.F18.2.2.2.13.11.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.13.11.1.1" class="ltx_text" style="font-size:80%;">Kurdish: Li herêma Serengetîyê, Parka Neteweyî ya Serengetî ya Tanzanyayê, Cihê Parastina Ngorongoro û</span></span></span>
<span id="A1.F18.2.2.2.14.12" class="ltx_tr">
<span id="A1.F18.2.2.2.14.12.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.14.12.1.1" class="ltx_text" style="font-size:80%;">Cihê Parastina Gîyanewerên Nêçîrê Maswa û Cihê Parastina Neteweyî ya Masaî Mara ya Kendyayê hene.</span></span></span>
<span id="A1.F18.1.1.1.1" class="ltx_tr">
<span id="A1.F18.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><math id="A1.F18.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="A1.F18.1.1.1.1.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="A1.F18.1.1.1.1.1.m1.1.1" xref="A1.F18.1.1.1.1.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="A1.F18.1.1.1.1.1.m1.1b"><ci id="A1.F18.1.1.1.1.1.m1.1.1.cmml" xref="A1.F18.1.1.1.1.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F18.1.1.1.1.1.m1.1c">\cdots</annotation></semantics></math></span></span>
<span id="A1.F18.2.2.2.2" class="ltx_tr">
<span id="A1.F18.2.2.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.2.1.1" class="ltx_text" style="font-size:80%;">English: </span><math id="A1.F18.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="A1.F18.2.2.2.2.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="A1.F18.2.2.2.2.1.m1.1.1" xref="A1.F18.2.2.2.2.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="A1.F18.2.2.2.2.1.m1.1b"><ci id="A1.F18.2.2.2.2.1.m1.1.1.cmml" xref="A1.F18.2.2.2.2.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F18.2.2.2.2.1.m1.1c">\cdots</annotation></semantics></math></span></span>
<span id="A1.F18.2.2.2.15.13" class="ltx_tr">
<span id="A1.F18.2.2.2.15.13.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F18.2.2.2.15.13.1.1" class="ltx_text" style="font-size:80%;">Kurdish:</span></span></span>
</span>
</span>
<span id="A1.F18.2.3" class="ltx_text" style="font-size:80%;"></span></p>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_figure">Figure A.18: </span>Example prompt with a test input for translation from English to Kurdish on FLORES-MT benchmark in §<a href="#S3.SS1" title="3.1 Machine Translation (MT) ‣ 3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>.</figcaption>
</figure>
<figure id="A1.F19" class="ltx_figure">
<p id="A1.F19.1" class="ltx_p ltx_align_center">

<span id="A1.F19.1.1" class="ltx_tabular ltx_align_middle ltx_framed ltx_framed_rectangle">
<span class="ltx_tbody">
<span id="A1.F19.1.1.1.1" class="ltx_tr">
<span id="A1.F19.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.1.1.1.1" class="ltx_text" style="font-size:80%;">I will first show a news article and then provide a very short one sentence long summary of it in fluent English.</span></span></span>
<span id="A1.F19.1.1.2.2" class="ltx_tr">
<span id="A1.F19.1.1.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.2.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Summarize the following article</span><span id="A1.F19.1.1.2.2.1.2" class="ltx_text" style="font-size:80%;">: Burberry reported pre-tax profits of £166m for the year to March.</span></span></span>
<span id="A1.F19.1.1.3.3" class="ltx_tr">
<span id="A1.F19.1.1.3.3.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.3.3.1.1" class="ltx_text" style="font-size:80%;">A year ago it made a loss of £16.1m, hit by charges at its Spanish operations.</span></span></span>
<span id="A1.F19.1.1.4.4" class="ltx_tr">
<span id="A1.F19.1.1.4.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.4.4.1.1" class="ltx_text" style="font-size:80%;">In the past year it has opened 21 new stores and closed nine. It plans to open 20-30 stores this year worldwide.</span></span></span>
<span id="A1.F19.1.1.5.5" class="ltx_tr">
<span id="A1.F19.1.1.5.5.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.5.5.1.1" class="ltx_text" style="font-size:80%;">The group has also focused on promoting the Burberry brand online.</span></span></span>
<span id="A1.F19.1.1.6.6" class="ltx_tr">
<span id="A1.F19.1.1.6.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.6.6.1.1" class="ltx_text" style="font-size:80%;">Sales rose 7% to £1.28bn, with the company recording double-digit sales growth in Europe and Asia Pacific.</span></span></span>
<span id="A1.F19.1.1.7.7" class="ltx_tr">
<span id="A1.F19.1.1.7.7.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.7.7.1.1" class="ltx_text" style="font-size:80%;">Adjusted profit rose 23% to £215m, taking into account one-off items and a favourable exchange rate.</span></span></span>
<span id="A1.F19.1.1.8.8" class="ltx_tr">
<span id="A1.F19.1.1.8.8.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.8.8.1.1" class="ltx_text" style="font-size:80%;">Stores in London in particular benefited from favourable currency movements and increased tourism.</span></span></span>
<span id="A1.F19.1.1.9.9" class="ltx_tr">
<span id="A1.F19.1.1.9.9.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.9.9.1.1" class="ltx_text" style="font-size:80%;">“Looking forward, while mindful of the economic environment, Burberry plans to build on its strong financial position</span></span></span>
<span id="A1.F19.1.1.10.10" class="ltx_tr">
<span id="A1.F19.1.1.10.10.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.10.10.1.1" class="ltx_text" style="font-size:80%;">by accelerating investment in growth initiatives in retail, digital and new markets, while continuing to enhance the</span></span></span>
<span id="A1.F19.1.1.11.11" class="ltx_tr">
<span id="A1.F19.1.1.11.11.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.11.11.1.1" class="ltx_text" style="font-size:80%;">brand,” said chief executive Angela Ahrendts.</span></span></span>
<span id="A1.F19.1.1.12.12" class="ltx_tr">
<span id="A1.F19.1.1.12.12.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.12.12.1.1" class="ltx_text" style="font-size:80%;">Burberry shares were up 7.6% at 659 pence in afternoon trading.</span></span></span>
<span id="A1.F19.1.1.13.13" class="ltx_tr">
<span id="A1.F19.1.1.13.13.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.13.13.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Summary</span><span id="A1.F19.1.1.13.13.1.2" class="ltx_text" style="font-size:80%;">: Luxury fashion designer Burberry has returned to profit after opening new stores and spending more</span></span></span>
<span id="A1.F19.1.1.14.14" class="ltx_tr">
<span id="A1.F19.1.1.14.14.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F19.1.1.14.14.1.1" class="ltx_text" style="font-size:80%;">on online marketing</span></span></span>
</span>
</span>
<span id="A1.F19.1.2" class="ltx_text" style="font-size:80%;"></span></p>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_figure">Figure A.19: </span>Example 1-shot prompt used for summarization on XSum and XLSum in §<a href="#S3.SS2" title="3.2 Summarization ‣ 3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>.</figcaption>
</figure>
<figure id="A1.F20" class="ltx_figure">
<p id="A1.F20.1" class="ltx_p ltx_align_center"><span id="A1.F20.1.1" class="ltx_text ltx_framed ltx_framed_rectangle" style="border-color: #000000;">

<span id="A1.F20.1.1.1" class="ltx_tabular ltx_align_middle">
<span class="ltx_tbody">
<span id="A1.F20.1.1.1.2.1" class="ltx_tr">
<span id="A1.F20.1.1.1.2.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.2.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Please solve the problem</span><span id="A1.F20.1.1.1.2.1.1.2" class="ltx_text" style="font-size:80%;">:</span></span></span>
<span id="A1.F20.1.1.1.3.2" class="ltx_tr">
<span id="A1.F20.1.1.1.3.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.3.2.1.1" class="ltx_text" style="font-size:80%;">(define (problem logistics-c2-s1-p1-a2)</span></span></span>
<span id="A1.F20.1.1.1.4.3" class="ltx_tr">
<span id="A1.F20.1.1.1.4.3.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.4.3.1.1" class="ltx_text" style="font-size:80%;">(:domain logistics-strips)</span></span></span>
<span id="A1.F20.1.1.1.5.4" class="ltx_tr">
<span id="A1.F20.1.1.1.5.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.5.4.1.1" class="ltx_text" style="font-size:80%;">(:objects</span></span></span>
<span id="A1.F20.1.1.1.6.5" class="ltx_tr">
<span id="A1.F20.1.1.1.6.5.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.6.5.1.1" class="ltx_text" style="font-size:80%;">a0 a1</span></span></span>
<span id="A1.F20.1.1.1.7.6" class="ltx_tr">
<span id="A1.F20.1.1.1.7.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.7.6.1.1" class="ltx_text" style="font-size:80%;">c0 c1</span></span></span>
<span id="A1.F20.1.1.1.8.7" class="ltx_tr">
<span id="A1.F20.1.1.1.8.7.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.8.7.1.1" class="ltx_text" style="font-size:80%;">t0 t1</span></span></span>
<span id="A1.F20.1.1.1.9.8" class="ltx_tr">
<span id="A1.F20.1.1.1.9.8.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.9.8.1.1" class="ltx_text" style="font-size:80%;">l0-0 l1-0</span></span></span>
<span id="A1.F20.1.1.1.10.9" class="ltx_tr">
<span id="A1.F20.1.1.1.10.9.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.10.9.1.1" class="ltx_text" style="font-size:80%;">p0</span></span></span>
<span id="A1.F20.1.1.1.11.10" class="ltx_tr">
<span id="A1.F20.1.1.1.11.10.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.11.10.1.1" class="ltx_text" style="font-size:80%;">)</span></span></span>
<span id="A1.F20.1.1.1.12.11" class="ltx_tr">
<span id="A1.F20.1.1.1.12.11.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.12.11.1.1" class="ltx_text" style="font-size:80%;">(:init</span></span></span>
<span id="A1.F20.1.1.1.13.12" class="ltx_tr">
<span id="A1.F20.1.1.1.13.12.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.13.12.1.1" class="ltx_text" style="font-size:80%;">(AIRPLANE a0)</span></span></span>
<span id="A1.F20.1.1.1.14.13" class="ltx_tr">
<span id="A1.F20.1.1.1.14.13.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.14.13.1.1" class="ltx_text" style="font-size:80%;">(AIRPLANE a1)</span></span></span>
<span id="A1.F20.1.1.1.15.14" class="ltx_tr">
<span id="A1.F20.1.1.1.15.14.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.15.14.1.1" class="ltx_text" style="font-size:80%;">(CITY c0)</span></span></span>
<span id="A1.F20.1.1.1.16.15" class="ltx_tr">
<span id="A1.F20.1.1.1.16.15.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.16.15.1.1" class="ltx_text" style="font-size:80%;">(CITY c1)</span></span></span>
<span id="A1.F20.1.1.1.17.16" class="ltx_tr">
<span id="A1.F20.1.1.1.17.16.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.17.16.1.1" class="ltx_text" style="font-size:80%;">(TRUCK t0)</span></span></span>
<span id="A1.F20.1.1.1.18.17" class="ltx_tr">
<span id="A1.F20.1.1.1.18.17.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.18.17.1.1" class="ltx_text" style="font-size:80%;">(TRUCK t1)</span></span></span>
<span id="A1.F20.1.1.1.19.18" class="ltx_tr">
<span id="A1.F20.1.1.1.19.18.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.19.18.1.1" class="ltx_text" style="font-size:80%;">(LOCATION l0-0)</span></span></span>
<span id="A1.F20.1.1.1.20.19" class="ltx_tr">
<span id="A1.F20.1.1.1.20.19.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.20.19.1.1" class="ltx_text" style="font-size:80%;">(in-city l0-0 c0)</span></span></span>
<span id="A1.F20.1.1.1.21.20" class="ltx_tr">
<span id="A1.F20.1.1.1.21.20.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.21.20.1.1" class="ltx_text" style="font-size:80%;">(LOCATION l1-0)</span></span></span>
<span id="A1.F20.1.1.1.22.21" class="ltx_tr">
<span id="A1.F20.1.1.1.22.21.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.22.21.1.1" class="ltx_text" style="font-size:80%;">(in-city l1-0 c1)</span></span></span>
<span id="A1.F20.1.1.1.23.22" class="ltx_tr">
<span id="A1.F20.1.1.1.23.22.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.23.22.1.1" class="ltx_text" style="font-size:80%;">(AIRPORT l0-0)</span></span></span>
<span id="A1.F20.1.1.1.24.23" class="ltx_tr">
<span id="A1.F20.1.1.1.24.23.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.24.23.1.1" class="ltx_text" style="font-size:80%;">(AIRPORT l1-0)</span></span></span>
<span id="A1.F20.1.1.1.25.24" class="ltx_tr">
<span id="A1.F20.1.1.1.25.24.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.25.24.1.1" class="ltx_text" style="font-size:80%;">(OBJ p0)</span></span></span>
<span id="A1.F20.1.1.1.26.25" class="ltx_tr">
<span id="A1.F20.1.1.1.26.25.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.26.25.1.1" class="ltx_text" style="font-size:80%;">(at t0 l0-0)</span></span></span>
<span id="A1.F20.1.1.1.27.26" class="ltx_tr">
<span id="A1.F20.1.1.1.27.26.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.27.26.1.1" class="ltx_text" style="font-size:80%;">(at t1 l1-0)</span></span></span>
<span id="A1.F20.1.1.1.28.27" class="ltx_tr">
<span id="A1.F20.1.1.1.28.27.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.28.27.1.1" class="ltx_text" style="font-size:80%;">(at p0 l1-0)</span></span></span>
<span id="A1.F20.1.1.1.29.28" class="ltx_tr">
<span id="A1.F20.1.1.1.29.28.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.29.28.1.1" class="ltx_text" style="font-size:80%;">(at a0 l0-0)</span></span></span>
<span id="A1.F20.1.1.1.30.29" class="ltx_tr">
<span id="A1.F20.1.1.1.30.29.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.30.29.1.1" class="ltx_text" style="font-size:80%;">(at a1 l1-0)</span></span></span>
<span id="A1.F20.1.1.1.31.30" class="ltx_tr">
<span id="A1.F20.1.1.1.31.30.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.31.30.1.1" class="ltx_text" style="font-size:80%;">)</span></span></span>
<span id="A1.F20.1.1.1.32.31" class="ltx_tr">
<span id="A1.F20.1.1.1.32.31.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.32.31.1.1" class="ltx_text" style="font-size:80%;">(:goal</span></span></span>
<span id="A1.F20.1.1.1.33.32" class="ltx_tr">
<span id="A1.F20.1.1.1.33.32.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.33.32.1.1" class="ltx_text" style="font-size:80%;">(and</span></span></span>
<span id="A1.F20.1.1.1.34.33" class="ltx_tr">
<span id="A1.F20.1.1.1.34.33.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.34.33.1.1" class="ltx_text" style="font-size:80%;">(at p0 l0-0)</span></span></span>
<span id="A1.F20.1.1.1.35.34" class="ltx_tr">
<span id="A1.F20.1.1.1.35.34.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.35.34.1.1" class="ltx_text" style="font-size:80%;">)</span></span></span>
<span id="A1.F20.1.1.1.36.35" class="ltx_tr">
<span id="A1.F20.1.1.1.36.35.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.36.35.1.1" class="ltx_text" style="font-size:80%;">)</span></span></span>
<span id="A1.F20.1.1.1.37.36" class="ltx_tr">
<span id="A1.F20.1.1.1.37.36.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.37.36.1.1" class="ltx_text" style="font-size:80%;">)</span></span></span>
<span id="A1.F20.1.1.1.38.37" class="ltx_tr">
<span id="A1.F20.1.1.1.38.37.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.38.37.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Your plan as plain text without formatting</span><span id="A1.F20.1.1.1.38.37.1.2" class="ltx_text" style="font-size:80%;">:</span></span></span>
<span id="A1.F20.1.1.1.39.38" class="ltx_tr">
<span id="A1.F20.1.1.1.39.38.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.39.38.1.1" class="ltx_text" style="font-size:80%;">(load-airplane p0 a1 l1-0)</span></span></span>
<span id="A1.F20.1.1.1.40.39" class="ltx_tr">
<span id="A1.F20.1.1.1.40.39.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.40.39.1.1" class="ltx_text" style="font-size:80%;">(fly-airplane a1 l1-0 l0-0)</span></span></span>
<span id="A1.F20.1.1.1.41.40" class="ltx_tr">
<span id="A1.F20.1.1.1.41.40.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.41.40.1.1" class="ltx_text" style="font-size:80%;">(unload-airplane p0 a1 l0-0)</span></span></span>
<span id="A1.F20.1.1.1.42.41" class="ltx_tr">
<span id="A1.F20.1.1.1.42.41.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.42.41.1.1" class="ltx_text" style="font-size:80%;">done.</span></span></span>
<span id="A1.F20.1.1.1.43.42" class="ltx_tr">
<span id="A1.F20.1.1.1.43.42.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.43.42.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Please solve the problem</span><span id="A1.F20.1.1.1.43.42.1.2" class="ltx_text" style="font-size:80%;">:</span></span></span>
<span id="A1.F20.1.1.1.1" class="ltx_tr">
<span id="A1.F20.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.1.1.1" class="ltx_text" style="font-size:80%;">(define (problem </span><math id="A1.F20.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="A1.F20.1.1.1.1.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="A1.F20.1.1.1.1.1.m1.1.1" xref="A1.F20.1.1.1.1.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="A1.F20.1.1.1.1.1.m1.1b"><ci id="A1.F20.1.1.1.1.1.m1.1.1.cmml" xref="A1.F20.1.1.1.1.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F20.1.1.1.1.1.m1.1c">\cdots</annotation></semantics></math><span id="A1.F20.1.1.1.1.1.2" class="ltx_text" style="font-size:80%;">)</span></span></span>
<span id="A1.F20.1.1.1.44.43" class="ltx_tr">
<span id="A1.F20.1.1.1.44.43.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F20.1.1.1.44.43.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Your plan as plain text without formatting</span><span id="A1.F20.1.1.1.44.43.1.2" class="ltx_text" style="font-size:80%;">:</span></span></span>
</span>
</span><span id="A1.F20.1.1.2" class="ltx_text" style="font-size:80%;">
</span></span></p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure A.20: </span>An example 1-shot PDDL <cite class="ltx_cite ltx_citemacro_citep">(Ghallab et al., <a href="#bib.bib9" title="" class="ltx_ref">1998</a>)</cite> prompt, with a test example for the Logistics domain in §<a href="#S3.SS3" title="3.3 Planning: Logistics Domain ‣ 3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>. Within a city, the locations are directly linked, allowing trucks to travel
between any two of these locations. Similarly, cities are directly connected to each other allowing
airplanes to travel between any two cities. Each city is equipped with one truck and has a designated
location that functions as an airport</figcaption>
</figure>
<figure id="A1.F21" class="ltx_figure">
<p id="A1.F21.1" class="ltx_p ltx_align_center">

<span id="A1.F21.1.1" class="ltx_tabular ltx_align_middle ltx_framed ltx_framed_rectangle">
<span class="ltx_tbody">
<span id="A1.F21.1.1.1.1" class="ltx_tr">
<span id="A1.F21.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F21.1.1.1.1.1.1" class="ltx_text" style="font-size:80%;">You will be given a multiple choice question with different choices such as (A), (B), (C), (D). Think step by step</span></span></span>
<span id="A1.F21.1.1.2.2" class="ltx_tr">
<span id="A1.F21.1.1.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F21.1.1.2.2.1.1" class="ltx_text" style="font-size:80%;">before giving a final answer to this question. Always finish your answer with ’Final Answer: (X)’, where X is the</span></span></span>
<span id="A1.F21.1.1.3.3" class="ltx_tr">
<span id="A1.F21.1.1.3.3.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F21.1.1.3.3.1.1" class="ltx_text" style="font-size:80%;">correct answer choice. If none of the options match, choose the closest option as the final answer.</span></span></span>
</span>
</span>
<span id="A1.F21.1.2" class="ltx_text" style="font-size:80%;"></span></p>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_figure">Figure A.21: </span>Zero-shot prompt for GPQA.</figcaption>
</figure>
<figure id="A1.F22" class="ltx_figure">
<p id="A1.F22.1" class="ltx_p ltx_align_center">

<span id="A1.F22.1.1" class="ltx_tabular ltx_align_middle ltx_framed ltx_framed_rectangle">
<span class="ltx_tbody">
<span id="A1.F22.1.1.1.1" class="ltx_tr">
<span id="A1.F22.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.1.1.1.1" class="ltx_text" style="font-size:80%;"># problem:</span></span></span>
<span id="A1.F22.1.1.2.2" class="ltx_tr">
<span id="A1.F22.1.1.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.2.2.1.1" class="ltx_text" style="font-size:80%;">It starts raining at 7:00 and pours heavily until its stops at 17:00 on a particular day.</span></span></span>
<span id="A1.F22.1.1.3.3" class="ltx_tr">
<span id="A1.F22.1.1.3.3.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.3.3.1.1" class="ltx_text" style="font-size:80%;">On the second day, the rain takes 2 more hours than it took on the first day to stop.</span></span></span>
<span id="A1.F22.1.1.4.4" class="ltx_tr">
<span id="A1.F22.1.1.4.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.4.4.1.1" class="ltx_text" style="font-size:80%;">On the third day, the rain pours for twice the amount of time it took on the second day.</span></span></span>
<span id="A1.F22.1.1.5.5" class="ltx_tr">
<span id="A1.F22.1.1.5.5.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.5.5.1.1" class="ltx_text" style="font-size:80%;">Calculate the total time it was raining in the three days.</span></span></span>
<span id="A1.F22.1.1.6.6" class="ltx_tr">
<span id="A1.F22.1.1.6.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.6.6.1.1" class="ltx_text" style="font-size:80%;"># solution:</span></span></span>
<span id="A1.F22.1.1.7.7" class="ltx_tr">
<span id="A1.F22.1.1.7.7.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.7.7.1.1" class="ltx_text" style="font-size:80%;">def solution():</span></span></span>
<span id="A1.F22.1.1.8.8" class="ltx_tr">
<span id="A1.F22.1.1.8.8.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.8.8.1.1" class="ltx_text" style="font-size:80%;">"""It starts raining at 7:00 and pours heavily until its stops at 17:00 on a particular day.</span></span></span>
<span id="A1.F22.1.1.9.9" class="ltx_tr">
<span id="A1.F22.1.1.9.9.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.9.9.1.1" class="ltx_text" style="font-size:80%;">On the second day, the rain takes 2 more hours than it took on the first day to stop.</span></span></span>
<span id="A1.F22.1.1.10.10" class="ltx_tr">
<span id="A1.F22.1.1.10.10.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.10.10.1.1" class="ltx_text" style="font-size:80%;">On the third day, the rain pours for twice the amount of time it took on the second day.</span></span></span>
<span id="A1.F22.1.1.11.11" class="ltx_tr">
<span id="A1.F22.1.1.11.11.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.11.11.1.1" class="ltx_text" style="font-size:80%;">Calculate the total time it was raining in the three days."""</span></span></span>
<span id="A1.F22.1.1.12.12" class="ltx_tr">
<span id="A1.F22.1.1.12.12.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.12.12.1.1" class="ltx_text" style="font-size:80%;">    first_day_rain_duration = 17 - 7 # 10 hours</span></span></span>
<span id="A1.F22.1.1.13.13" class="ltx_tr">
<span id="A1.F22.1.1.13.13.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.13.13.1.1" class="ltx_text" style="font-size:80%;">    second_day_rain_duration = first_day_rain_duration + 2 # 12 hours</span></span></span>
<span id="A1.F22.1.1.14.14" class="ltx_tr">
<span id="A1.F22.1.1.14.14.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.14.14.1.1" class="ltx_text" style="font-size:80%;">    third_day_rain_duration = second_day_rain_duration * 2 # 24 hours</span></span></span>
<span id="A1.F22.1.1.15.15" class="ltx_tr">
<span id="A1.F22.1.1.15.15.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.15.15.1.1" class="ltx_text" style="font-size:80%;">    total_rain_duration = first_day_rain_duration + second_day_rain_duration + third_day_rain_duration</span></span></span>
<span id="A1.F22.1.1.16.16" class="ltx_tr">
<span id="A1.F22.1.1.16.16.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.16.16.1.1" class="ltx_text" style="font-size:80%;">    result = total_rain_duration</span></span></span>
<span id="A1.F22.1.1.17.17" class="ltx_tr">
<span id="A1.F22.1.1.17.17.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.17.17.1.1" class="ltx_text" style="font-size:80%;">    return result</span></span></span>
<span id="A1.F22.1.1.18.18" class="ltx_tr">
<span id="A1.F22.1.1.18.18.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.18.18.1.1" class="ltx_text" style="font-size:80%;"># is the solution correct?</span></span></span>
<span id="A1.F22.1.1.19.19" class="ltx_tr">
<span id="A1.F22.1.1.19.19.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.19.19.1.1" class="ltx_text" style="font-size:80%;">Yes</span></span></span>
<span id="A1.F22.1.1.20.20" class="ltx_tr">
<span id="A1.F22.1.1.20.20.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.20.20.1.1" class="ltx_text" style="font-size:80%;"># problem:</span></span></span>
<span id="A1.F22.1.1.21.21" class="ltx_tr">
<span id="A1.F22.1.1.21.21.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.21.21.1.1" class="ltx_text" style="font-size:80%;">Haley is getting ready to watch a comet fly over her house.</span></span></span>
<span id="A1.F22.1.1.22.22" class="ltx_tr">
<span id="A1.F22.1.1.22.22.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.22.22.1.1" class="ltx_text" style="font-size:80%;">She spends two hours shopping for a telescope, half an hour getting everything set up in the backyard,</span></span></span>
<span id="A1.F22.1.1.23.23" class="ltx_tr">
<span id="A1.F22.1.1.23.23.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.23.23.1.1" class="ltx_text" style="font-size:80%;">three times the setup time making snacks, and 20 minutes watching the comet.</span></span></span>
<span id="A1.F22.1.1.24.24" class="ltx_tr">
<span id="A1.F22.1.1.24.24.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.24.24.1.1" class="ltx_text" style="font-size:80%;">What percentage of the total time she spent on all those activities was spent watching the comet,</span></span></span>
<span id="A1.F22.1.1.25.25" class="ltx_tr">
<span id="A1.F22.1.1.25.25.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.25.25.1.1" class="ltx_text" style="font-size:80%;">rounded to the nearest percent?</span></span></span>
<span id="A1.F22.1.1.26.26" class="ltx_tr">
<span id="A1.F22.1.1.26.26.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.26.26.1.1" class="ltx_text" style="font-size:80%;"># solution:</span></span></span>
<span id="A1.F22.1.1.27.27" class="ltx_tr">
<span id="A1.F22.1.1.27.27.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.27.27.1.1" class="ltx_text" style="font-size:80%;">def solution():</span></span></span>
<span id="A1.F22.1.1.28.28" class="ltx_tr">
<span id="A1.F22.1.1.28.28.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.28.28.1.1" class="ltx_text" style="font-size:80%;">"""Haley is getting ready to watch a comet fly over her house.</span></span></span>
<span id="A1.F22.1.1.29.29" class="ltx_tr">
<span id="A1.F22.1.1.29.29.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.29.29.1.1" class="ltx_text" style="font-size:80%;">She spends two hours shopping for a telescope, half an hour getting everything set up in the backyard,</span></span></span>
<span id="A1.F22.1.1.30.30" class="ltx_tr">
<span id="A1.F22.1.1.30.30.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.30.30.1.1" class="ltx_text" style="font-size:80%;">three times the setup time making snacks, and 20 minutes watching the comet.</span></span></span>
<span id="A1.F22.1.1.31.31" class="ltx_tr">
<span id="A1.F22.1.1.31.31.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.31.31.1.1" class="ltx_text" style="font-size:80%;">What percentage of the total time she spent on all those activities was spent watching the comet,</span></span></span>
<span id="A1.F22.1.1.32.32" class="ltx_tr">
<span id="A1.F22.1.1.32.32.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.32.32.1.1" class="ltx_text" style="font-size:80%;">rounded to the nearest percent?"""</span></span></span>
<span id="A1.F22.1.1.33.33" class="ltx_tr">
<span id="A1.F22.1.1.33.33.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.33.33.1.1" class="ltx_text" style="font-size:80%;">    shopping_time = 2 # hours</span></span></span>
<span id="A1.F22.1.1.34.34" class="ltx_tr">
<span id="A1.F22.1.1.34.34.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.34.34.1.1" class="ltx_text" style="font-size:80%;">    setup_time = 0.5 # hours</span></span></span>
<span id="A1.F22.1.1.35.35" class="ltx_tr">
<span id="A1.F22.1.1.35.35.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.35.35.1.1" class="ltx_text" style="font-size:80%;">    snack_time = setup_time * 3 # hours</span></span></span>
<span id="A1.F22.1.1.36.36" class="ltx_tr">
<span id="A1.F22.1.1.36.36.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.36.36.1.1" class="ltx_text" style="font-size:80%;">    comet_time = 20 # minutes</span></span></span>
<span id="A1.F22.1.1.37.37" class="ltx_tr">
<span id="A1.F22.1.1.37.37.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.37.37.1.1" class="ltx_text" style="font-size:80%;">    total_time = shopping_time + setup_time + snack_time + comet_time</span></span></span>
<span id="A1.F22.1.1.38.38" class="ltx_tr">
<span id="A1.F22.1.1.38.38.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.38.38.1.1" class="ltx_text" style="font-size:80%;">    percentage_comet_time = (comet_time / total_time) * 100</span></span></span>
<span id="A1.F22.1.1.39.39" class="ltx_tr">
<span id="A1.F22.1.1.39.39.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.39.39.1.1" class="ltx_text" style="font-size:80%;">    rounded_percentage_comet_time = round(percentage_comet_time)</span></span></span>
<span id="A1.F22.1.1.40.40" class="ltx_tr">
<span id="A1.F22.1.1.40.40.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.40.40.1.1" class="ltx_text" style="font-size:80%;">    result = rounded_percentage_comet_time</span></span></span>
<span id="A1.F22.1.1.41.41" class="ltx_tr">
<span id="A1.F22.1.1.41.41.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.41.41.1.1" class="ltx_text" style="font-size:80%;">    return result</span></span></span>
<span id="A1.F22.1.1.42.42" class="ltx_tr">
<span id="A1.F22.1.1.42.42.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.42.42.1.1" class="ltx_text" style="font-size:80%;"># is the solution correct?</span></span></span>
<span id="A1.F22.1.1.43.43" class="ltx_tr">
<span id="A1.F22.1.1.43.43.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F22.1.1.43.43.1.1" class="ltx_text" style="font-size:80%;">No</span></span></span>
</span>
</span>
<span id="A1.F22.1.2" class="ltx_text" style="font-size:80%;"></span></p>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_figure">Figure A.22: </span>An example 2-shot code correctness verifier prompt on GSM8K. Here, the solutions are generated using the Gemini 1.0 Pro <cite class="ltx_cite ltx_citemacro_citep">(Gemini Team, <a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite> model. See results in §<a href="#S3.SS4" title="3.4 Reward Modelling: Learning Code Verifiers In-Context ‣ 3 Scaling In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a>.</figcaption>
</figure>
<figure id="A1.F23" class="ltx_figure">
<p id="A1.F23.31" class="ltx_p ltx_align_center"><span id="A1.F23.31.31" class="ltx_text ltx_framed ltx_framed_rectangle" style="font-size:80%;border-color: #000000;">

<span id="A1.F23.31.31.31" class="ltx_inline-block ltx_minipage ltx_align_middle" style="width:390.3pt;">
<span id="A1.F23.1.1.1.1" class="ltx_p">Problem:
Find the domain of the expression <math id="A1.F23.1.1.1.1.m1.1" class="ltx_Math" alttext="\frac{\sqrt{x-2}}{\sqrt{5-x}}" display="inline"><semantics id="A1.F23.1.1.1.1.m1.1a"><mfrac id="A1.F23.1.1.1.1.m1.1.1" xref="A1.F23.1.1.1.1.m1.1.1.cmml"><msqrt id="A1.F23.1.1.1.1.m1.1.1.2" xref="A1.F23.1.1.1.1.m1.1.1.2.cmml"><mrow id="A1.F23.1.1.1.1.m1.1.1.2.2" xref="A1.F23.1.1.1.1.m1.1.1.2.2.cmml"><mi id="A1.F23.1.1.1.1.m1.1.1.2.2.2" xref="A1.F23.1.1.1.1.m1.1.1.2.2.2.cmml">x</mi><mo id="A1.F23.1.1.1.1.m1.1.1.2.2.1" xref="A1.F23.1.1.1.1.m1.1.1.2.2.1.cmml">−</mo><mn id="A1.F23.1.1.1.1.m1.1.1.2.2.3" xref="A1.F23.1.1.1.1.m1.1.1.2.2.3.cmml">2</mn></mrow></msqrt><msqrt id="A1.F23.1.1.1.1.m1.1.1.3" xref="A1.F23.1.1.1.1.m1.1.1.3.cmml"><mrow id="A1.F23.1.1.1.1.m1.1.1.3.2" xref="A1.F23.1.1.1.1.m1.1.1.3.2.cmml"><mn id="A1.F23.1.1.1.1.m1.1.1.3.2.2" xref="A1.F23.1.1.1.1.m1.1.1.3.2.2.cmml">5</mn><mo id="A1.F23.1.1.1.1.m1.1.1.3.2.1" xref="A1.F23.1.1.1.1.m1.1.1.3.2.1.cmml">−</mo><mi id="A1.F23.1.1.1.1.m1.1.1.3.2.3" xref="A1.F23.1.1.1.1.m1.1.1.3.2.3.cmml">x</mi></mrow></msqrt></mfrac><annotation-xml encoding="MathML-Content" id="A1.F23.1.1.1.1.m1.1b"><apply id="A1.F23.1.1.1.1.m1.1.1.cmml" xref="A1.F23.1.1.1.1.m1.1.1"><divide id="A1.F23.1.1.1.1.m1.1.1.1.cmml" xref="A1.F23.1.1.1.1.m1.1.1"></divide><apply id="A1.F23.1.1.1.1.m1.1.1.2.cmml" xref="A1.F23.1.1.1.1.m1.1.1.2"><root id="A1.F23.1.1.1.1.m1.1.1.2a.cmml" xref="A1.F23.1.1.1.1.m1.1.1.2"></root><apply id="A1.F23.1.1.1.1.m1.1.1.2.2.cmml" xref="A1.F23.1.1.1.1.m1.1.1.2.2"><minus id="A1.F23.1.1.1.1.m1.1.1.2.2.1.cmml" xref="A1.F23.1.1.1.1.m1.1.1.2.2.1"></minus><ci id="A1.F23.1.1.1.1.m1.1.1.2.2.2.cmml" xref="A1.F23.1.1.1.1.m1.1.1.2.2.2">𝑥</ci><cn type="integer" id="A1.F23.1.1.1.1.m1.1.1.2.2.3.cmml" xref="A1.F23.1.1.1.1.m1.1.1.2.2.3">2</cn></apply></apply><apply id="A1.F23.1.1.1.1.m1.1.1.3.cmml" xref="A1.F23.1.1.1.1.m1.1.1.3"><root id="A1.F23.1.1.1.1.m1.1.1.3a.cmml" xref="A1.F23.1.1.1.1.m1.1.1.3"></root><apply id="A1.F23.1.1.1.1.m1.1.1.3.2.cmml" xref="A1.F23.1.1.1.1.m1.1.1.3.2"><minus id="A1.F23.1.1.1.1.m1.1.1.3.2.1.cmml" xref="A1.F23.1.1.1.1.m1.1.1.3.2.1"></minus><cn type="integer" id="A1.F23.1.1.1.1.m1.1.1.3.2.2.cmml" xref="A1.F23.1.1.1.1.m1.1.1.3.2.2">5</cn><ci id="A1.F23.1.1.1.1.m1.1.1.3.2.3.cmml" xref="A1.F23.1.1.1.1.m1.1.1.3.2.3">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.1.1.1.1.m1.1c">\frac{\sqrt{x-2}}{\sqrt{5-x}}</annotation></semantics></math>.</span>
<span id="A1.F23.31.31.31.32" class="ltx_p">Solution:</span>
<span id="A1.F23.31.31.31.33" class="ltx_p">I need to find the domain of this expression.</span>
<span id="A1.F23.31.31.31.34" class="ltx_p">This expression has two square root expressions.</span>
<span id="A1.F23.31.31.31.35" class="ltx_p">I know the values under each square root must be non-negative.</span>
<span id="A1.F23.31.31.31.36" class="ltx_p">So let me start by ensuring that.</span>
<span id="A1.F23.3.3.3.3" class="ltx_p">I set <math id="A1.F23.2.2.2.2.m1.1" class="ltx_Math" alttext="x-2\geq 0" display="inline"><semantics id="A1.F23.2.2.2.2.m1.1a"><mrow id="A1.F23.2.2.2.2.m1.1.1" xref="A1.F23.2.2.2.2.m1.1.1.cmml"><mrow id="A1.F23.2.2.2.2.m1.1.1.2" xref="A1.F23.2.2.2.2.m1.1.1.2.cmml"><mi id="A1.F23.2.2.2.2.m1.1.1.2.2" xref="A1.F23.2.2.2.2.m1.1.1.2.2.cmml">x</mi><mo id="A1.F23.2.2.2.2.m1.1.1.2.1" xref="A1.F23.2.2.2.2.m1.1.1.2.1.cmml">−</mo><mn id="A1.F23.2.2.2.2.m1.1.1.2.3" xref="A1.F23.2.2.2.2.m1.1.1.2.3.cmml">2</mn></mrow><mo id="A1.F23.2.2.2.2.m1.1.1.1" xref="A1.F23.2.2.2.2.m1.1.1.1.cmml">≥</mo><mn id="A1.F23.2.2.2.2.m1.1.1.3" xref="A1.F23.2.2.2.2.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.2.2.2.2.m1.1b"><apply id="A1.F23.2.2.2.2.m1.1.1.cmml" xref="A1.F23.2.2.2.2.m1.1.1"><geq id="A1.F23.2.2.2.2.m1.1.1.1.cmml" xref="A1.F23.2.2.2.2.m1.1.1.1"></geq><apply id="A1.F23.2.2.2.2.m1.1.1.2.cmml" xref="A1.F23.2.2.2.2.m1.1.1.2"><minus id="A1.F23.2.2.2.2.m1.1.1.2.1.cmml" xref="A1.F23.2.2.2.2.m1.1.1.2.1"></minus><ci id="A1.F23.2.2.2.2.m1.1.1.2.2.cmml" xref="A1.F23.2.2.2.2.m1.1.1.2.2">𝑥</ci><cn type="integer" id="A1.F23.2.2.2.2.m1.1.1.2.3.cmml" xref="A1.F23.2.2.2.2.m1.1.1.2.3">2</cn></apply><cn type="integer" id="A1.F23.2.2.2.2.m1.1.1.3.cmml" xref="A1.F23.2.2.2.2.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.2.2.2.2.m1.1c">x-2\geq 0</annotation></semantics></math>, which simplifies to <math id="A1.F23.3.3.3.3.m2.1" class="ltx_Math" alttext="x\geq 2" display="inline"><semantics id="A1.F23.3.3.3.3.m2.1a"><mrow id="A1.F23.3.3.3.3.m2.1.1" xref="A1.F23.3.3.3.3.m2.1.1.cmml"><mi id="A1.F23.3.3.3.3.m2.1.1.2" xref="A1.F23.3.3.3.3.m2.1.1.2.cmml">x</mi><mo id="A1.F23.3.3.3.3.m2.1.1.1" xref="A1.F23.3.3.3.3.m2.1.1.1.cmml">≥</mo><mn id="A1.F23.3.3.3.3.m2.1.1.3" xref="A1.F23.3.3.3.3.m2.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.3.3.3.3.m2.1b"><apply id="A1.F23.3.3.3.3.m2.1.1.cmml" xref="A1.F23.3.3.3.3.m2.1.1"><geq id="A1.F23.3.3.3.3.m2.1.1.1.cmml" xref="A1.F23.3.3.3.3.m2.1.1.1"></geq><ci id="A1.F23.3.3.3.3.m2.1.1.2.cmml" xref="A1.F23.3.3.3.3.m2.1.1.2">𝑥</ci><cn type="integer" id="A1.F23.3.3.3.3.m2.1.1.3.cmml" xref="A1.F23.3.3.3.3.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.3.3.3.3.m2.1c">x\geq 2</annotation></semantics></math>.</span>
<span id="A1.F23.5.5.5.5" class="ltx_p">Then, I set <math id="A1.F23.4.4.4.4.m1.1" class="ltx_Math" alttext="5-x\geq 0" display="inline"><semantics id="A1.F23.4.4.4.4.m1.1a"><mrow id="A1.F23.4.4.4.4.m1.1.1" xref="A1.F23.4.4.4.4.m1.1.1.cmml"><mrow id="A1.F23.4.4.4.4.m1.1.1.2" xref="A1.F23.4.4.4.4.m1.1.1.2.cmml"><mn id="A1.F23.4.4.4.4.m1.1.1.2.2" xref="A1.F23.4.4.4.4.m1.1.1.2.2.cmml">5</mn><mo id="A1.F23.4.4.4.4.m1.1.1.2.1" xref="A1.F23.4.4.4.4.m1.1.1.2.1.cmml">−</mo><mi id="A1.F23.4.4.4.4.m1.1.1.2.3" xref="A1.F23.4.4.4.4.m1.1.1.2.3.cmml">x</mi></mrow><mo id="A1.F23.4.4.4.4.m1.1.1.1" xref="A1.F23.4.4.4.4.m1.1.1.1.cmml">≥</mo><mn id="A1.F23.4.4.4.4.m1.1.1.3" xref="A1.F23.4.4.4.4.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.4.4.4.4.m1.1b"><apply id="A1.F23.4.4.4.4.m1.1.1.cmml" xref="A1.F23.4.4.4.4.m1.1.1"><geq id="A1.F23.4.4.4.4.m1.1.1.1.cmml" xref="A1.F23.4.4.4.4.m1.1.1.1"></geq><apply id="A1.F23.4.4.4.4.m1.1.1.2.cmml" xref="A1.F23.4.4.4.4.m1.1.1.2"><minus id="A1.F23.4.4.4.4.m1.1.1.2.1.cmml" xref="A1.F23.4.4.4.4.m1.1.1.2.1"></minus><cn type="integer" id="A1.F23.4.4.4.4.m1.1.1.2.2.cmml" xref="A1.F23.4.4.4.4.m1.1.1.2.2">5</cn><ci id="A1.F23.4.4.4.4.m1.1.1.2.3.cmml" xref="A1.F23.4.4.4.4.m1.1.1.2.3">𝑥</ci></apply><cn type="integer" id="A1.F23.4.4.4.4.m1.1.1.3.cmml" xref="A1.F23.4.4.4.4.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.4.4.4.4.m1.1c">5-x\geq 0</annotation></semantics></math>, which simplifies to <math id="A1.F23.5.5.5.5.m2.1" class="ltx_Math" alttext="x\leq 5" display="inline"><semantics id="A1.F23.5.5.5.5.m2.1a"><mrow id="A1.F23.5.5.5.5.m2.1.1" xref="A1.F23.5.5.5.5.m2.1.1.cmml"><mi id="A1.F23.5.5.5.5.m2.1.1.2" xref="A1.F23.5.5.5.5.m2.1.1.2.cmml">x</mi><mo id="A1.F23.5.5.5.5.m2.1.1.1" xref="A1.F23.5.5.5.5.m2.1.1.1.cmml">≤</mo><mn id="A1.F23.5.5.5.5.m2.1.1.3" xref="A1.F23.5.5.5.5.m2.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.5.5.5.5.m2.1b"><apply id="A1.F23.5.5.5.5.m2.1.1.cmml" xref="A1.F23.5.5.5.5.m2.1.1"><leq id="A1.F23.5.5.5.5.m2.1.1.1.cmml" xref="A1.F23.5.5.5.5.m2.1.1.1"></leq><ci id="A1.F23.5.5.5.5.m2.1.1.2.cmml" xref="A1.F23.5.5.5.5.m2.1.1.2">𝑥</ci><cn type="integer" id="A1.F23.5.5.5.5.m2.1.1.3.cmml" xref="A1.F23.5.5.5.5.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.5.5.5.5.m2.1c">x\leq 5</annotation></semantics></math>.</span>
<span id="A1.F23.31.31.31.37" class="ltx_p">But I need to make sure the denominator is not zero.</span>
<span id="A1.F23.7.7.7.7" class="ltx_p">So, <math id="A1.F23.6.6.6.6.m1.1" class="ltx_Math" alttext="5-x" display="inline"><semantics id="A1.F23.6.6.6.6.m1.1a"><mrow id="A1.F23.6.6.6.6.m1.1.1" xref="A1.F23.6.6.6.6.m1.1.1.cmml"><mn id="A1.F23.6.6.6.6.m1.1.1.2" xref="A1.F23.6.6.6.6.m1.1.1.2.cmml">5</mn><mo id="A1.F23.6.6.6.6.m1.1.1.1" xref="A1.F23.6.6.6.6.m1.1.1.1.cmml">−</mo><mi id="A1.F23.6.6.6.6.m1.1.1.3" xref="A1.F23.6.6.6.6.m1.1.1.3.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.6.6.6.6.m1.1b"><apply id="A1.F23.6.6.6.6.m1.1.1.cmml" xref="A1.F23.6.6.6.6.m1.1.1"><minus id="A1.F23.6.6.6.6.m1.1.1.1.cmml" xref="A1.F23.6.6.6.6.m1.1.1.1"></minus><cn type="integer" id="A1.F23.6.6.6.6.m1.1.1.2.cmml" xref="A1.F23.6.6.6.6.m1.1.1.2">5</cn><ci id="A1.F23.6.6.6.6.m1.1.1.3.cmml" xref="A1.F23.6.6.6.6.m1.1.1.3">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.6.6.6.6.m1.1c">5-x</annotation></semantics></math> must also be greater than zero, which gives me <math id="A1.F23.7.7.7.7.m2.1" class="ltx_Math" alttext="x&lt;5" display="inline"><semantics id="A1.F23.7.7.7.7.m2.1a"><mrow id="A1.F23.7.7.7.7.m2.1.1" xref="A1.F23.7.7.7.7.m2.1.1.cmml"><mi id="A1.F23.7.7.7.7.m2.1.1.2" xref="A1.F23.7.7.7.7.m2.1.1.2.cmml">x</mi><mo id="A1.F23.7.7.7.7.m2.1.1.1" xref="A1.F23.7.7.7.7.m2.1.1.1.cmml">&lt;</mo><mn id="A1.F23.7.7.7.7.m2.1.1.3" xref="A1.F23.7.7.7.7.m2.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.7.7.7.7.m2.1b"><apply id="A1.F23.7.7.7.7.m2.1.1.cmml" xref="A1.F23.7.7.7.7.m2.1.1"><lt id="A1.F23.7.7.7.7.m2.1.1.1.cmml" xref="A1.F23.7.7.7.7.m2.1.1.1"></lt><ci id="A1.F23.7.7.7.7.m2.1.1.2.cmml" xref="A1.F23.7.7.7.7.m2.1.1.2">𝑥</ci><cn type="integer" id="A1.F23.7.7.7.7.m2.1.1.3.cmml" xref="A1.F23.7.7.7.7.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.7.7.7.7.m2.1c">x&lt;5</annotation></semantics></math>.</span>
<span id="A1.F23.8.8.8.8" class="ltx_p">Therefore, the domain of the expression is <math id="A1.F23.8.8.8.8.m1.2" class="ltx_Math" alttext="\boxed{[2,5)}" display="inline"><semantics id="A1.F23.8.8.8.8.m1.2a"><menclose notation="box" id="A1.F23.8.8.8.8.m1.2.2" xref="A1.F23.8.8.8.8.m1.2.2.3.cmml"><mrow id="A1.F23.8.8.8.8.m1.2.2.4" xref="A1.F23.8.8.8.8.m1.2.2.3.cmml"><mo stretchy="false" id="A1.F23.8.8.8.8.m1.2.2.4.1" xref="A1.F23.8.8.8.8.m1.2.2.3.cmml">[</mo><mn id="A1.F23.8.8.8.8.m1.1.1.1" xref="A1.F23.8.8.8.8.m1.1.1.1.cmml">2</mn><mo id="A1.F23.8.8.8.8.m1.2.2.4.2" xref="A1.F23.8.8.8.8.m1.2.2.3.cmml">,</mo><mn id="A1.F23.8.8.8.8.m1.2.2.2" xref="A1.F23.8.8.8.8.m1.2.2.2.cmml">5</mn><mo stretchy="false" id="A1.F23.8.8.8.8.m1.2.2.4.3" xref="A1.F23.8.8.8.8.m1.2.2.3.cmml">)</mo></mrow></menclose><annotation-xml encoding="MathML-Content" id="A1.F23.8.8.8.8.m1.2b"><interval closure="closed-open" id="A1.F23.8.8.8.8.m1.2.2.3.cmml" xref="A1.F23.8.8.8.8.m1.2.2"><cn type="integer" id="A1.F23.8.8.8.8.m1.1.1.1.cmml" xref="A1.F23.8.8.8.8.m1.1.1.1">2</cn><cn type="integer" id="A1.F23.8.8.8.8.m1.2.2.2.cmml" xref="A1.F23.8.8.8.8.m1.2.2.2">5</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.8.8.8.8.m1.2c">\boxed{[2,5)}</annotation></semantics></math>.</span>
<span id="A1.F23.9.9.9.9" class="ltx_p">Answer: <math id="A1.F23.9.9.9.9.m1.2" class="ltx_Math" alttext="[2,5)" display="inline"><semantics id="A1.F23.9.9.9.9.m1.2a"><mrow id="A1.F23.9.9.9.9.m1.2.3.2" xref="A1.F23.9.9.9.9.m1.2.3.1.cmml"><mo stretchy="false" id="A1.F23.9.9.9.9.m1.2.3.2.1" xref="A1.F23.9.9.9.9.m1.2.3.1.cmml">[</mo><mn id="A1.F23.9.9.9.9.m1.1.1" xref="A1.F23.9.9.9.9.m1.1.1.cmml">2</mn><mo id="A1.F23.9.9.9.9.m1.2.3.2.2" xref="A1.F23.9.9.9.9.m1.2.3.1.cmml">,</mo><mn id="A1.F23.9.9.9.9.m1.2.2" xref="A1.F23.9.9.9.9.m1.2.2.cmml">5</mn><mo stretchy="false" id="A1.F23.9.9.9.9.m1.2.3.2.3" xref="A1.F23.9.9.9.9.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.9.9.9.9.m1.2b"><interval closure="closed-open" id="A1.F23.9.9.9.9.m1.2.3.1.cmml" xref="A1.F23.9.9.9.9.m1.2.3.2"><cn type="integer" id="A1.F23.9.9.9.9.m1.1.1.cmml" xref="A1.F23.9.9.9.9.m1.1.1">2</cn><cn type="integer" id="A1.F23.9.9.9.9.m1.2.2.cmml" xref="A1.F23.9.9.9.9.m1.2.2">5</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.9.9.9.9.m1.2c">[2,5)</annotation></semantics></math>.</span>
<span id="A1.F23.10.10.10.10" class="ltx_p">Final Answer: The final answer is <math id="A1.F23.10.10.10.10.m1.2" class="ltx_Math" alttext="[2,5)" display="inline"><semantics id="A1.F23.10.10.10.10.m1.2a"><mrow id="A1.F23.10.10.10.10.m1.2.3.2" xref="A1.F23.10.10.10.10.m1.2.3.1.cmml"><mo stretchy="false" id="A1.F23.10.10.10.10.m1.2.3.2.1" xref="A1.F23.10.10.10.10.m1.2.3.1.cmml">[</mo><mn id="A1.F23.10.10.10.10.m1.1.1" xref="A1.F23.10.10.10.10.m1.1.1.cmml">2</mn><mo id="A1.F23.10.10.10.10.m1.2.3.2.2" xref="A1.F23.10.10.10.10.m1.2.3.1.cmml">,</mo><mn id="A1.F23.10.10.10.10.m1.2.2" xref="A1.F23.10.10.10.10.m1.2.2.cmml">5</mn><mo stretchy="false" id="A1.F23.10.10.10.10.m1.2.3.2.3" xref="A1.F23.10.10.10.10.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.10.10.10.10.m1.2b"><interval closure="closed-open" id="A1.F23.10.10.10.10.m1.2.3.1.cmml" xref="A1.F23.10.10.10.10.m1.2.3.2"><cn type="integer" id="A1.F23.10.10.10.10.m1.1.1.cmml" xref="A1.F23.10.10.10.10.m1.1.1">2</cn><cn type="integer" id="A1.F23.10.10.10.10.m1.2.2.cmml" xref="A1.F23.10.10.10.10.m1.2.2">5</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.10.10.10.10.m1.2c">[2,5)</annotation></semantics></math>. I hope it is correct.</span>
<span id="A1.F23.31.31.31.38" class="ltx_p">—</span>
<span id="A1.F23.13.13.13.13" class="ltx_p">Problem:
If <math id="A1.F23.11.11.11.11.m1.1" class="ltx_Math" alttext="\det\mathbf{A}=2" display="inline"><semantics id="A1.F23.11.11.11.11.m1.1a"><mrow id="A1.F23.11.11.11.11.m1.1.1" xref="A1.F23.11.11.11.11.m1.1.1.cmml"><mrow id="A1.F23.11.11.11.11.m1.1.1.2" xref="A1.F23.11.11.11.11.m1.1.1.2.cmml"><mo rspace="0.167em" id="A1.F23.11.11.11.11.m1.1.1.2.1" xref="A1.F23.11.11.11.11.m1.1.1.2.1.cmml">det</mo><mi id="A1.F23.11.11.11.11.m1.1.1.2.2" xref="A1.F23.11.11.11.11.m1.1.1.2.2.cmml">𝐀</mi></mrow><mo id="A1.F23.11.11.11.11.m1.1.1.1" xref="A1.F23.11.11.11.11.m1.1.1.1.cmml">=</mo><mn id="A1.F23.11.11.11.11.m1.1.1.3" xref="A1.F23.11.11.11.11.m1.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.11.11.11.11.m1.1b"><apply id="A1.F23.11.11.11.11.m1.1.1.cmml" xref="A1.F23.11.11.11.11.m1.1.1"><eq id="A1.F23.11.11.11.11.m1.1.1.1.cmml" xref="A1.F23.11.11.11.11.m1.1.1.1"></eq><apply id="A1.F23.11.11.11.11.m1.1.1.2.cmml" xref="A1.F23.11.11.11.11.m1.1.1.2"><determinant id="A1.F23.11.11.11.11.m1.1.1.2.1.cmml" xref="A1.F23.11.11.11.11.m1.1.1.2.1"></determinant><ci id="A1.F23.11.11.11.11.m1.1.1.2.2.cmml" xref="A1.F23.11.11.11.11.m1.1.1.2.2">𝐀</ci></apply><cn type="integer" id="A1.F23.11.11.11.11.m1.1.1.3.cmml" xref="A1.F23.11.11.11.11.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.11.11.11.11.m1.1c">\det\mathbf{A}=2</annotation></semantics></math> and <math id="A1.F23.12.12.12.12.m2.1" class="ltx_Math" alttext="\det\mathbf{B}=12," display="inline"><semantics id="A1.F23.12.12.12.12.m2.1a"><mrow id="A1.F23.12.12.12.12.m2.1.1.1" xref="A1.F23.12.12.12.12.m2.1.1.1.1.cmml"><mrow id="A1.F23.12.12.12.12.m2.1.1.1.1" xref="A1.F23.12.12.12.12.m2.1.1.1.1.cmml"><mrow id="A1.F23.12.12.12.12.m2.1.1.1.1.2" xref="A1.F23.12.12.12.12.m2.1.1.1.1.2.cmml"><mo rspace="0.167em" id="A1.F23.12.12.12.12.m2.1.1.1.1.2.1" xref="A1.F23.12.12.12.12.m2.1.1.1.1.2.1.cmml">det</mo><mi id="A1.F23.12.12.12.12.m2.1.1.1.1.2.2" xref="A1.F23.12.12.12.12.m2.1.1.1.1.2.2.cmml">𝐁</mi></mrow><mo id="A1.F23.12.12.12.12.m2.1.1.1.1.1" xref="A1.F23.12.12.12.12.m2.1.1.1.1.1.cmml">=</mo><mn id="A1.F23.12.12.12.12.m2.1.1.1.1.3" xref="A1.F23.12.12.12.12.m2.1.1.1.1.3.cmml">12</mn></mrow><mo id="A1.F23.12.12.12.12.m2.1.1.1.2" xref="A1.F23.12.12.12.12.m2.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.12.12.12.12.m2.1b"><apply id="A1.F23.12.12.12.12.m2.1.1.1.1.cmml" xref="A1.F23.12.12.12.12.m2.1.1.1"><eq id="A1.F23.12.12.12.12.m2.1.1.1.1.1.cmml" xref="A1.F23.12.12.12.12.m2.1.1.1.1.1"></eq><apply id="A1.F23.12.12.12.12.m2.1.1.1.1.2.cmml" xref="A1.F23.12.12.12.12.m2.1.1.1.1.2"><determinant id="A1.F23.12.12.12.12.m2.1.1.1.1.2.1.cmml" xref="A1.F23.12.12.12.12.m2.1.1.1.1.2.1"></determinant><ci id="A1.F23.12.12.12.12.m2.1.1.1.1.2.2.cmml" xref="A1.F23.12.12.12.12.m2.1.1.1.1.2.2">𝐁</ci></apply><cn type="integer" id="A1.F23.12.12.12.12.m2.1.1.1.1.3.cmml" xref="A1.F23.12.12.12.12.m2.1.1.1.1.3">12</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.12.12.12.12.m2.1c">\det\mathbf{B}=12,</annotation></semantics></math> then find <math id="A1.F23.13.13.13.13.m3.2" class="ltx_Math" alttext="\det(\mathbf{A}\mathbf{B})." display="inline"><semantics id="A1.F23.13.13.13.13.m3.2a"><mrow id="A1.F23.13.13.13.13.m3.2.2.1" xref="A1.F23.13.13.13.13.m3.2.2.1.1.cmml"><mrow id="A1.F23.13.13.13.13.m3.2.2.1.1" xref="A1.F23.13.13.13.13.m3.2.2.1.1.cmml"><mo rspace="0em" id="A1.F23.13.13.13.13.m3.2.2.1.1.1" xref="A1.F23.13.13.13.13.m3.2.2.1.1.1.cmml">det</mo><mrow id="A1.F23.13.13.13.13.m3.2.2.1.1.2.2" xref="A1.F23.13.13.13.13.m3.2.2.1.1.cmml"><mo stretchy="false" id="A1.F23.13.13.13.13.m3.2.2.1.1.2.2.1" xref="A1.F23.13.13.13.13.m3.2.2.1.1.cmml">(</mo><mi id="A1.F23.13.13.13.13.m3.1.1" xref="A1.F23.13.13.13.13.m3.1.1.cmml">𝐀𝐁</mi><mo stretchy="false" id="A1.F23.13.13.13.13.m3.2.2.1.1.2.2.2" xref="A1.F23.13.13.13.13.m3.2.2.1.1.cmml">)</mo></mrow></mrow><mo lspace="0em" id="A1.F23.13.13.13.13.m3.2.2.1.2" xref="A1.F23.13.13.13.13.m3.2.2.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.13.13.13.13.m3.2b"><apply id="A1.F23.13.13.13.13.m3.2.2.1.1.cmml" xref="A1.F23.13.13.13.13.m3.2.2.1"><determinant id="A1.F23.13.13.13.13.m3.2.2.1.1.1.cmml" xref="A1.F23.13.13.13.13.m3.2.2.1.1.1"></determinant><ci id="A1.F23.13.13.13.13.m3.1.1.cmml" xref="A1.F23.13.13.13.13.m3.1.1">𝐀𝐁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.13.13.13.13.m3.2c">\det(\mathbf{A}\mathbf{B}).</annotation></semantics></math></span>
<span id="A1.F23.31.31.31.39" class="ltx_p">Solution:</span>
<span id="A1.F23.31.31.31.40" class="ltx_p">I need to find the determinant of the product of matrices A and B.</span>
<span id="A1.F23.31.31.31.41" class="ltx_p">I remember that the determinant of the product of two matrices equals the product of their determinants.</span>
<span id="A1.F23.14.14.14.14" class="ltx_p">So, <math id="A1.F23.14.14.14.14.m1.5" class="ltx_Math" alttext="\det(\mathbf{A}\mathbf{B})=(\det\mathbf{A})(\det\mathbf{B})=(2)(12)=\boxed{24}" display="inline"><semantics id="A1.F23.14.14.14.14.m1.5a"><mrow id="A1.F23.14.14.14.14.m1.5.5" xref="A1.F23.14.14.14.14.m1.5.5.cmml"><mrow id="A1.F23.14.14.14.14.m1.5.5.4" xref="A1.F23.14.14.14.14.m1.5.5.4.cmml"><mo rspace="0em" id="A1.F23.14.14.14.14.m1.5.5.4.1" xref="A1.F23.14.14.14.14.m1.5.5.4.1.cmml">det</mo><mrow id="A1.F23.14.14.14.14.m1.5.5.4.2.2" xref="A1.F23.14.14.14.14.m1.5.5.4.cmml"><mo stretchy="false" id="A1.F23.14.14.14.14.m1.5.5.4.2.2.1" xref="A1.F23.14.14.14.14.m1.5.5.4.cmml">(</mo><mi id="A1.F23.14.14.14.14.m1.1.1" xref="A1.F23.14.14.14.14.m1.1.1.cmml">𝐀𝐁</mi><mo stretchy="false" id="A1.F23.14.14.14.14.m1.5.5.4.2.2.2" xref="A1.F23.14.14.14.14.m1.5.5.4.cmml">)</mo></mrow></mrow><mo id="A1.F23.14.14.14.14.m1.5.5.5" xref="A1.F23.14.14.14.14.m1.5.5.5.cmml">=</mo><mrow id="A1.F23.14.14.14.14.m1.5.5.2" xref="A1.F23.14.14.14.14.m1.5.5.2.cmml"><mrow id="A1.F23.14.14.14.14.m1.4.4.1.1.1" xref="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.cmml"><mo stretchy="false" id="A1.F23.14.14.14.14.m1.4.4.1.1.1.2" xref="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.cmml">(</mo><mrow id="A1.F23.14.14.14.14.m1.4.4.1.1.1.1" xref="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.cmml"><mo lspace="0em" rspace="0.167em" id="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.1" xref="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.1.cmml">det</mo><mi id="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.2" xref="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.2.cmml">𝐀</mi></mrow><mo stretchy="false" id="A1.F23.14.14.14.14.m1.4.4.1.1.1.3" xref="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.F23.14.14.14.14.m1.5.5.2.3" xref="A1.F23.14.14.14.14.m1.5.5.2.3.cmml">​</mo><mrow id="A1.F23.14.14.14.14.m1.5.5.2.2.1" xref="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.cmml"><mo stretchy="false" id="A1.F23.14.14.14.14.m1.5.5.2.2.1.2" xref="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.cmml">(</mo><mrow id="A1.F23.14.14.14.14.m1.5.5.2.2.1.1" xref="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.cmml"><mo lspace="0em" rspace="0.167em" id="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.1" xref="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.1.cmml">det</mo><mi id="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.2" xref="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.2.cmml">𝐁</mi></mrow><mo stretchy="false" id="A1.F23.14.14.14.14.m1.5.5.2.2.1.3" xref="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.cmml">)</mo></mrow></mrow><mo id="A1.F23.14.14.14.14.m1.5.5.6" xref="A1.F23.14.14.14.14.m1.5.5.6.cmml">=</mo><mrow id="A1.F23.14.14.14.14.m1.5.5.7" xref="A1.F23.14.14.14.14.m1.5.5.7.cmml"><mrow id="A1.F23.14.14.14.14.m1.5.5.7.2.2" xref="A1.F23.14.14.14.14.m1.5.5.7.cmml"><mo stretchy="false" id="A1.F23.14.14.14.14.m1.5.5.7.2.2.1" xref="A1.F23.14.14.14.14.m1.5.5.7.cmml">(</mo><mn id="A1.F23.14.14.14.14.m1.2.2" xref="A1.F23.14.14.14.14.m1.2.2.cmml">2</mn><mo stretchy="false" id="A1.F23.14.14.14.14.m1.5.5.7.2.2.2" xref="A1.F23.14.14.14.14.m1.5.5.7.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.F23.14.14.14.14.m1.5.5.7.1" xref="A1.F23.14.14.14.14.m1.5.5.7.1.cmml">​</mo><mrow id="A1.F23.14.14.14.14.m1.5.5.7.3.2" xref="A1.F23.14.14.14.14.m1.5.5.7.cmml"><mo stretchy="false" id="A1.F23.14.14.14.14.m1.5.5.7.3.2.1" xref="A1.F23.14.14.14.14.m1.5.5.7.cmml">(</mo><mn id="A1.F23.14.14.14.14.m1.3.3" xref="A1.F23.14.14.14.14.m1.3.3.cmml">12</mn><mo stretchy="false" id="A1.F23.14.14.14.14.m1.5.5.7.3.2.2" xref="A1.F23.14.14.14.14.m1.5.5.7.cmml">)</mo></mrow></mrow><mo id="A1.F23.14.14.14.14.m1.5.5.8" xref="A1.F23.14.14.14.14.m1.5.5.8.cmml">=</mo><menclose notation="box" id="A1.F23.14.14.14.14.m1.5.5.9" xref="A1.F23.14.14.14.14.m1.5.5.9.cmml"><mn id="A1.F23.14.14.14.14.m1.5.5.9a" xref="A1.F23.14.14.14.14.m1.5.5.9.cmml">24</mn></menclose></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.14.14.14.14.m1.5b"><apply id="A1.F23.14.14.14.14.m1.5.5.cmml" xref="A1.F23.14.14.14.14.m1.5.5"><and id="A1.F23.14.14.14.14.m1.5.5a.cmml" xref="A1.F23.14.14.14.14.m1.5.5"></and><apply id="A1.F23.14.14.14.14.m1.5.5b.cmml" xref="A1.F23.14.14.14.14.m1.5.5"><eq id="A1.F23.14.14.14.14.m1.5.5.5.cmml" xref="A1.F23.14.14.14.14.m1.5.5.5"></eq><apply id="A1.F23.14.14.14.14.m1.5.5.4.cmml" xref="A1.F23.14.14.14.14.m1.5.5.4"><determinant id="A1.F23.14.14.14.14.m1.5.5.4.1.cmml" xref="A1.F23.14.14.14.14.m1.5.5.4.1"></determinant><ci id="A1.F23.14.14.14.14.m1.1.1.cmml" xref="A1.F23.14.14.14.14.m1.1.1">𝐀𝐁</ci></apply><apply id="A1.F23.14.14.14.14.m1.5.5.2.cmml" xref="A1.F23.14.14.14.14.m1.5.5.2"><times id="A1.F23.14.14.14.14.m1.5.5.2.3.cmml" xref="A1.F23.14.14.14.14.m1.5.5.2.3"></times><apply id="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.cmml" xref="A1.F23.14.14.14.14.m1.4.4.1.1.1"><determinant id="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.1.cmml" xref="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.1"></determinant><ci id="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.2.cmml" xref="A1.F23.14.14.14.14.m1.4.4.1.1.1.1.2">𝐀</ci></apply><apply id="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.cmml" xref="A1.F23.14.14.14.14.m1.5.5.2.2.1"><determinant id="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.1.cmml" xref="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.1"></determinant><ci id="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.2.cmml" xref="A1.F23.14.14.14.14.m1.5.5.2.2.1.1.2">𝐁</ci></apply></apply></apply><apply id="A1.F23.14.14.14.14.m1.5.5c.cmml" xref="A1.F23.14.14.14.14.m1.5.5"><eq id="A1.F23.14.14.14.14.m1.5.5.6.cmml" xref="A1.F23.14.14.14.14.m1.5.5.6"></eq><share href="#A1.F23.14.14.14.14.m1.5.5.2.cmml" id="A1.F23.14.14.14.14.m1.5.5d.cmml" xref="A1.F23.14.14.14.14.m1.5.5"></share><apply id="A1.F23.14.14.14.14.m1.5.5.7.cmml" xref="A1.F23.14.14.14.14.m1.5.5.7"><times id="A1.F23.14.14.14.14.m1.5.5.7.1.cmml" xref="A1.F23.14.14.14.14.m1.5.5.7.1"></times><cn type="integer" id="A1.F23.14.14.14.14.m1.2.2.cmml" xref="A1.F23.14.14.14.14.m1.2.2">2</cn><cn type="integer" id="A1.F23.14.14.14.14.m1.3.3.cmml" xref="A1.F23.14.14.14.14.m1.3.3">12</cn></apply></apply><apply id="A1.F23.14.14.14.14.m1.5.5e.cmml" xref="A1.F23.14.14.14.14.m1.5.5"><eq id="A1.F23.14.14.14.14.m1.5.5.8.cmml" xref="A1.F23.14.14.14.14.m1.5.5.8"></eq><share href="#A1.F23.14.14.14.14.m1.5.5.7.cmml" id="A1.F23.14.14.14.14.m1.5.5f.cmml" xref="A1.F23.14.14.14.14.m1.5.5"></share><cn type="integer" id="A1.F23.14.14.14.14.m1.5.5.9.cmml" xref="A1.F23.14.14.14.14.m1.5.5.9">24</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.14.14.14.14.m1.5c">\det(\mathbf{A}\mathbf{B})=(\det\mathbf{A})(\det\mathbf{B})=(2)(12)=\boxed{24}</annotation></semantics></math>.</span>
<span id="A1.F23.15.15.15.15" class="ltx_p">Answer: <math id="A1.F23.15.15.15.15.m1.1" class="ltx_Math" alttext="24" display="inline"><semantics id="A1.F23.15.15.15.15.m1.1a"><mn id="A1.F23.15.15.15.15.m1.1.1" xref="A1.F23.15.15.15.15.m1.1.1.cmml">24</mn><annotation-xml encoding="MathML-Content" id="A1.F23.15.15.15.15.m1.1b"><cn type="integer" id="A1.F23.15.15.15.15.m1.1.1.cmml" xref="A1.F23.15.15.15.15.m1.1.1">24</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.15.15.15.15.m1.1c">24</annotation></semantics></math>.</span>
<span id="A1.F23.16.16.16.16" class="ltx_p">Final Answer: The final answer is <math id="A1.F23.16.16.16.16.m1.1" class="ltx_Math" alttext="24" display="inline"><semantics id="A1.F23.16.16.16.16.m1.1a"><mn id="A1.F23.16.16.16.16.m1.1.1" xref="A1.F23.16.16.16.16.m1.1.1.cmml">24</mn><annotation-xml encoding="MathML-Content" id="A1.F23.16.16.16.16.m1.1b"><cn type="integer" id="A1.F23.16.16.16.16.m1.1.1.cmml" xref="A1.F23.16.16.16.16.m1.1.1">24</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.16.16.16.16.m1.1c">24</annotation></semantics></math>. I hope it is correct.</span>
<span id="A1.F23.31.31.31.42" class="ltx_p">—</span>
<span id="A1.F23.31.31.31.43" class="ltx_p">Problem:
Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?</span>
<span id="A1.F23.31.31.31.44" class="ltx_p">Solution:</span>
<span id="A1.F23.17.17.17.17" class="ltx_p">Okay, so Terrell lifts a total of <math id="A1.F23.17.17.17.17.m1.1" class="ltx_Math" alttext="2\cdot 12\cdot 20=480" display="inline"><semantics id="A1.F23.17.17.17.17.m1.1a"><mrow id="A1.F23.17.17.17.17.m1.1.1" xref="A1.F23.17.17.17.17.m1.1.1.cmml"><mrow id="A1.F23.17.17.17.17.m1.1.1.2" xref="A1.F23.17.17.17.17.m1.1.1.2.cmml"><mn id="A1.F23.17.17.17.17.m1.1.1.2.2" xref="A1.F23.17.17.17.17.m1.1.1.2.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="A1.F23.17.17.17.17.m1.1.1.2.1" xref="A1.F23.17.17.17.17.m1.1.1.2.1.cmml">⋅</mo><mn id="A1.F23.17.17.17.17.m1.1.1.2.3" xref="A1.F23.17.17.17.17.m1.1.1.2.3.cmml">12</mn><mo lspace="0.222em" rspace="0.222em" id="A1.F23.17.17.17.17.m1.1.1.2.1a" xref="A1.F23.17.17.17.17.m1.1.1.2.1.cmml">⋅</mo><mn id="A1.F23.17.17.17.17.m1.1.1.2.4" xref="A1.F23.17.17.17.17.m1.1.1.2.4.cmml">20</mn></mrow><mo id="A1.F23.17.17.17.17.m1.1.1.1" xref="A1.F23.17.17.17.17.m1.1.1.1.cmml">=</mo><mn id="A1.F23.17.17.17.17.m1.1.1.3" xref="A1.F23.17.17.17.17.m1.1.1.3.cmml">480</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.17.17.17.17.m1.1b"><apply id="A1.F23.17.17.17.17.m1.1.1.cmml" xref="A1.F23.17.17.17.17.m1.1.1"><eq id="A1.F23.17.17.17.17.m1.1.1.1.cmml" xref="A1.F23.17.17.17.17.m1.1.1.1"></eq><apply id="A1.F23.17.17.17.17.m1.1.1.2.cmml" xref="A1.F23.17.17.17.17.m1.1.1.2"><ci id="A1.F23.17.17.17.17.m1.1.1.2.1.cmml" xref="A1.F23.17.17.17.17.m1.1.1.2.1">⋅</ci><cn type="integer" id="A1.F23.17.17.17.17.m1.1.1.2.2.cmml" xref="A1.F23.17.17.17.17.m1.1.1.2.2">2</cn><cn type="integer" id="A1.F23.17.17.17.17.m1.1.1.2.3.cmml" xref="A1.F23.17.17.17.17.m1.1.1.2.3">12</cn><cn type="integer" id="A1.F23.17.17.17.17.m1.1.1.2.4.cmml" xref="A1.F23.17.17.17.17.m1.1.1.2.4">20</cn></apply><cn type="integer" id="A1.F23.17.17.17.17.m1.1.1.3.cmml" xref="A1.F23.17.17.17.17.m1.1.1.3">480</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.17.17.17.17.m1.1c">2\cdot 12\cdot 20=480</annotation></semantics></math> pounds of weight with the 20-pound weights.</span>
<span id="A1.F23.18.18.18.18" class="ltx_p">Well, if he switches to 15-pound weights, the total weight lifted will be <math id="A1.F23.18.18.18.18.m1.1" class="ltx_Math" alttext="2\cdot 15\cdot n=30n" display="inline"><semantics id="A1.F23.18.18.18.18.m1.1a"><mrow id="A1.F23.18.18.18.18.m1.1.1" xref="A1.F23.18.18.18.18.m1.1.1.cmml"><mrow id="A1.F23.18.18.18.18.m1.1.1.2" xref="A1.F23.18.18.18.18.m1.1.1.2.cmml"><mn id="A1.F23.18.18.18.18.m1.1.1.2.2" xref="A1.F23.18.18.18.18.m1.1.1.2.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="A1.F23.18.18.18.18.m1.1.1.2.1" xref="A1.F23.18.18.18.18.m1.1.1.2.1.cmml">⋅</mo><mn id="A1.F23.18.18.18.18.m1.1.1.2.3" xref="A1.F23.18.18.18.18.m1.1.1.2.3.cmml">15</mn><mo lspace="0.222em" rspace="0.222em" id="A1.F23.18.18.18.18.m1.1.1.2.1a" xref="A1.F23.18.18.18.18.m1.1.1.2.1.cmml">⋅</mo><mi id="A1.F23.18.18.18.18.m1.1.1.2.4" xref="A1.F23.18.18.18.18.m1.1.1.2.4.cmml">n</mi></mrow><mo id="A1.F23.18.18.18.18.m1.1.1.1" xref="A1.F23.18.18.18.18.m1.1.1.1.cmml">=</mo><mrow id="A1.F23.18.18.18.18.m1.1.1.3" xref="A1.F23.18.18.18.18.m1.1.1.3.cmml"><mn id="A1.F23.18.18.18.18.m1.1.1.3.2" xref="A1.F23.18.18.18.18.m1.1.1.3.2.cmml">30</mn><mo lspace="0em" rspace="0em" id="A1.F23.18.18.18.18.m1.1.1.3.1" xref="A1.F23.18.18.18.18.m1.1.1.3.1.cmml">​</mo><mi id="A1.F23.18.18.18.18.m1.1.1.3.3" xref="A1.F23.18.18.18.18.m1.1.1.3.3.cmml">n</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.18.18.18.18.m1.1b"><apply id="A1.F23.18.18.18.18.m1.1.1.cmml" xref="A1.F23.18.18.18.18.m1.1.1"><eq id="A1.F23.18.18.18.18.m1.1.1.1.cmml" xref="A1.F23.18.18.18.18.m1.1.1.1"></eq><apply id="A1.F23.18.18.18.18.m1.1.1.2.cmml" xref="A1.F23.18.18.18.18.m1.1.1.2"><ci id="A1.F23.18.18.18.18.m1.1.1.2.1.cmml" xref="A1.F23.18.18.18.18.m1.1.1.2.1">⋅</ci><cn type="integer" id="A1.F23.18.18.18.18.m1.1.1.2.2.cmml" xref="A1.F23.18.18.18.18.m1.1.1.2.2">2</cn><cn type="integer" id="A1.F23.18.18.18.18.m1.1.1.2.3.cmml" xref="A1.F23.18.18.18.18.m1.1.1.2.3">15</cn><ci id="A1.F23.18.18.18.18.m1.1.1.2.4.cmml" xref="A1.F23.18.18.18.18.m1.1.1.2.4">𝑛</ci></apply><apply id="A1.F23.18.18.18.18.m1.1.1.3.cmml" xref="A1.F23.18.18.18.18.m1.1.1.3"><times id="A1.F23.18.18.18.18.m1.1.1.3.1.cmml" xref="A1.F23.18.18.18.18.m1.1.1.3.1"></times><cn type="integer" id="A1.F23.18.18.18.18.m1.1.1.3.2.cmml" xref="A1.F23.18.18.18.18.m1.1.1.3.2">30</cn><ci id="A1.F23.18.18.18.18.m1.1.1.3.3.cmml" xref="A1.F23.18.18.18.18.m1.1.1.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.18.18.18.18.m1.1c">2\cdot 15\cdot n=30n</annotation></semantics></math> pounds, where n is the number of lifts.</span>
<span id="A1.F23.31.31.31.45" class="ltx_p">I want to find the number of lifts, n, for the total weight lifted to be the same.</span>
<span id="A1.F23.19.19.19.19" class="ltx_p">I equate <math id="A1.F23.19.19.19.19.m1.1" class="ltx_Math" alttext="30n" display="inline"><semantics id="A1.F23.19.19.19.19.m1.1a"><mrow id="A1.F23.19.19.19.19.m1.1.1" xref="A1.F23.19.19.19.19.m1.1.1.cmml"><mn id="A1.F23.19.19.19.19.m1.1.1.2" xref="A1.F23.19.19.19.19.m1.1.1.2.cmml">30</mn><mo lspace="0em" rspace="0em" id="A1.F23.19.19.19.19.m1.1.1.1" xref="A1.F23.19.19.19.19.m1.1.1.1.cmml">​</mo><mi id="A1.F23.19.19.19.19.m1.1.1.3" xref="A1.F23.19.19.19.19.m1.1.1.3.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.19.19.19.19.m1.1b"><apply id="A1.F23.19.19.19.19.m1.1.1.cmml" xref="A1.F23.19.19.19.19.m1.1.1"><times id="A1.F23.19.19.19.19.m1.1.1.1.cmml" xref="A1.F23.19.19.19.19.m1.1.1.1"></times><cn type="integer" id="A1.F23.19.19.19.19.m1.1.1.2.cmml" xref="A1.F23.19.19.19.19.m1.1.1.2">30</cn><ci id="A1.F23.19.19.19.19.m1.1.1.3.cmml" xref="A1.F23.19.19.19.19.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.19.19.19.19.m1.1c">30n</annotation></semantics></math> to 480 pounds and solve for n.</span>
<span id="A1.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<span id="A1.Ex1"><span class="ltx_equation ltx_eqn_row ltx_align_baseline">
<span class="ltx_eqn_cell ltx_eqn_center_padleft"></span>
<span class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.Ex1.m1.1" class="ltx_Math" alttext="\displaystyle 30n" display="inline"><semantics id="A1.Ex1.m1.1a"><mrow id="A1.Ex1.m1.1.1" xref="A1.Ex1.m1.1.1.cmml"><mn id="A1.Ex1.m1.1.1.2" xref="A1.Ex1.m1.1.1.2.cmml">30</mn><mo lspace="0em" rspace="0em" id="A1.Ex1.m1.1.1.1" xref="A1.Ex1.m1.1.1.1.cmml">​</mo><mi id="A1.Ex1.m1.1.1.3" xref="A1.Ex1.m1.1.1.3.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex1.m1.1b"><apply id="A1.Ex1.m1.1.1.cmml" xref="A1.Ex1.m1.1.1"><times id="A1.Ex1.m1.1.1.1.cmml" xref="A1.Ex1.m1.1.1.1"></times><cn type="integer" id="A1.Ex1.m1.1.1.2.cmml" xref="A1.Ex1.m1.1.1.2">30</cn><ci id="A1.Ex1.m1.1.1.3.cmml" xref="A1.Ex1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex1.m1.1c">\displaystyle 30n</annotation></semantics></math></span>
<span class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.Ex1.m2.1" class="ltx_Math" alttext="\displaystyle=480" display="inline"><semantics id="A1.Ex1.m2.1a"><mrow id="A1.Ex1.m2.1.1" xref="A1.Ex1.m2.1.1.cmml"><mi id="A1.Ex1.m2.1.1.2" xref="A1.Ex1.m2.1.1.2.cmml"></mi><mo id="A1.Ex1.m2.1.1.1" xref="A1.Ex1.m2.1.1.1.cmml">=</mo><mn id="A1.Ex1.m2.1.1.3" xref="A1.Ex1.m2.1.1.3.cmml">480</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex1.m2.1b"><apply id="A1.Ex1.m2.1.1.cmml" xref="A1.Ex1.m2.1.1"><eq id="A1.Ex1.m2.1.1.1.cmml" xref="A1.Ex1.m2.1.1.1"></eq><csymbol cd="latexml" id="A1.Ex1.m2.1.1.2.cmml" xref="A1.Ex1.m2.1.1.2">absent</csymbol><cn type="integer" id="A1.Ex1.m2.1.1.3.cmml" xref="A1.Ex1.m2.1.1.3">480</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex1.m2.1c">\displaystyle=480</annotation></semantics></math></span>
<span class="ltx_eqn_cell ltx_eqn_center_padright"></span></span></span>
<span id="A1.Ex2"><span class="ltx_equation ltx_eqn_row ltx_align_baseline">
<span class="ltx_eqn_cell ltx_eqn_center_padleft"></span>
<span class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.Ex2.m1.2" class="ltx_Math" alttext="\displaystyle\Rightarrow\qquad n" display="inline"><semantics id="A1.Ex2.m1.2a"><mrow id="A1.Ex2.m1.2.3.2" xref="A1.Ex2.m1.2.3.1.cmml"><mo stretchy="false" id="A1.Ex2.m1.1.1" xref="A1.Ex2.m1.1.1.cmml">⇒</mo><mspace width="1.6em" id="A1.Ex2.m1.2.3.2.1" xref="A1.Ex2.m1.2.3.1.cmml"></mspace><mi id="A1.Ex2.m1.2.2" xref="A1.Ex2.m1.2.2.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex2.m1.2b"><list id="A1.Ex2.m1.2.3.1.cmml" xref="A1.Ex2.m1.2.3.2"><ci id="A1.Ex2.m1.1.1.cmml" xref="A1.Ex2.m1.1.1">⇒</ci><ci id="A1.Ex2.m1.2.2.cmml" xref="A1.Ex2.m1.2.2">𝑛</ci></list></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex2.m1.2c">\displaystyle\Rightarrow\qquad n</annotation></semantics></math></span>
<span class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.Ex2.m2.1" class="ltx_Math" alttext="\displaystyle=480/30=\boxed{16}" display="inline"><semantics id="A1.Ex2.m2.1a"><mrow id="A1.Ex2.m2.1.1" xref="A1.Ex2.m2.1.1.cmml"><mi id="A1.Ex2.m2.1.1.2" xref="A1.Ex2.m2.1.1.2.cmml"></mi><mo id="A1.Ex2.m2.1.1.3" xref="A1.Ex2.m2.1.1.3.cmml">=</mo><mrow id="A1.Ex2.m2.1.1.4" xref="A1.Ex2.m2.1.1.4.cmml"><mn id="A1.Ex2.m2.1.1.4.2" xref="A1.Ex2.m2.1.1.4.2.cmml">480</mn><mo id="A1.Ex2.m2.1.1.4.1" xref="A1.Ex2.m2.1.1.4.1.cmml">/</mo><mn id="A1.Ex2.m2.1.1.4.3" xref="A1.Ex2.m2.1.1.4.3.cmml">30</mn></mrow><mo id="A1.Ex2.m2.1.1.5" xref="A1.Ex2.m2.1.1.5.cmml">=</mo><menclose notation="box" id="A1.Ex2.m2.1.1.6" xref="A1.Ex2.m2.1.1.6.cmml"><mn id="A1.Ex2.m2.1.1.6a" xref="A1.Ex2.m2.1.1.6.cmml">16</mn></menclose></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex2.m2.1b"><apply id="A1.Ex2.m2.1.1.cmml" xref="A1.Ex2.m2.1.1"><and id="A1.Ex2.m2.1.1a.cmml" xref="A1.Ex2.m2.1.1"></and><apply id="A1.Ex2.m2.1.1b.cmml" xref="A1.Ex2.m2.1.1"><eq id="A1.Ex2.m2.1.1.3.cmml" xref="A1.Ex2.m2.1.1.3"></eq><csymbol cd="latexml" id="A1.Ex2.m2.1.1.2.cmml" xref="A1.Ex2.m2.1.1.2">absent</csymbol><apply id="A1.Ex2.m2.1.1.4.cmml" xref="A1.Ex2.m2.1.1.4"><divide id="A1.Ex2.m2.1.1.4.1.cmml" xref="A1.Ex2.m2.1.1.4.1"></divide><cn type="integer" id="A1.Ex2.m2.1.1.4.2.cmml" xref="A1.Ex2.m2.1.1.4.2">480</cn><cn type="integer" id="A1.Ex2.m2.1.1.4.3.cmml" xref="A1.Ex2.m2.1.1.4.3">30</cn></apply></apply><apply id="A1.Ex2.m2.1.1c.cmml" xref="A1.Ex2.m2.1.1"><eq id="A1.Ex2.m2.1.1.5.cmml" xref="A1.Ex2.m2.1.1.5"></eq><share href="#A1.Ex2.m2.1.1.4.cmml" id="A1.Ex2.m2.1.1d.cmml" xref="A1.Ex2.m2.1.1"></share><cn type="integer" id="A1.Ex2.m2.1.1.6.cmml" xref="A1.Ex2.m2.1.1.6">16</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex2.m2.1c">\displaystyle=480/30=\boxed{16}</annotation></semantics></math></span>
<span class="ltx_eqn_cell ltx_eqn_center_padright"></span></span></span>
</span>
<span id="A1.F23.20.20.20.20" class="ltx_p">Answer: <math id="A1.F23.20.20.20.20.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="A1.F23.20.20.20.20.m1.1a"><mn id="A1.F23.20.20.20.20.m1.1.1" xref="A1.F23.20.20.20.20.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="A1.F23.20.20.20.20.m1.1b"><cn type="integer" id="A1.F23.20.20.20.20.m1.1.1.cmml" xref="A1.F23.20.20.20.20.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.20.20.20.20.m1.1c">16</annotation></semantics></math>.</span>
<span id="A1.F23.21.21.21.21" class="ltx_p">Final Answer: The final answer is <math id="A1.F23.21.21.21.21.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="A1.F23.21.21.21.21.m1.1a"><mn id="A1.F23.21.21.21.21.m1.1.1" xref="A1.F23.21.21.21.21.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="A1.F23.21.21.21.21.m1.1b"><cn type="integer" id="A1.F23.21.21.21.21.m1.1.1.cmml" xref="A1.F23.21.21.21.21.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.21.21.21.21.m1.1c">16</annotation></semantics></math>. I hope it is correct.</span>
<span id="A1.F23.31.31.31.46" class="ltx_p">—</span>
<span id="A1.F23.31.31.31.47" class="ltx_p">Problem:
If the system of equations</span>
<span id="A1.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<span id="A1.Ex3"><span class="ltx_equation ltx_eqn_row ltx_align_baseline">
<span class="ltx_eqn_cell ltx_eqn_center_padleft"></span>
<span class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.Ex3.m1.1" class="ltx_Math" alttext="\displaystyle 6x-4y" display="inline"><semantics id="A1.Ex3.m1.1a"><mrow id="A1.Ex3.m1.1.1" xref="A1.Ex3.m1.1.1.cmml"><mrow id="A1.Ex3.m1.1.1.2" xref="A1.Ex3.m1.1.1.2.cmml"><mn id="A1.Ex3.m1.1.1.2.2" xref="A1.Ex3.m1.1.1.2.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="A1.Ex3.m1.1.1.2.1" xref="A1.Ex3.m1.1.1.2.1.cmml">​</mo><mi id="A1.Ex3.m1.1.1.2.3" xref="A1.Ex3.m1.1.1.2.3.cmml">x</mi></mrow><mo id="A1.Ex3.m1.1.1.1" xref="A1.Ex3.m1.1.1.1.cmml">−</mo><mrow id="A1.Ex3.m1.1.1.3" xref="A1.Ex3.m1.1.1.3.cmml"><mn id="A1.Ex3.m1.1.1.3.2" xref="A1.Ex3.m1.1.1.3.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="A1.Ex3.m1.1.1.3.1" xref="A1.Ex3.m1.1.1.3.1.cmml">​</mo><mi id="A1.Ex3.m1.1.1.3.3" xref="A1.Ex3.m1.1.1.3.3.cmml">y</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex3.m1.1b"><apply id="A1.Ex3.m1.1.1.cmml" xref="A1.Ex3.m1.1.1"><minus id="A1.Ex3.m1.1.1.1.cmml" xref="A1.Ex3.m1.1.1.1"></minus><apply id="A1.Ex3.m1.1.1.2.cmml" xref="A1.Ex3.m1.1.1.2"><times id="A1.Ex3.m1.1.1.2.1.cmml" xref="A1.Ex3.m1.1.1.2.1"></times><cn type="integer" id="A1.Ex3.m1.1.1.2.2.cmml" xref="A1.Ex3.m1.1.1.2.2">6</cn><ci id="A1.Ex3.m1.1.1.2.3.cmml" xref="A1.Ex3.m1.1.1.2.3">𝑥</ci></apply><apply id="A1.Ex3.m1.1.1.3.cmml" xref="A1.Ex3.m1.1.1.3"><times id="A1.Ex3.m1.1.1.3.1.cmml" xref="A1.Ex3.m1.1.1.3.1"></times><cn type="integer" id="A1.Ex3.m1.1.1.3.2.cmml" xref="A1.Ex3.m1.1.1.3.2">4</cn><ci id="A1.Ex3.m1.1.1.3.3.cmml" xref="A1.Ex3.m1.1.1.3.3">𝑦</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex3.m1.1c">\displaystyle 6x-4y</annotation></semantics></math></span>
<span class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.Ex3.m2.1" class="ltx_Math" alttext="\displaystyle=a," display="inline"><semantics id="A1.Ex3.m2.1a"><mrow id="A1.Ex3.m2.1.1.1" xref="A1.Ex3.m2.1.1.1.1.cmml"><mrow id="A1.Ex3.m2.1.1.1.1" xref="A1.Ex3.m2.1.1.1.1.cmml"><mi id="A1.Ex3.m2.1.1.1.1.2" xref="A1.Ex3.m2.1.1.1.1.2.cmml"></mi><mo id="A1.Ex3.m2.1.1.1.1.1" xref="A1.Ex3.m2.1.1.1.1.1.cmml">=</mo><mi id="A1.Ex3.m2.1.1.1.1.3" xref="A1.Ex3.m2.1.1.1.1.3.cmml">a</mi></mrow><mo id="A1.Ex3.m2.1.1.1.2" xref="A1.Ex3.m2.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex3.m2.1b"><apply id="A1.Ex3.m2.1.1.1.1.cmml" xref="A1.Ex3.m2.1.1.1"><eq id="A1.Ex3.m2.1.1.1.1.1.cmml" xref="A1.Ex3.m2.1.1.1.1.1"></eq><csymbol cd="latexml" id="A1.Ex3.m2.1.1.1.1.2.cmml" xref="A1.Ex3.m2.1.1.1.1.2">absent</csymbol><ci id="A1.Ex3.m2.1.1.1.1.3.cmml" xref="A1.Ex3.m2.1.1.1.1.3">𝑎</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex3.m2.1c">\displaystyle=a,</annotation></semantics></math></span>
<span class="ltx_eqn_cell ltx_eqn_center_padright"></span></span></span>
<span id="A1.Ex4"><span class="ltx_equation ltx_eqn_row ltx_align_baseline">
<span class="ltx_eqn_cell ltx_eqn_center_padleft"></span>
<span class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.Ex4.m1.1" class="ltx_Math" alttext="\displaystyle 6y-9x" display="inline"><semantics id="A1.Ex4.m1.1a"><mrow id="A1.Ex4.m1.1.1" xref="A1.Ex4.m1.1.1.cmml"><mrow id="A1.Ex4.m1.1.1.2" xref="A1.Ex4.m1.1.1.2.cmml"><mn id="A1.Ex4.m1.1.1.2.2" xref="A1.Ex4.m1.1.1.2.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.1.1.2.1" xref="A1.Ex4.m1.1.1.2.1.cmml">​</mo><mi id="A1.Ex4.m1.1.1.2.3" xref="A1.Ex4.m1.1.1.2.3.cmml">y</mi></mrow><mo id="A1.Ex4.m1.1.1.1" xref="A1.Ex4.m1.1.1.1.cmml">−</mo><mrow id="A1.Ex4.m1.1.1.3" xref="A1.Ex4.m1.1.1.3.cmml"><mn id="A1.Ex4.m1.1.1.3.2" xref="A1.Ex4.m1.1.1.3.2.cmml">9</mn><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.1.1.3.1" xref="A1.Ex4.m1.1.1.3.1.cmml">​</mo><mi id="A1.Ex4.m1.1.1.3.3" xref="A1.Ex4.m1.1.1.3.3.cmml">x</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex4.m1.1b"><apply id="A1.Ex4.m1.1.1.cmml" xref="A1.Ex4.m1.1.1"><minus id="A1.Ex4.m1.1.1.1.cmml" xref="A1.Ex4.m1.1.1.1"></minus><apply id="A1.Ex4.m1.1.1.2.cmml" xref="A1.Ex4.m1.1.1.2"><times id="A1.Ex4.m1.1.1.2.1.cmml" xref="A1.Ex4.m1.1.1.2.1"></times><cn type="integer" id="A1.Ex4.m1.1.1.2.2.cmml" xref="A1.Ex4.m1.1.1.2.2">6</cn><ci id="A1.Ex4.m1.1.1.2.3.cmml" xref="A1.Ex4.m1.1.1.2.3">𝑦</ci></apply><apply id="A1.Ex4.m1.1.1.3.cmml" xref="A1.Ex4.m1.1.1.3"><times id="A1.Ex4.m1.1.1.3.1.cmml" xref="A1.Ex4.m1.1.1.3.1"></times><cn type="integer" id="A1.Ex4.m1.1.1.3.2.cmml" xref="A1.Ex4.m1.1.1.3.2">9</cn><ci id="A1.Ex4.m1.1.1.3.3.cmml" xref="A1.Ex4.m1.1.1.3.3">𝑥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex4.m1.1c">\displaystyle 6y-9x</annotation></semantics></math></span>
<span class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.Ex4.m2.1" class="ltx_Math" alttext="\displaystyle=b." display="inline"><semantics id="A1.Ex4.m2.1a"><mrow id="A1.Ex4.m2.1.1.1" xref="A1.Ex4.m2.1.1.1.1.cmml"><mrow id="A1.Ex4.m2.1.1.1.1" xref="A1.Ex4.m2.1.1.1.1.cmml"><mi id="A1.Ex4.m2.1.1.1.1.2" xref="A1.Ex4.m2.1.1.1.1.2.cmml"></mi><mo id="A1.Ex4.m2.1.1.1.1.1" xref="A1.Ex4.m2.1.1.1.1.1.cmml">=</mo><mi id="A1.Ex4.m2.1.1.1.1.3" xref="A1.Ex4.m2.1.1.1.1.3.cmml">b</mi></mrow><mo lspace="0em" id="A1.Ex4.m2.1.1.1.2" xref="A1.Ex4.m2.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex4.m2.1b"><apply id="A1.Ex4.m2.1.1.1.1.cmml" xref="A1.Ex4.m2.1.1.1"><eq id="A1.Ex4.m2.1.1.1.1.1.cmml" xref="A1.Ex4.m2.1.1.1.1.1"></eq><csymbol cd="latexml" id="A1.Ex4.m2.1.1.1.1.2.cmml" xref="A1.Ex4.m2.1.1.1.1.2">absent</csymbol><ci id="A1.Ex4.m2.1.1.1.1.3.cmml" xref="A1.Ex4.m2.1.1.1.1.3">𝑏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex4.m2.1c">\displaystyle=b.</annotation></semantics></math></span>
<span class="ltx_eqn_cell ltx_eqn_center_padright"></span></span></span>
</span>
<span id="A1.F23.26.26.26.26" class="ltx_p">has a solution <math id="A1.F23.22.22.22.22.m1.2" class="ltx_Math" alttext="(x,y)" display="inline"><semantics id="A1.F23.22.22.22.22.m1.2a"><mrow id="A1.F23.22.22.22.22.m1.2.3.2" xref="A1.F23.22.22.22.22.m1.2.3.1.cmml"><mo stretchy="false" id="A1.F23.22.22.22.22.m1.2.3.2.1" xref="A1.F23.22.22.22.22.m1.2.3.1.cmml">(</mo><mi id="A1.F23.22.22.22.22.m1.1.1" xref="A1.F23.22.22.22.22.m1.1.1.cmml">x</mi><mo id="A1.F23.22.22.22.22.m1.2.3.2.2" xref="A1.F23.22.22.22.22.m1.2.3.1.cmml">,</mo><mi id="A1.F23.22.22.22.22.m1.2.2" xref="A1.F23.22.22.22.22.m1.2.2.cmml">y</mi><mo stretchy="false" id="A1.F23.22.22.22.22.m1.2.3.2.3" xref="A1.F23.22.22.22.22.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.22.22.22.22.m1.2b"><interval closure="open" id="A1.F23.22.22.22.22.m1.2.3.1.cmml" xref="A1.F23.22.22.22.22.m1.2.3.2"><ci id="A1.F23.22.22.22.22.m1.1.1.cmml" xref="A1.F23.22.22.22.22.m1.1.1">𝑥</ci><ci id="A1.F23.22.22.22.22.m1.2.2.cmml" xref="A1.F23.22.22.22.22.m1.2.2">𝑦</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.22.22.22.22.m1.2c">(x,y)</annotation></semantics></math> where <math id="A1.F23.23.23.23.23.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A1.F23.23.23.23.23.m2.1a"><mi id="A1.F23.23.23.23.23.m2.1.1" xref="A1.F23.23.23.23.23.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A1.F23.23.23.23.23.m2.1b"><ci id="A1.F23.23.23.23.23.m2.1.1.cmml" xref="A1.F23.23.23.23.23.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.23.23.23.23.m2.1c">x</annotation></semantics></math> and <math id="A1.F23.24.24.24.24.m3.1" class="ltx_Math" alttext="y" display="inline"><semantics id="A1.F23.24.24.24.24.m3.1a"><mi id="A1.F23.24.24.24.24.m3.1.1" xref="A1.F23.24.24.24.24.m3.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="A1.F23.24.24.24.24.m3.1b"><ci id="A1.F23.24.24.24.24.m3.1.1.cmml" xref="A1.F23.24.24.24.24.m3.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.24.24.24.24.m3.1c">y</annotation></semantics></math> are both nonzero, find <math id="A1.F23.25.25.25.25.m4.1" class="ltx_Math" alttext="\frac{a}{b}," display="inline"><semantics id="A1.F23.25.25.25.25.m4.1a"><mrow id="A1.F23.25.25.25.25.m4.1.2.2" xref="A1.F23.25.25.25.25.m4.1.1.cmml"><mfrac id="A1.F23.25.25.25.25.m4.1.1" xref="A1.F23.25.25.25.25.m4.1.1.cmml"><mi id="A1.F23.25.25.25.25.m4.1.1.2" xref="A1.F23.25.25.25.25.m4.1.1.2.cmml">a</mi><mi id="A1.F23.25.25.25.25.m4.1.1.3" xref="A1.F23.25.25.25.25.m4.1.1.3.cmml">b</mi></mfrac><mo id="A1.F23.25.25.25.25.m4.1.2.2.1" xref="A1.F23.25.25.25.25.m4.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.25.25.25.25.m4.1b"><apply id="A1.F23.25.25.25.25.m4.1.1.cmml" xref="A1.F23.25.25.25.25.m4.1.2.2"><divide id="A1.F23.25.25.25.25.m4.1.1.1.cmml" xref="A1.F23.25.25.25.25.m4.1.2.2"></divide><ci id="A1.F23.25.25.25.25.m4.1.1.2.cmml" xref="A1.F23.25.25.25.25.m4.1.1.2">𝑎</ci><ci id="A1.F23.25.25.25.25.m4.1.1.3.cmml" xref="A1.F23.25.25.25.25.m4.1.1.3">𝑏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.25.25.25.25.m4.1c">\frac{a}{b},</annotation></semantics></math> assuming <math id="A1.F23.26.26.26.26.m5.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A1.F23.26.26.26.26.m5.1a"><mi id="A1.F23.26.26.26.26.m5.1.1" xref="A1.F23.26.26.26.26.m5.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A1.F23.26.26.26.26.m5.1b"><ci id="A1.F23.26.26.26.26.m5.1.1.cmml" xref="A1.F23.26.26.26.26.m5.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.26.26.26.26.m5.1c">b</annotation></semantics></math> is nonzero.</span>
<span id="A1.F23.31.31.31.48" class="ltx_p">Solution:</span>
<span id="A1.F23.31.31.31.49" class="ltx_p">I’m given a system of two equations.</span>
<span id="A1.F23.28.28.28.28" class="ltx_p">I see that if I multiply the first equation by <math id="A1.F23.27.27.27.27.m1.1" class="ltx_Math" alttext="-\frac{3}{2}" display="inline"><semantics id="A1.F23.27.27.27.27.m1.1a"><mrow id="A1.F23.27.27.27.27.m1.1.1" xref="A1.F23.27.27.27.27.m1.1.1.cmml"><mo id="A1.F23.27.27.27.27.m1.1.1a" xref="A1.F23.27.27.27.27.m1.1.1.cmml">−</mo><mfrac id="A1.F23.27.27.27.27.m1.1.1.2" xref="A1.F23.27.27.27.27.m1.1.1.2.cmml"><mn id="A1.F23.27.27.27.27.m1.1.1.2.2" xref="A1.F23.27.27.27.27.m1.1.1.2.2.cmml">3</mn><mn id="A1.F23.27.27.27.27.m1.1.1.2.3" xref="A1.F23.27.27.27.27.m1.1.1.2.3.cmml">2</mn></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.27.27.27.27.m1.1b"><apply id="A1.F23.27.27.27.27.m1.1.1.cmml" xref="A1.F23.27.27.27.27.m1.1.1"><minus id="A1.F23.27.27.27.27.m1.1.1.1.cmml" xref="A1.F23.27.27.27.27.m1.1.1"></minus><apply id="A1.F23.27.27.27.27.m1.1.1.2.cmml" xref="A1.F23.27.27.27.27.m1.1.1.2"><divide id="A1.F23.27.27.27.27.m1.1.1.2.1.cmml" xref="A1.F23.27.27.27.27.m1.1.1.2"></divide><cn type="integer" id="A1.F23.27.27.27.27.m1.1.1.2.2.cmml" xref="A1.F23.27.27.27.27.m1.1.1.2.2">3</cn><cn type="integer" id="A1.F23.27.27.27.27.m1.1.1.2.3.cmml" xref="A1.F23.27.27.27.27.m1.1.1.2.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.27.27.27.27.m1.1c">-\frac{3}{2}</annotation></semantics></math>, I’ll get another equation that has the same left-hand side as the second equation, <math id="A1.F23.28.28.28.28.m2.1" class="ltx_Math" alttext="6y-9x" display="inline"><semantics id="A1.F23.28.28.28.28.m2.1a"><mrow id="A1.F23.28.28.28.28.m2.1.1" xref="A1.F23.28.28.28.28.m2.1.1.cmml"><mrow id="A1.F23.28.28.28.28.m2.1.1.2" xref="A1.F23.28.28.28.28.m2.1.1.2.cmml"><mn id="A1.F23.28.28.28.28.m2.1.1.2.2" xref="A1.F23.28.28.28.28.m2.1.1.2.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="A1.F23.28.28.28.28.m2.1.1.2.1" xref="A1.F23.28.28.28.28.m2.1.1.2.1.cmml">​</mo><mi id="A1.F23.28.28.28.28.m2.1.1.2.3" xref="A1.F23.28.28.28.28.m2.1.1.2.3.cmml">y</mi></mrow><mo id="A1.F23.28.28.28.28.m2.1.1.1" xref="A1.F23.28.28.28.28.m2.1.1.1.cmml">−</mo><mrow id="A1.F23.28.28.28.28.m2.1.1.3" xref="A1.F23.28.28.28.28.m2.1.1.3.cmml"><mn id="A1.F23.28.28.28.28.m2.1.1.3.2" xref="A1.F23.28.28.28.28.m2.1.1.3.2.cmml">9</mn><mo lspace="0em" rspace="0em" id="A1.F23.28.28.28.28.m2.1.1.3.1" xref="A1.F23.28.28.28.28.m2.1.1.3.1.cmml">​</mo><mi id="A1.F23.28.28.28.28.m2.1.1.3.3" xref="A1.F23.28.28.28.28.m2.1.1.3.3.cmml">x</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.28.28.28.28.m2.1b"><apply id="A1.F23.28.28.28.28.m2.1.1.cmml" xref="A1.F23.28.28.28.28.m2.1.1"><minus id="A1.F23.28.28.28.28.m2.1.1.1.cmml" xref="A1.F23.28.28.28.28.m2.1.1.1"></minus><apply id="A1.F23.28.28.28.28.m2.1.1.2.cmml" xref="A1.F23.28.28.28.28.m2.1.1.2"><times id="A1.F23.28.28.28.28.m2.1.1.2.1.cmml" xref="A1.F23.28.28.28.28.m2.1.1.2.1"></times><cn type="integer" id="A1.F23.28.28.28.28.m2.1.1.2.2.cmml" xref="A1.F23.28.28.28.28.m2.1.1.2.2">6</cn><ci id="A1.F23.28.28.28.28.m2.1.1.2.3.cmml" xref="A1.F23.28.28.28.28.m2.1.1.2.3">𝑦</ci></apply><apply id="A1.F23.28.28.28.28.m2.1.1.3.cmml" xref="A1.F23.28.28.28.28.m2.1.1.3"><times id="A1.F23.28.28.28.28.m2.1.1.3.1.cmml" xref="A1.F23.28.28.28.28.m2.1.1.3.1"></times><cn type="integer" id="A1.F23.28.28.28.28.m2.1.1.3.2.cmml" xref="A1.F23.28.28.28.28.m2.1.1.3.2">9</cn><ci id="A1.F23.28.28.28.28.m2.1.1.3.3.cmml" xref="A1.F23.28.28.28.28.m2.1.1.3.3">𝑥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.28.28.28.28.m2.1c">6y-9x</annotation></semantics></math>.</span>
<span id="A1.F23.31.31.31.50" class="ltx_p">Let me try that</span>
<span id="A1.Ex5" class="ltx_equation ltx_eqn_table">

<span><span class="ltx_equation ltx_eqn_row ltx_align_baseline">
<span class="ltx_eqn_cell ltx_eqn_center_padleft"></span>
<span class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex5.m1.1" class="ltx_Math" alttext="6y-9x=-\frac{3}{2}a." display="block"><semantics id="A1.Ex5.m1.1a"><mrow id="A1.Ex5.m1.1.1.1" xref="A1.Ex5.m1.1.1.1.1.cmml"><mrow id="A1.Ex5.m1.1.1.1.1" xref="A1.Ex5.m1.1.1.1.1.cmml"><mrow id="A1.Ex5.m1.1.1.1.1.2" xref="A1.Ex5.m1.1.1.1.1.2.cmml"><mrow id="A1.Ex5.m1.1.1.1.1.2.2" xref="A1.Ex5.m1.1.1.1.1.2.2.cmml"><mn id="A1.Ex5.m1.1.1.1.1.2.2.2" xref="A1.Ex5.m1.1.1.1.1.2.2.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="A1.Ex5.m1.1.1.1.1.2.2.1" xref="A1.Ex5.m1.1.1.1.1.2.2.1.cmml">​</mo><mi id="A1.Ex5.m1.1.1.1.1.2.2.3" xref="A1.Ex5.m1.1.1.1.1.2.2.3.cmml">y</mi></mrow><mo id="A1.Ex5.m1.1.1.1.1.2.1" xref="A1.Ex5.m1.1.1.1.1.2.1.cmml">−</mo><mrow id="A1.Ex5.m1.1.1.1.1.2.3" xref="A1.Ex5.m1.1.1.1.1.2.3.cmml"><mn id="A1.Ex5.m1.1.1.1.1.2.3.2" xref="A1.Ex5.m1.1.1.1.1.2.3.2.cmml">9</mn><mo lspace="0em" rspace="0em" id="A1.Ex5.m1.1.1.1.1.2.3.1" xref="A1.Ex5.m1.1.1.1.1.2.3.1.cmml">​</mo><mi id="A1.Ex5.m1.1.1.1.1.2.3.3" xref="A1.Ex5.m1.1.1.1.1.2.3.3.cmml">x</mi></mrow></mrow><mo id="A1.Ex5.m1.1.1.1.1.1" xref="A1.Ex5.m1.1.1.1.1.1.cmml">=</mo><mrow id="A1.Ex5.m1.1.1.1.1.3" xref="A1.Ex5.m1.1.1.1.1.3.cmml"><mo id="A1.Ex5.m1.1.1.1.1.3a" xref="A1.Ex5.m1.1.1.1.1.3.cmml">−</mo><mrow id="A1.Ex5.m1.1.1.1.1.3.2" xref="A1.Ex5.m1.1.1.1.1.3.2.cmml"><mfrac id="A1.Ex5.m1.1.1.1.1.3.2.2" xref="A1.Ex5.m1.1.1.1.1.3.2.2.cmml"><mn mathsize="80%" id="A1.Ex5.m1.1.1.1.1.3.2.2.2" xref="A1.Ex5.m1.1.1.1.1.3.2.2.2.cmml">3</mn><mn mathsize="80%" id="A1.Ex5.m1.1.1.1.1.3.2.2.3" xref="A1.Ex5.m1.1.1.1.1.3.2.2.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="A1.Ex5.m1.1.1.1.1.3.2.1" xref="A1.Ex5.m1.1.1.1.1.3.2.1.cmml">​</mo><mi id="A1.Ex5.m1.1.1.1.1.3.2.3" xref="A1.Ex5.m1.1.1.1.1.3.2.3.cmml">a</mi></mrow></mrow></mrow><mo lspace="0em" id="A1.Ex5.m1.1.1.1.2" xref="A1.Ex5.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex5.m1.1b"><apply id="A1.Ex5.m1.1.1.1.1.cmml" xref="A1.Ex5.m1.1.1.1"><eq id="A1.Ex5.m1.1.1.1.1.1.cmml" xref="A1.Ex5.m1.1.1.1.1.1"></eq><apply id="A1.Ex5.m1.1.1.1.1.2.cmml" xref="A1.Ex5.m1.1.1.1.1.2"><minus id="A1.Ex5.m1.1.1.1.1.2.1.cmml" xref="A1.Ex5.m1.1.1.1.1.2.1"></minus><apply id="A1.Ex5.m1.1.1.1.1.2.2.cmml" xref="A1.Ex5.m1.1.1.1.1.2.2"><times id="A1.Ex5.m1.1.1.1.1.2.2.1.cmml" xref="A1.Ex5.m1.1.1.1.1.2.2.1"></times><cn type="integer" id="A1.Ex5.m1.1.1.1.1.2.2.2.cmml" xref="A1.Ex5.m1.1.1.1.1.2.2.2">6</cn><ci id="A1.Ex5.m1.1.1.1.1.2.2.3.cmml" xref="A1.Ex5.m1.1.1.1.1.2.2.3">𝑦</ci></apply><apply id="A1.Ex5.m1.1.1.1.1.2.3.cmml" xref="A1.Ex5.m1.1.1.1.1.2.3"><times id="A1.Ex5.m1.1.1.1.1.2.3.1.cmml" xref="A1.Ex5.m1.1.1.1.1.2.3.1"></times><cn type="integer" id="A1.Ex5.m1.1.1.1.1.2.3.2.cmml" xref="A1.Ex5.m1.1.1.1.1.2.3.2">9</cn><ci id="A1.Ex5.m1.1.1.1.1.2.3.3.cmml" xref="A1.Ex5.m1.1.1.1.1.2.3.3">𝑥</ci></apply></apply><apply id="A1.Ex5.m1.1.1.1.1.3.cmml" xref="A1.Ex5.m1.1.1.1.1.3"><minus id="A1.Ex5.m1.1.1.1.1.3.1.cmml" xref="A1.Ex5.m1.1.1.1.1.3"></minus><apply id="A1.Ex5.m1.1.1.1.1.3.2.cmml" xref="A1.Ex5.m1.1.1.1.1.3.2"><times id="A1.Ex5.m1.1.1.1.1.3.2.1.cmml" xref="A1.Ex5.m1.1.1.1.1.3.2.1"></times><apply id="A1.Ex5.m1.1.1.1.1.3.2.2.cmml" xref="A1.Ex5.m1.1.1.1.1.3.2.2"><divide id="A1.Ex5.m1.1.1.1.1.3.2.2.1.cmml" xref="A1.Ex5.m1.1.1.1.1.3.2.2"></divide><cn type="integer" id="A1.Ex5.m1.1.1.1.1.3.2.2.2.cmml" xref="A1.Ex5.m1.1.1.1.1.3.2.2.2">3</cn><cn type="integer" id="A1.Ex5.m1.1.1.1.1.3.2.2.3.cmml" xref="A1.Ex5.m1.1.1.1.1.3.2.2.3">2</cn></apply><ci id="A1.Ex5.m1.1.1.1.1.3.2.3.cmml" xref="A1.Ex5.m1.1.1.1.1.3.2.3">𝑎</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex5.m1.1c">6y-9x=-\frac{3}{2}a.</annotation></semantics></math></span>
<span class="ltx_eqn_cell ltx_eqn_center_padright"></span></span></span>
</span>
<span id="A1.F23.29.29.29.29" class="ltx_p">Ah, I also know that <math id="A1.F23.29.29.29.29.m1.1" class="ltx_Math" alttext="6y-9x=b" display="inline"><semantics id="A1.F23.29.29.29.29.m1.1a"><mrow id="A1.F23.29.29.29.29.m1.1.1" xref="A1.F23.29.29.29.29.m1.1.1.cmml"><mrow id="A1.F23.29.29.29.29.m1.1.1.2" xref="A1.F23.29.29.29.29.m1.1.1.2.cmml"><mrow id="A1.F23.29.29.29.29.m1.1.1.2.2" xref="A1.F23.29.29.29.29.m1.1.1.2.2.cmml"><mn id="A1.F23.29.29.29.29.m1.1.1.2.2.2" xref="A1.F23.29.29.29.29.m1.1.1.2.2.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="A1.F23.29.29.29.29.m1.1.1.2.2.1" xref="A1.F23.29.29.29.29.m1.1.1.2.2.1.cmml">​</mo><mi id="A1.F23.29.29.29.29.m1.1.1.2.2.3" xref="A1.F23.29.29.29.29.m1.1.1.2.2.3.cmml">y</mi></mrow><mo id="A1.F23.29.29.29.29.m1.1.1.2.1" xref="A1.F23.29.29.29.29.m1.1.1.2.1.cmml">−</mo><mrow id="A1.F23.29.29.29.29.m1.1.1.2.3" xref="A1.F23.29.29.29.29.m1.1.1.2.3.cmml"><mn id="A1.F23.29.29.29.29.m1.1.1.2.3.2" xref="A1.F23.29.29.29.29.m1.1.1.2.3.2.cmml">9</mn><mo lspace="0em" rspace="0em" id="A1.F23.29.29.29.29.m1.1.1.2.3.1" xref="A1.F23.29.29.29.29.m1.1.1.2.3.1.cmml">​</mo><mi id="A1.F23.29.29.29.29.m1.1.1.2.3.3" xref="A1.F23.29.29.29.29.m1.1.1.2.3.3.cmml">x</mi></mrow></mrow><mo id="A1.F23.29.29.29.29.m1.1.1.1" xref="A1.F23.29.29.29.29.m1.1.1.1.cmml">=</mo><mi id="A1.F23.29.29.29.29.m1.1.1.3" xref="A1.F23.29.29.29.29.m1.1.1.3.cmml">b</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.29.29.29.29.m1.1b"><apply id="A1.F23.29.29.29.29.m1.1.1.cmml" xref="A1.F23.29.29.29.29.m1.1.1"><eq id="A1.F23.29.29.29.29.m1.1.1.1.cmml" xref="A1.F23.29.29.29.29.m1.1.1.1"></eq><apply id="A1.F23.29.29.29.29.m1.1.1.2.cmml" xref="A1.F23.29.29.29.29.m1.1.1.2"><minus id="A1.F23.29.29.29.29.m1.1.1.2.1.cmml" xref="A1.F23.29.29.29.29.m1.1.1.2.1"></minus><apply id="A1.F23.29.29.29.29.m1.1.1.2.2.cmml" xref="A1.F23.29.29.29.29.m1.1.1.2.2"><times id="A1.F23.29.29.29.29.m1.1.1.2.2.1.cmml" xref="A1.F23.29.29.29.29.m1.1.1.2.2.1"></times><cn type="integer" id="A1.F23.29.29.29.29.m1.1.1.2.2.2.cmml" xref="A1.F23.29.29.29.29.m1.1.1.2.2.2">6</cn><ci id="A1.F23.29.29.29.29.m1.1.1.2.2.3.cmml" xref="A1.F23.29.29.29.29.m1.1.1.2.2.3">𝑦</ci></apply><apply id="A1.F23.29.29.29.29.m1.1.1.2.3.cmml" xref="A1.F23.29.29.29.29.m1.1.1.2.3"><times id="A1.F23.29.29.29.29.m1.1.1.2.3.1.cmml" xref="A1.F23.29.29.29.29.m1.1.1.2.3.1"></times><cn type="integer" id="A1.F23.29.29.29.29.m1.1.1.2.3.2.cmml" xref="A1.F23.29.29.29.29.m1.1.1.2.3.2">9</cn><ci id="A1.F23.29.29.29.29.m1.1.1.2.3.3.cmml" xref="A1.F23.29.29.29.29.m1.1.1.2.3.3">𝑥</ci></apply></apply><ci id="A1.F23.29.29.29.29.m1.1.1.3.cmml" xref="A1.F23.29.29.29.29.m1.1.1.3">𝑏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.29.29.29.29.m1.1c">6y-9x=b</annotation></semantics></math>, so I can equate these two equations.</span>
<span id="A1.F23.31.31.31.51" class="ltx_p">So,</span>
<span id="A1.Ex6" class="ltx_equation ltx_eqn_table">

<span><span class="ltx_equation ltx_eqn_row ltx_align_baseline">
<span class="ltx_eqn_cell ltx_eqn_center_padleft"></span>
<span class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex6.m1.1" class="ltx_Math" alttext="-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=\boxed{-\frac{2}{3}}." display="block"><semantics id="A1.Ex6.m1.1a"><mrow id="A1.Ex6.m1.1.1.1" xref="A1.Ex6.m1.1.1.1.1.cmml"><mrow id="A1.Ex6.m1.1.1.1.1" xref="A1.Ex6.m1.1.1.1.1.cmml"><mrow id="A1.Ex6.m1.1.1.1.1.2" xref="A1.Ex6.m1.1.1.1.1.2.cmml"><mo id="A1.Ex6.m1.1.1.1.1.2a" xref="A1.Ex6.m1.1.1.1.1.2.cmml">−</mo><mrow id="A1.Ex6.m1.1.1.1.1.2.2" xref="A1.Ex6.m1.1.1.1.1.2.2.cmml"><mfrac id="A1.Ex6.m1.1.1.1.1.2.2.2" xref="A1.Ex6.m1.1.1.1.1.2.2.2.cmml"><mn mathsize="80%" id="A1.Ex6.m1.1.1.1.1.2.2.2.2" xref="A1.Ex6.m1.1.1.1.1.2.2.2.2.cmml">3</mn><mn mathsize="80%" id="A1.Ex6.m1.1.1.1.1.2.2.2.3" xref="A1.Ex6.m1.1.1.1.1.2.2.2.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="A1.Ex6.m1.1.1.1.1.2.2.1" xref="A1.Ex6.m1.1.1.1.1.2.2.1.cmml">​</mo><mi id="A1.Ex6.m1.1.1.1.1.2.2.3" xref="A1.Ex6.m1.1.1.1.1.2.2.3.cmml">a</mi></mrow></mrow><mo id="A1.Ex6.m1.1.1.1.1.3" xref="A1.Ex6.m1.1.1.1.1.3.cmml">=</mo><mi id="A1.Ex6.m1.1.1.1.1.4" xref="A1.Ex6.m1.1.1.1.1.4.cmml">b</mi><mo stretchy="false" id="A1.Ex6.m1.1.1.1.1.5" xref="A1.Ex6.m1.1.1.1.1.5.cmml">⇒</mo><mfrac id="A1.Ex6.m1.1.1.1.1.6" xref="A1.Ex6.m1.1.1.1.1.6.cmml"><mi mathsize="80%" id="A1.Ex6.m1.1.1.1.1.6.2" xref="A1.Ex6.m1.1.1.1.1.6.2.cmml">a</mi><mi mathsize="80%" id="A1.Ex6.m1.1.1.1.1.6.3" xref="A1.Ex6.m1.1.1.1.1.6.3.cmml">b</mi></mfrac><mo id="A1.Ex6.m1.1.1.1.1.7" xref="A1.Ex6.m1.1.1.1.1.7.cmml">=</mo><menclose notation="box" id="A1.Ex6.m1.1.1.1.1.8" xref="A1.Ex6.m1.1.1.1.1.8.cmml"><mrow id="A1.Ex6.m1.1.1.1.1.8a" xref="A1.Ex6.m1.1.1.1.1.8.cmml"><mo id="A1.Ex6.m1.1.1.1.1.8b" xref="A1.Ex6.m1.1.1.1.1.8.cmml">−</mo><mfrac id="A1.Ex6.m1.1.1.1.1.8.2" xref="A1.Ex6.m1.1.1.1.1.8.2.cmml"><mn mathsize="80%" id="A1.Ex6.m1.1.1.1.1.8.2.2" xref="A1.Ex6.m1.1.1.1.1.8.2.2.cmml">2</mn><mn mathsize="80%" id="A1.Ex6.m1.1.1.1.1.8.2.3" xref="A1.Ex6.m1.1.1.1.1.8.2.3.cmml">3</mn></mfrac></mrow></menclose></mrow><mo lspace="0em" id="A1.Ex6.m1.1.1.1.2" xref="A1.Ex6.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex6.m1.1b"><apply id="A1.Ex6.m1.1.1.1.1.cmml" xref="A1.Ex6.m1.1.1.1"><and id="A1.Ex6.m1.1.1.1.1a.cmml" xref="A1.Ex6.m1.1.1.1"></and><apply id="A1.Ex6.m1.1.1.1.1b.cmml" xref="A1.Ex6.m1.1.1.1"><eq id="A1.Ex6.m1.1.1.1.1.3.cmml" xref="A1.Ex6.m1.1.1.1.1.3"></eq><apply id="A1.Ex6.m1.1.1.1.1.2.cmml" xref="A1.Ex6.m1.1.1.1.1.2"><minus id="A1.Ex6.m1.1.1.1.1.2.1.cmml" xref="A1.Ex6.m1.1.1.1.1.2"></minus><apply id="A1.Ex6.m1.1.1.1.1.2.2.cmml" xref="A1.Ex6.m1.1.1.1.1.2.2"><times id="A1.Ex6.m1.1.1.1.1.2.2.1.cmml" xref="A1.Ex6.m1.1.1.1.1.2.2.1"></times><apply id="A1.Ex6.m1.1.1.1.1.2.2.2.cmml" xref="A1.Ex6.m1.1.1.1.1.2.2.2"><divide id="A1.Ex6.m1.1.1.1.1.2.2.2.1.cmml" xref="A1.Ex6.m1.1.1.1.1.2.2.2"></divide><cn type="integer" id="A1.Ex6.m1.1.1.1.1.2.2.2.2.cmml" xref="A1.Ex6.m1.1.1.1.1.2.2.2.2">3</cn><cn type="integer" id="A1.Ex6.m1.1.1.1.1.2.2.2.3.cmml" xref="A1.Ex6.m1.1.1.1.1.2.2.2.3">2</cn></apply><ci id="A1.Ex6.m1.1.1.1.1.2.2.3.cmml" xref="A1.Ex6.m1.1.1.1.1.2.2.3">𝑎</ci></apply></apply><ci id="A1.Ex6.m1.1.1.1.1.4.cmml" xref="A1.Ex6.m1.1.1.1.1.4">𝑏</ci></apply><apply id="A1.Ex6.m1.1.1.1.1c.cmml" xref="A1.Ex6.m1.1.1.1"><ci id="A1.Ex6.m1.1.1.1.1.5.cmml" xref="A1.Ex6.m1.1.1.1.1.5">⇒</ci><share href="#A1.Ex6.m1.1.1.1.1.4.cmml" id="A1.Ex6.m1.1.1.1.1d.cmml" xref="A1.Ex6.m1.1.1.1"></share><apply id="A1.Ex6.m1.1.1.1.1.6.cmml" xref="A1.Ex6.m1.1.1.1.1.6"><divide id="A1.Ex6.m1.1.1.1.1.6.1.cmml" xref="A1.Ex6.m1.1.1.1.1.6"></divide><ci id="A1.Ex6.m1.1.1.1.1.6.2.cmml" xref="A1.Ex6.m1.1.1.1.1.6.2">𝑎</ci><ci id="A1.Ex6.m1.1.1.1.1.6.3.cmml" xref="A1.Ex6.m1.1.1.1.1.6.3">𝑏</ci></apply></apply><apply id="A1.Ex6.m1.1.1.1.1e.cmml" xref="A1.Ex6.m1.1.1.1"><eq id="A1.Ex6.m1.1.1.1.1.7.cmml" xref="A1.Ex6.m1.1.1.1.1.7"></eq><share href="#A1.Ex6.m1.1.1.1.1.6.cmml" id="A1.Ex6.m1.1.1.1.1f.cmml" xref="A1.Ex6.m1.1.1.1"></share><apply id="A1.Ex6.m1.1.1.1.1.8.cmml" xref="A1.Ex6.m1.1.1.1.1.8"><minus id="A1.Ex6.m1.1.1.1.1.8.1.cmml" xref="A1.Ex6.m1.1.1.1.1.8"></minus><apply id="A1.Ex6.m1.1.1.1.1.8.2.cmml" xref="A1.Ex6.m1.1.1.1.1.8.2"><divide id="A1.Ex6.m1.1.1.1.1.8.2.1.cmml" xref="A1.Ex6.m1.1.1.1.1.8.2"></divide><cn type="integer" id="A1.Ex6.m1.1.1.1.1.8.2.2.cmml" xref="A1.Ex6.m1.1.1.1.1.8.2.2">2</cn><cn type="integer" id="A1.Ex6.m1.1.1.1.1.8.2.3.cmml" xref="A1.Ex6.m1.1.1.1.1.8.2.3">3</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex6.m1.1c">-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=\boxed{-\frac{2}{3}}.</annotation></semantics></math></span>
<span class="ltx_eqn_cell ltx_eqn_center_padright"></span></span></span>
</span>
<span id="A1.F23.30.30.30.30" class="ltx_p">Answer: <math id="A1.F23.30.30.30.30.m1.1" class="ltx_Math" alttext="-\frac{2}{3}" display="inline"><semantics id="A1.F23.30.30.30.30.m1.1a"><mrow id="A1.F23.30.30.30.30.m1.1.1" xref="A1.F23.30.30.30.30.m1.1.1.cmml"><mo id="A1.F23.30.30.30.30.m1.1.1a" xref="A1.F23.30.30.30.30.m1.1.1.cmml">−</mo><mfrac id="A1.F23.30.30.30.30.m1.1.1.2" xref="A1.F23.30.30.30.30.m1.1.1.2.cmml"><mn id="A1.F23.30.30.30.30.m1.1.1.2.2" xref="A1.F23.30.30.30.30.m1.1.1.2.2.cmml">2</mn><mn id="A1.F23.30.30.30.30.m1.1.1.2.3" xref="A1.F23.30.30.30.30.m1.1.1.2.3.cmml">3</mn></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.30.30.30.30.m1.1b"><apply id="A1.F23.30.30.30.30.m1.1.1.cmml" xref="A1.F23.30.30.30.30.m1.1.1"><minus id="A1.F23.30.30.30.30.m1.1.1.1.cmml" xref="A1.F23.30.30.30.30.m1.1.1"></minus><apply id="A1.F23.30.30.30.30.m1.1.1.2.cmml" xref="A1.F23.30.30.30.30.m1.1.1.2"><divide id="A1.F23.30.30.30.30.m1.1.1.2.1.cmml" xref="A1.F23.30.30.30.30.m1.1.1.2"></divide><cn type="integer" id="A1.F23.30.30.30.30.m1.1.1.2.2.cmml" xref="A1.F23.30.30.30.30.m1.1.1.2.2">2</cn><cn type="integer" id="A1.F23.30.30.30.30.m1.1.1.2.3.cmml" xref="A1.F23.30.30.30.30.m1.1.1.2.3">3</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.30.30.30.30.m1.1c">-\frac{2}{3}</annotation></semantics></math>.</span>
<span id="A1.F23.31.31.31.31" class="ltx_p">Final Answer: The final answer is <math id="A1.F23.31.31.31.31.m1.1" class="ltx_Math" alttext="-\frac{2}{3}" display="inline"><semantics id="A1.F23.31.31.31.31.m1.1a"><mrow id="A1.F23.31.31.31.31.m1.1.1" xref="A1.F23.31.31.31.31.m1.1.1.cmml"><mo id="A1.F23.31.31.31.31.m1.1.1a" xref="A1.F23.31.31.31.31.m1.1.1.cmml">−</mo><mfrac id="A1.F23.31.31.31.31.m1.1.1.2" xref="A1.F23.31.31.31.31.m1.1.1.2.cmml"><mn id="A1.F23.31.31.31.31.m1.1.1.2.2" xref="A1.F23.31.31.31.31.m1.1.1.2.2.cmml">2</mn><mn id="A1.F23.31.31.31.31.m1.1.1.2.3" xref="A1.F23.31.31.31.31.m1.1.1.2.3.cmml">3</mn></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A1.F23.31.31.31.31.m1.1b"><apply id="A1.F23.31.31.31.31.m1.1.1.cmml" xref="A1.F23.31.31.31.31.m1.1.1"><minus id="A1.F23.31.31.31.31.m1.1.1.1.cmml" xref="A1.F23.31.31.31.31.m1.1.1"></minus><apply id="A1.F23.31.31.31.31.m1.1.1.2.cmml" xref="A1.F23.31.31.31.31.m1.1.1.2"><divide id="A1.F23.31.31.31.31.m1.1.1.2.1.cmml" xref="A1.F23.31.31.31.31.m1.1.1.2"></divide><cn type="integer" id="A1.F23.31.31.31.31.m1.1.1.2.2.cmml" xref="A1.F23.31.31.31.31.m1.1.1.2.2">2</cn><cn type="integer" id="A1.F23.31.31.31.31.m1.1.1.2.3.cmml" xref="A1.F23.31.31.31.31.m1.1.1.2.3">3</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F23.31.31.31.31.m1.1c">-\frac{2}{3}</annotation></semantics></math>. I hope it is correct.</span>
<span id="A1.F23.31.31.31.52" class="ltx_p">—</span>
</span>
</span><span id="A1.F23.31.32" class="ltx_text" style="font-size:80%;"></span></p>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_figure">Figure A.23: </span>4-Shot Inner Monologue prompt used for MATH and GSM8K.</figcaption>
</figure>
<figure id="A1.F24" class="ltx_figure">
<p id="A1.F24.1" class="ltx_p ltx_align_center">

<span id="A1.F24.1.1.1" class="ltx_tabular ltx_align_middle ltx_framed ltx_framed_rectangle">
<span class="ltx_tbody">
<span id="A1.F24.1.1.1.2.1" class="ltx_tr">
<span id="A1.F24.1.1.1.2.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.2.1.1.1" class="ltx_text" style="font-size:80%;">Input: 255 378 650 363 42 447 898 211 104 145 975 6 827 769 977 901</span></span></span>
<span id="A1.F24.1.1.1.3.2" class="ltx_tr">
<span id="A1.F24.1.1.1.3.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.3.2.1.1" class="ltx_text" style="font-size:80%;">Output: Foo</span></span></span>
<span id="A1.F24.1.1.1.4.3" class="ltx_tr">
<span id="A1.F24.1.1.1.4.3.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.4.3.1.1" class="ltx_text" style="font-size:80%;">Input: 111 677 874 692 540 800 771 325 295 106 980 148 275 882 246 136</span></span></span>
<span id="A1.F24.1.1.1.5.4" class="ltx_tr">
<span id="A1.F24.1.1.1.5.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.5.4.1.1" class="ltx_text" style="font-size:80%;">Output: Foo</span></span></span>
<span id="A1.F24.1.1.1.6.5" class="ltx_tr">
<span id="A1.F24.1.1.1.6.5.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.6.5.1.1" class="ltx_text" style="font-size:80%;">Input: 136 215 529 65 265 475 45 639 678 95 460 902 746 919 181 838</span></span></span>
<span id="A1.F24.1.1.1.7.6" class="ltx_tr">
<span id="A1.F24.1.1.1.7.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.7.6.1.1" class="ltx_text" style="font-size:80%;">Output: Foo</span></span></span>
<span id="A1.F24.1.1.1.8.7" class="ltx_tr">
<span id="A1.F24.1.1.1.8.7.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.8.7.1.1" class="ltx_text" style="font-size:80%;">Input: 62 583 498 50 198 277 519 22 935 351 142 369 349 272 880 125</span></span></span>
<span id="A1.F24.1.1.1.9.8" class="ltx_tr">
<span id="A1.F24.1.1.1.9.8.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.9.8.1.1" class="ltx_text" style="font-size:80%;">Output: Bar</span></span></span>
<span id="A1.F24.1.1.1.10.9" class="ltx_tr">
<span id="A1.F24.1.1.1.10.9.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.10.9.1.1" class="ltx_text" style="font-size:80%;">Input: 101 99 830 735 732 76 243 703 564 3 225 20 136 333 195 441</span></span></span>
<span id="A1.F24.1.1.1.11.10" class="ltx_tr">
<span id="A1.F24.1.1.1.11.10.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.11.10.1.1" class="ltx_text" style="font-size:80%;">Output: Bar</span></span></span>
<span id="A1.F24.1.1.1.12.11" class="ltx_tr">
<span id="A1.F24.1.1.1.12.11.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.12.11.1.1" class="ltx_text" style="font-size:80%;">Input: 242 430 80 153 39 269 898 6 530 524 89 377 238 697 212 539</span></span></span>
<span id="A1.F24.1.1.1.13.12" class="ltx_tr">
<span id="A1.F24.1.1.1.13.12.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.13.12.1.1" class="ltx_text" style="font-size:80%;">Output: Bar</span></span></span>
<span id="A1.F24.1.1.1.14.13" class="ltx_tr">
<span id="A1.F24.1.1.1.14.13.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.14.13.1.1" class="ltx_text" style="font-size:80%;">Input: 261 83 244 37 170 277 161 779 544 272 893 535 71 394 64 607</span></span></span>
<span id="A1.F24.1.1.1.15.14" class="ltx_tr">
<span id="A1.F24.1.1.1.15.14.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.15.14.1.1" class="ltx_text" style="font-size:80%;">Output: Bar</span></span></span>
<span id="A1.F24.1.1.1.16.15" class="ltx_tr">
<span id="A1.F24.1.1.1.16.15.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.16.15.1.1" class="ltx_text" style="font-size:80%;">Input: 402 863 114 193 413 905 894 143 193 288 174 646 411 938 212 285</span></span></span>
<span id="A1.F24.1.1.1.17.16" class="ltx_tr">
<span id="A1.F24.1.1.1.17.16.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.17.16.1.1" class="ltx_text" style="font-size:80%;">Output: Bar</span></span></span>
<span id="A1.F24.1.1.1.18.17" class="ltx_tr">
<span id="A1.F24.1.1.1.18.17.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.18.17.1.1" class="ltx_text" style="font-size:80%;">Input: 869 365 622 671 191 780 492 836 381 450 184 388 604 79 924 926</span></span></span>
<span id="A1.F24.1.1.1.19.18" class="ltx_tr">
<span id="A1.F24.1.1.1.19.18.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.19.18.1.1" class="ltx_text" style="font-size:80%;">Output: Foo</span></span></span>
<span id="A1.F24.1.1.1.20.19" class="ltx_tr">
<span id="A1.F24.1.1.1.20.19.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.20.19.1.1" class="ltx_text" style="font-size:80%;">Input: 548 823 66 658 380 81 779 449 641 673 94 130 258 229 299 278</span></span></span>
<span id="A1.F24.1.1.1.21.20" class="ltx_tr">
<span id="A1.F24.1.1.1.21.20.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.21.20.1.1" class="ltx_text" style="font-size:80%;">Output: Bar</span></span></span>
<span id="A1.F24.1.1.1.22.21" class="ltx_tr">
<span id="A1.F24.1.1.1.22.21.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.22.21.1.1" class="ltx_text" style="font-size:80%;">Input: 700 409 398 375 236 745 32 33 333 173 902 399 176 95 851 897</span></span></span>
<span id="A1.F24.1.1.1.23.22" class="ltx_tr">
<span id="A1.F24.1.1.1.23.22.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.23.22.1.1" class="ltx_text" style="font-size:80%;">Output: Foo</span></span></span>
<span id="A1.F24.1.1.1.24.23" class="ltx_tr">
<span id="A1.F24.1.1.1.24.23.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.24.23.1.1" class="ltx_text" style="font-size:80%;">Input: 673 211 14 221 508 752 147 309 338 23 827 980 373 861 980 946</span></span></span>
<span id="A1.F24.1.1.1.25.24" class="ltx_tr">
<span id="A1.F24.1.1.1.25.24.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.25.24.1.1" class="ltx_text" style="font-size:80%;">Output: Foo</span></span></span>
<span id="A1.F24.1.1.1.26.25" class="ltx_tr">
<span id="A1.F24.1.1.1.26.25.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.26.25.1.1" class="ltx_text" style="font-size:80%;">Input: 528 608 334 210 228 186 559 20 302 93 84 436 726 114 785 865</span></span></span>
<span id="A1.F24.1.1.1.27.26" class="ltx_tr">
<span id="A1.F24.1.1.1.27.26.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.27.26.1.1" class="ltx_text" style="font-size:80%;">Output: Bar</span></span></span>
<span id="A1.F24.1.1.1.28.27" class="ltx_tr">
<span id="A1.F24.1.1.1.28.27.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.28.27.1.1" class="ltx_text" style="font-size:80%;">Input: 117 190 66 628 31 838 183 687 598 11 187 226 381 979 171 39</span></span></span>
<span id="A1.F24.1.1.1.29.28" class="ltx_tr">
<span id="A1.F24.1.1.1.29.28.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.29.28.1.1" class="ltx_text" style="font-size:80%;">Output: Bar</span></span></span>
<span id="A1.F24.1.1.1.30.29" class="ltx_tr">
<span id="A1.F24.1.1.1.30.29.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.30.29.1.1" class="ltx_text" style="font-size:80%;">Input: 802 730 854 392 529 95 15 987 800 266 551 816 145 390 419 686</span></span></span>
<span id="A1.F24.1.1.1.31.30" class="ltx_tr">
<span id="A1.F24.1.1.1.31.30.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.31.30.1.1" class="ltx_text" style="font-size:80%;">Output: Foo</span></span></span>
<span id="A1.F24.1.1.1.32.31" class="ltx_tr">
<span id="A1.F24.1.1.1.32.31.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.32.31.1.1" class="ltx_text" style="font-size:80%;">Input: 723 701 860 30 217 633 226 477 720 839 548 880 277 178 512 585</span></span></span>
<span id="A1.F24.1.1.1.33.32" class="ltx_tr">
<span id="A1.F24.1.1.1.33.32.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.33.32.1.1" class="ltx_text" style="font-size:80%;">Output: Foo</span></span></span>
<span id="A1.F24.1.1.1.1" class="ltx_tr">
<span id="A1.F24.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.1.1.1" class="ltx_text" style="font-size:80%;">Input: </span><math id="A1.F24.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="A1.F24.1.1.1.1.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="A1.F24.1.1.1.1.1.m1.1.1" xref="A1.F24.1.1.1.1.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="A1.F24.1.1.1.1.1.m1.1b"><ci id="A1.F24.1.1.1.1.1.m1.1.1.cmml" xref="A1.F24.1.1.1.1.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F24.1.1.1.1.1.m1.1c">\cdots</annotation></semantics></math></span></span>
<span id="A1.F24.1.1.1.34.33" class="ltx_tr">
<span id="A1.F24.1.1.1.34.33.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F24.1.1.1.34.33.1.1" class="ltx_text" style="font-size:80%;">Output:</span></span></span>
</span>
</span>
<span id="A1.F24.1.2" class="ltx_text" style="font-size:80%;"></span></p>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_figure">Figure A.24: </span>Example prompt with 8 shots per class for the linear classification in 16 dimensions, discussed in §<a href="#S5.SS2" title="5.2 Learning Non-Natural Language Tasks ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.2</span></a>. Here, we use semantically-unrelated labels (‘Foo’ and ‘Bar’) following <cite class="ltx_cite ltx_citemacro_citet">Wei et al. (<a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite>.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A1.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.6 </span>Prompts for Unsupervised ICL</h3>

<figure id="A1.F25" class="ltx_figure">
<p id="A1.F25.29" class="ltx_p ltx_align_center"><span id="A1.F25.29.29" class="ltx_text ltx_framed ltx_framed_rectangle" style="font-size:80%;border-color: #000000;">

<span id="A1.F25.29.29.29" class="ltx_inline-block ltx_minipage ltx_align_middle" style="width:390.3pt;">
<span id="A1.F25.29.29.29.30" class="ltx_p">You will be provided Problems similar to the ones below:</span>
<span id="A1.F25.29.29.29.31" class="ltx_p">Problem:
What is the remainder when 369,963 is divided by 6?</span>
<span id="A1.F25.29.29.29.32" class="ltx_p">Problem:
The solution to the inequality</span>
<span id="A1.Ex7" class="ltx_equation ltx_eqn_table">

<span><span class="ltx_equation ltx_eqn_row ltx_align_baseline">
<span class="ltx_eqn_cell ltx_eqn_center_padleft"></span>
<span class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex7.m1.1" class="ltx_Math" alttext="y=-x^{2}+ax+b\leq 0" display="block"><semantics id="A1.Ex7.m1.1a"><mrow id="A1.Ex7.m1.1.1" xref="A1.Ex7.m1.1.1.cmml"><mi id="A1.Ex7.m1.1.1.2" xref="A1.Ex7.m1.1.1.2.cmml">y</mi><mo id="A1.Ex7.m1.1.1.3" xref="A1.Ex7.m1.1.1.3.cmml">=</mo><mrow id="A1.Ex7.m1.1.1.4" xref="A1.Ex7.m1.1.1.4.cmml"><mrow id="A1.Ex7.m1.1.1.4.2" xref="A1.Ex7.m1.1.1.4.2.cmml"><mo id="A1.Ex7.m1.1.1.4.2a" xref="A1.Ex7.m1.1.1.4.2.cmml">−</mo><msup id="A1.Ex7.m1.1.1.4.2.2" xref="A1.Ex7.m1.1.1.4.2.2.cmml"><mi id="A1.Ex7.m1.1.1.4.2.2.2" xref="A1.Ex7.m1.1.1.4.2.2.2.cmml">x</mi><mn id="A1.Ex7.m1.1.1.4.2.2.3" xref="A1.Ex7.m1.1.1.4.2.2.3.cmml">2</mn></msup></mrow><mo id="A1.Ex7.m1.1.1.4.1" xref="A1.Ex7.m1.1.1.4.1.cmml">+</mo><mrow id="A1.Ex7.m1.1.1.4.3" xref="A1.Ex7.m1.1.1.4.3.cmml"><mi id="A1.Ex7.m1.1.1.4.3.2" xref="A1.Ex7.m1.1.1.4.3.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="A1.Ex7.m1.1.1.4.3.1" xref="A1.Ex7.m1.1.1.4.3.1.cmml">​</mo><mi id="A1.Ex7.m1.1.1.4.3.3" xref="A1.Ex7.m1.1.1.4.3.3.cmml">x</mi></mrow><mo id="A1.Ex7.m1.1.1.4.1a" xref="A1.Ex7.m1.1.1.4.1.cmml">+</mo><mi id="A1.Ex7.m1.1.1.4.4" xref="A1.Ex7.m1.1.1.4.4.cmml">b</mi></mrow><mo id="A1.Ex7.m1.1.1.5" xref="A1.Ex7.m1.1.1.5.cmml">≤</mo><mn id="A1.Ex7.m1.1.1.6" xref="A1.Ex7.m1.1.1.6.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex7.m1.1b"><apply id="A1.Ex7.m1.1.1.cmml" xref="A1.Ex7.m1.1.1"><and id="A1.Ex7.m1.1.1a.cmml" xref="A1.Ex7.m1.1.1"></and><apply id="A1.Ex7.m1.1.1b.cmml" xref="A1.Ex7.m1.1.1"><eq id="A1.Ex7.m1.1.1.3.cmml" xref="A1.Ex7.m1.1.1.3"></eq><ci id="A1.Ex7.m1.1.1.2.cmml" xref="A1.Ex7.m1.1.1.2">𝑦</ci><apply id="A1.Ex7.m1.1.1.4.cmml" xref="A1.Ex7.m1.1.1.4"><plus id="A1.Ex7.m1.1.1.4.1.cmml" xref="A1.Ex7.m1.1.1.4.1"></plus><apply id="A1.Ex7.m1.1.1.4.2.cmml" xref="A1.Ex7.m1.1.1.4.2"><minus id="A1.Ex7.m1.1.1.4.2.1.cmml" xref="A1.Ex7.m1.1.1.4.2"></minus><apply id="A1.Ex7.m1.1.1.4.2.2.cmml" xref="A1.Ex7.m1.1.1.4.2.2"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.4.2.2.1.cmml" xref="A1.Ex7.m1.1.1.4.2.2">superscript</csymbol><ci id="A1.Ex7.m1.1.1.4.2.2.2.cmml" xref="A1.Ex7.m1.1.1.4.2.2.2">𝑥</ci><cn type="integer" id="A1.Ex7.m1.1.1.4.2.2.3.cmml" xref="A1.Ex7.m1.1.1.4.2.2.3">2</cn></apply></apply><apply id="A1.Ex7.m1.1.1.4.3.cmml" xref="A1.Ex7.m1.1.1.4.3"><times id="A1.Ex7.m1.1.1.4.3.1.cmml" xref="A1.Ex7.m1.1.1.4.3.1"></times><ci id="A1.Ex7.m1.1.1.4.3.2.cmml" xref="A1.Ex7.m1.1.1.4.3.2">𝑎</ci><ci id="A1.Ex7.m1.1.1.4.3.3.cmml" xref="A1.Ex7.m1.1.1.4.3.3">𝑥</ci></apply><ci id="A1.Ex7.m1.1.1.4.4.cmml" xref="A1.Ex7.m1.1.1.4.4">𝑏</ci></apply></apply><apply id="A1.Ex7.m1.1.1c.cmml" xref="A1.Ex7.m1.1.1"><leq id="A1.Ex7.m1.1.1.5.cmml" xref="A1.Ex7.m1.1.1.5"></leq><share href="#A1.Ex7.m1.1.1.4.cmml" id="A1.Ex7.m1.1.1d.cmml" xref="A1.Ex7.m1.1.1"></share><cn type="integer" id="A1.Ex7.m1.1.1.6.cmml" xref="A1.Ex7.m1.1.1.6">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex7.m1.1c">y=-x^{2}+ax+b\leq 0</annotation></semantics></math></span>
<span class="ltx_eqn_cell ltx_eqn_center_padright"></span></span></span>
</span>
<span id="A1.F25.2.2.2.2" class="ltx_p">is <math id="A1.F25.1.1.1.1.m1.3" class="ltx_Math" alttext="(-\infty,-3]\cup[5,\infty)." display="inline"><semantics id="A1.F25.1.1.1.1.m1.3a"><mrow id="A1.F25.1.1.1.1.m1.3.3.1" xref="A1.F25.1.1.1.1.m1.3.3.1.1.cmml"><mrow id="A1.F25.1.1.1.1.m1.3.3.1.1" xref="A1.F25.1.1.1.1.m1.3.3.1.1.cmml"><mrow id="A1.F25.1.1.1.1.m1.3.3.1.1.2.2" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.3.cmml"><mo stretchy="false" id="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.3" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.3.cmml">(</mo><mrow id="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1" xref="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1.cmml"><mo id="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1a" xref="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1.cmml">−</mo><mi mathvariant="normal" id="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1.2" xref="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1.2.cmml">∞</mi></mrow><mo id="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.4" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.3.cmml">,</mo><mrow id="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2.cmml"><mo id="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2a" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2.cmml">−</mo><mn id="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2.2" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2.2.cmml">3</mn></mrow><mo stretchy="false" id="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.5" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.3.cmml">]</mo></mrow><mo id="A1.F25.1.1.1.1.m1.3.3.1.1.3" xref="A1.F25.1.1.1.1.m1.3.3.1.1.3.cmml">∪</mo><mrow id="A1.F25.1.1.1.1.m1.3.3.1.1.4.2" xref="A1.F25.1.1.1.1.m1.3.3.1.1.4.1.cmml"><mo stretchy="false" id="A1.F25.1.1.1.1.m1.3.3.1.1.4.2.1" xref="A1.F25.1.1.1.1.m1.3.3.1.1.4.1.cmml">[</mo><mn id="A1.F25.1.1.1.1.m1.1.1" xref="A1.F25.1.1.1.1.m1.1.1.cmml">5</mn><mo id="A1.F25.1.1.1.1.m1.3.3.1.1.4.2.2" xref="A1.F25.1.1.1.1.m1.3.3.1.1.4.1.cmml">,</mo><mi mathvariant="normal" id="A1.F25.1.1.1.1.m1.2.2" xref="A1.F25.1.1.1.1.m1.2.2.cmml">∞</mi><mo stretchy="false" id="A1.F25.1.1.1.1.m1.3.3.1.1.4.2.3" xref="A1.F25.1.1.1.1.m1.3.3.1.1.4.1.cmml">)</mo></mrow></mrow><mo lspace="0em" id="A1.F25.1.1.1.1.m1.3.3.1.2" xref="A1.F25.1.1.1.1.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.1.1.1.1.m1.3b"><apply id="A1.F25.1.1.1.1.m1.3.3.1.1.cmml" xref="A1.F25.1.1.1.1.m1.3.3.1"><union id="A1.F25.1.1.1.1.m1.3.3.1.1.3.cmml" xref="A1.F25.1.1.1.1.m1.3.3.1.1.3"></union><interval closure="open-closed" id="A1.F25.1.1.1.1.m1.3.3.1.1.2.3.cmml" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.2"><apply id="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1.cmml" xref="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1"><minus id="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1.1.cmml" xref="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1"></minus><infinity id="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1.2.cmml" xref="A1.F25.1.1.1.1.m1.3.3.1.1.1.1.1.2"></infinity></apply><apply id="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2.cmml" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2"><minus id="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2.1.cmml" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2"></minus><cn type="integer" id="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2.2.cmml" xref="A1.F25.1.1.1.1.m1.3.3.1.1.2.2.2.2">3</cn></apply></interval><interval closure="closed-open" id="A1.F25.1.1.1.1.m1.3.3.1.1.4.1.cmml" xref="A1.F25.1.1.1.1.m1.3.3.1.1.4.2"><cn type="integer" id="A1.F25.1.1.1.1.m1.1.1.cmml" xref="A1.F25.1.1.1.1.m1.1.1">5</cn><infinity id="A1.F25.1.1.1.1.m1.2.2.cmml" xref="A1.F25.1.1.1.1.m1.2.2"></infinity></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.1.1.1.1.m1.3c">(-\infty,-3]\cup[5,\infty).</annotation></semantics></math> Find the vertex of the parabola <math id="A1.F25.2.2.2.2.m2.1" class="ltx_Math" alttext="y=-x^{2}+ax+b." display="inline"><semantics id="A1.F25.2.2.2.2.m2.1a"><mrow id="A1.F25.2.2.2.2.m2.1.1.1" xref="A1.F25.2.2.2.2.m2.1.1.1.1.cmml"><mrow id="A1.F25.2.2.2.2.m2.1.1.1.1" xref="A1.F25.2.2.2.2.m2.1.1.1.1.cmml"><mi id="A1.F25.2.2.2.2.m2.1.1.1.1.2" xref="A1.F25.2.2.2.2.m2.1.1.1.1.2.cmml">y</mi><mo id="A1.F25.2.2.2.2.m2.1.1.1.1.1" xref="A1.F25.2.2.2.2.m2.1.1.1.1.1.cmml">=</mo><mrow id="A1.F25.2.2.2.2.m2.1.1.1.1.3" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.cmml"><mrow id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.cmml"><mo id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2a" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.cmml">−</mo><msup id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.cmml"><mi id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.2" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.2.cmml">x</mi><mn id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.3" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.3.cmml">2</mn></msup></mrow><mo id="A1.F25.2.2.2.2.m2.1.1.1.1.3.1" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.1.cmml">+</mo><mrow id="A1.F25.2.2.2.2.m2.1.1.1.1.3.3" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.cmml"><mi id="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.2" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.1" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.1.cmml">​</mo><mi id="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.3" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.3.cmml">x</mi></mrow><mo id="A1.F25.2.2.2.2.m2.1.1.1.1.3.1a" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.1.cmml">+</mo><mi id="A1.F25.2.2.2.2.m2.1.1.1.1.3.4" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.4.cmml">b</mi></mrow></mrow><mo lspace="0em" id="A1.F25.2.2.2.2.m2.1.1.1.2" xref="A1.F25.2.2.2.2.m2.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.2.2.2.2.m2.1b"><apply id="A1.F25.2.2.2.2.m2.1.1.1.1.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1"><eq id="A1.F25.2.2.2.2.m2.1.1.1.1.1.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.1"></eq><ci id="A1.F25.2.2.2.2.m2.1.1.1.1.2.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.2">𝑦</ci><apply id="A1.F25.2.2.2.2.m2.1.1.1.1.3.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3"><plus id="A1.F25.2.2.2.2.m2.1.1.1.1.3.1.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.1"></plus><apply id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2"><minus id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.1.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2"></minus><apply id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2"><csymbol cd="ambiguous" id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.1.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2">superscript</csymbol><ci id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.2.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.2">𝑥</ci><cn type="integer" id="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.3.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.2.2.3">2</cn></apply></apply><apply id="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.3"><times id="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.1.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.1"></times><ci id="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.2.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.2">𝑎</ci><ci id="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.3.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.3.3">𝑥</ci></apply><ci id="A1.F25.2.2.2.2.m2.1.1.1.1.3.4.cmml" xref="A1.F25.2.2.2.2.m2.1.1.1.1.3.4">𝑏</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.2.2.2.2.m2.1c">y=-x^{2}+ax+b.</annotation></semantics></math></span>
<span id="A1.F25.8.8.8.8" class="ltx_p">Problem:
Let <math id="A1.F25.3.3.3.3.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A1.F25.3.3.3.3.m1.1a"><mi id="A1.F25.3.3.3.3.m1.1.1" xref="A1.F25.3.3.3.3.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A1.F25.3.3.3.3.m1.1b"><ci id="A1.F25.3.3.3.3.m1.1.1.cmml" xref="A1.F25.3.3.3.3.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.3.3.3.3.m1.1c">x</annotation></semantics></math> be an angle such that <math id="A1.F25.4.4.4.4.m2.1" class="ltx_Math" alttext="\tan x=\frac{a}{b}" display="inline"><semantics id="A1.F25.4.4.4.4.m2.1a"><mrow id="A1.F25.4.4.4.4.m2.1.1" xref="A1.F25.4.4.4.4.m2.1.1.cmml"><mrow id="A1.F25.4.4.4.4.m2.1.1.2" xref="A1.F25.4.4.4.4.m2.1.1.2.cmml"><mi id="A1.F25.4.4.4.4.m2.1.1.2.1" xref="A1.F25.4.4.4.4.m2.1.1.2.1.cmml">tan</mi><mo lspace="0.167em" id="A1.F25.4.4.4.4.m2.1.1.2a" xref="A1.F25.4.4.4.4.m2.1.1.2.cmml">⁡</mo><mi id="A1.F25.4.4.4.4.m2.1.1.2.2" xref="A1.F25.4.4.4.4.m2.1.1.2.2.cmml">x</mi></mrow><mo id="A1.F25.4.4.4.4.m2.1.1.1" xref="A1.F25.4.4.4.4.m2.1.1.1.cmml">=</mo><mfrac id="A1.F25.4.4.4.4.m2.1.1.3" xref="A1.F25.4.4.4.4.m2.1.1.3.cmml"><mi id="A1.F25.4.4.4.4.m2.1.1.3.2" xref="A1.F25.4.4.4.4.m2.1.1.3.2.cmml">a</mi><mi id="A1.F25.4.4.4.4.m2.1.1.3.3" xref="A1.F25.4.4.4.4.m2.1.1.3.3.cmml">b</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.4.4.4.4.m2.1b"><apply id="A1.F25.4.4.4.4.m2.1.1.cmml" xref="A1.F25.4.4.4.4.m2.1.1"><eq id="A1.F25.4.4.4.4.m2.1.1.1.cmml" xref="A1.F25.4.4.4.4.m2.1.1.1"></eq><apply id="A1.F25.4.4.4.4.m2.1.1.2.cmml" xref="A1.F25.4.4.4.4.m2.1.1.2"><tan id="A1.F25.4.4.4.4.m2.1.1.2.1.cmml" xref="A1.F25.4.4.4.4.m2.1.1.2.1"></tan><ci id="A1.F25.4.4.4.4.m2.1.1.2.2.cmml" xref="A1.F25.4.4.4.4.m2.1.1.2.2">𝑥</ci></apply><apply id="A1.F25.4.4.4.4.m2.1.1.3.cmml" xref="A1.F25.4.4.4.4.m2.1.1.3"><divide id="A1.F25.4.4.4.4.m2.1.1.3.1.cmml" xref="A1.F25.4.4.4.4.m2.1.1.3"></divide><ci id="A1.F25.4.4.4.4.m2.1.1.3.2.cmml" xref="A1.F25.4.4.4.4.m2.1.1.3.2">𝑎</ci><ci id="A1.F25.4.4.4.4.m2.1.1.3.3.cmml" xref="A1.F25.4.4.4.4.m2.1.1.3.3">𝑏</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.4.4.4.4.m2.1c">\tan x=\frac{a}{b}</annotation></semantics></math> and <math id="A1.F25.5.5.5.5.m3.1" class="ltx_Math" alttext="\tan 2x=\frac{b}{a+b}." display="inline"><semantics id="A1.F25.5.5.5.5.m3.1a"><mrow id="A1.F25.5.5.5.5.m3.1.1.1" xref="A1.F25.5.5.5.5.m3.1.1.1.1.cmml"><mrow id="A1.F25.5.5.5.5.m3.1.1.1.1" xref="A1.F25.5.5.5.5.m3.1.1.1.1.cmml"><mrow id="A1.F25.5.5.5.5.m3.1.1.1.1.2" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.cmml"><mi id="A1.F25.5.5.5.5.m3.1.1.1.1.2.1" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.1.cmml">tan</mi><mo lspace="0.167em" id="A1.F25.5.5.5.5.m3.1.1.1.1.2a" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.cmml">⁡</mo><mrow id="A1.F25.5.5.5.5.m3.1.1.1.1.2.2" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.cmml"><mn id="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.2" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.1" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.1.cmml">​</mo><mi id="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.3" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.3.cmml">x</mi></mrow></mrow><mo id="A1.F25.5.5.5.5.m3.1.1.1.1.1" xref="A1.F25.5.5.5.5.m3.1.1.1.1.1.cmml">=</mo><mfrac id="A1.F25.5.5.5.5.m3.1.1.1.1.3" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.cmml"><mi id="A1.F25.5.5.5.5.m3.1.1.1.1.3.2" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.2.cmml">b</mi><mrow id="A1.F25.5.5.5.5.m3.1.1.1.1.3.3" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.cmml"><mi id="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.2" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.2.cmml">a</mi><mo id="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.1" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.1.cmml">+</mo><mi id="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.3" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.3.cmml">b</mi></mrow></mfrac></mrow><mo lspace="0em" id="A1.F25.5.5.5.5.m3.1.1.1.2" xref="A1.F25.5.5.5.5.m3.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.5.5.5.5.m3.1b"><apply id="A1.F25.5.5.5.5.m3.1.1.1.1.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1"><eq id="A1.F25.5.5.5.5.m3.1.1.1.1.1.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.1"></eq><apply id="A1.F25.5.5.5.5.m3.1.1.1.1.2.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2"><tan id="A1.F25.5.5.5.5.m3.1.1.1.1.2.1.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.1"></tan><apply id="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.2"><times id="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.1.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.1"></times><cn type="integer" id="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.2.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.2">2</cn><ci id="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.3.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.2.2.3">𝑥</ci></apply></apply><apply id="A1.F25.5.5.5.5.m3.1.1.1.1.3.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3"><divide id="A1.F25.5.5.5.5.m3.1.1.1.1.3.1.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3"></divide><ci id="A1.F25.5.5.5.5.m3.1.1.1.1.3.2.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.2">𝑏</ci><apply id="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.3"><plus id="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.1.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.1"></plus><ci id="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.2.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.2">𝑎</ci><ci id="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.3.cmml" xref="A1.F25.5.5.5.5.m3.1.1.1.1.3.3.3">𝑏</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.5.5.5.5.m3.1c">\tan 2x=\frac{b}{a+b}.</annotation></semantics></math> Then the least positive value of <math id="A1.F25.6.6.6.6.m4.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A1.F25.6.6.6.6.m4.1a"><mi id="A1.F25.6.6.6.6.m4.1.1" xref="A1.F25.6.6.6.6.m4.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A1.F25.6.6.6.6.m4.1b"><ci id="A1.F25.6.6.6.6.m4.1.1.cmml" xref="A1.F25.6.6.6.6.m4.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.6.6.6.6.m4.1c">x</annotation></semantics></math> equals <math id="A1.F25.7.7.7.7.m5.1" class="ltx_Math" alttext="\tan^{-1}k." display="inline"><semantics id="A1.F25.7.7.7.7.m5.1a"><mrow id="A1.F25.7.7.7.7.m5.1.1.1" xref="A1.F25.7.7.7.7.m5.1.1.1.1.cmml"><mrow id="A1.F25.7.7.7.7.m5.1.1.1.1" xref="A1.F25.7.7.7.7.m5.1.1.1.1.cmml"><msup id="A1.F25.7.7.7.7.m5.1.1.1.1.1" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1.cmml"><mi id="A1.F25.7.7.7.7.m5.1.1.1.1.1.2" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1.2.cmml">tan</mi><mrow id="A1.F25.7.7.7.7.m5.1.1.1.1.1.3" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1.3.cmml"><mo id="A1.F25.7.7.7.7.m5.1.1.1.1.1.3a" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1.3.cmml">−</mo><mn id="A1.F25.7.7.7.7.m5.1.1.1.1.1.3.2" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0.167em" id="A1.F25.7.7.7.7.m5.1.1.1.1a" xref="A1.F25.7.7.7.7.m5.1.1.1.1.cmml">⁡</mo><mi id="A1.F25.7.7.7.7.m5.1.1.1.1.2" xref="A1.F25.7.7.7.7.m5.1.1.1.1.2.cmml">k</mi></mrow><mo lspace="0em" id="A1.F25.7.7.7.7.m5.1.1.1.2" xref="A1.F25.7.7.7.7.m5.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.7.7.7.7.m5.1b"><apply id="A1.F25.7.7.7.7.m5.1.1.1.1.cmml" xref="A1.F25.7.7.7.7.m5.1.1.1"><apply id="A1.F25.7.7.7.7.m5.1.1.1.1.1.cmml" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.F25.7.7.7.7.m5.1.1.1.1.1.1.cmml" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1">superscript</csymbol><tan id="A1.F25.7.7.7.7.m5.1.1.1.1.1.2.cmml" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1.2"></tan><apply id="A1.F25.7.7.7.7.m5.1.1.1.1.1.3.cmml" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1.3"><minus id="A1.F25.7.7.7.7.m5.1.1.1.1.1.3.1.cmml" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1.3"></minus><cn type="integer" id="A1.F25.7.7.7.7.m5.1.1.1.1.1.3.2.cmml" xref="A1.F25.7.7.7.7.m5.1.1.1.1.1.3.2">1</cn></apply></apply><ci id="A1.F25.7.7.7.7.m5.1.1.1.1.2.cmml" xref="A1.F25.7.7.7.7.m5.1.1.1.1.2">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.7.7.7.7.m5.1c">\tan^{-1}k.</annotation></semantics></math> Compute <math id="A1.F25.8.8.8.8.m6.1" class="ltx_Math" alttext="k." display="inline"><semantics id="A1.F25.8.8.8.8.m6.1a"><mrow id="A1.F25.8.8.8.8.m6.1.2.2"><mi id="A1.F25.8.8.8.8.m6.1.1" xref="A1.F25.8.8.8.8.m6.1.1.cmml">k</mi><mo lspace="0em" id="A1.F25.8.8.8.8.m6.1.2.2.1">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.8.8.8.8.m6.1b"><ci id="A1.F25.8.8.8.8.m6.1.1.cmml" xref="A1.F25.8.8.8.8.m6.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.8.8.8.8.m6.1c">k.</annotation></semantics></math></span>
<span id="A1.F25.9.9.9.9" class="ltx_p">Problem:
Compute <math id="A1.F25.9.9.9.9.m1.1" class="ltx_Math" alttext="\sin 0^{\circ}" display="inline"><semantics id="A1.F25.9.9.9.9.m1.1a"><mrow id="A1.F25.9.9.9.9.m1.1.1" xref="A1.F25.9.9.9.9.m1.1.1.cmml"><mi id="A1.F25.9.9.9.9.m1.1.1.1" xref="A1.F25.9.9.9.9.m1.1.1.1.cmml">sin</mi><mo lspace="0.167em" id="A1.F25.9.9.9.9.m1.1.1a" xref="A1.F25.9.9.9.9.m1.1.1.cmml">⁡</mo><msup id="A1.F25.9.9.9.9.m1.1.1.2" xref="A1.F25.9.9.9.9.m1.1.1.2.cmml"><mn id="A1.F25.9.9.9.9.m1.1.1.2.2" xref="A1.F25.9.9.9.9.m1.1.1.2.2.cmml">0</mn><mo id="A1.F25.9.9.9.9.m1.1.1.2.3" xref="A1.F25.9.9.9.9.m1.1.1.2.3.cmml">∘</mo></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.9.9.9.9.m1.1b"><apply id="A1.F25.9.9.9.9.m1.1.1.cmml" xref="A1.F25.9.9.9.9.m1.1.1"><sin id="A1.F25.9.9.9.9.m1.1.1.1.cmml" xref="A1.F25.9.9.9.9.m1.1.1.1"></sin><apply id="A1.F25.9.9.9.9.m1.1.1.2.cmml" xref="A1.F25.9.9.9.9.m1.1.1.2"><csymbol cd="ambiguous" id="A1.F25.9.9.9.9.m1.1.1.2.1.cmml" xref="A1.F25.9.9.9.9.m1.1.1.2">superscript</csymbol><cn type="integer" id="A1.F25.9.9.9.9.m1.1.1.2.2.cmml" xref="A1.F25.9.9.9.9.m1.1.1.2.2">0</cn><compose id="A1.F25.9.9.9.9.m1.1.1.2.3.cmml" xref="A1.F25.9.9.9.9.m1.1.1.2.3"></compose></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.9.9.9.9.m1.1c">\sin 0^{\circ}</annotation></semantics></math>.</span>
<span id="A1.F25.29.29.29.33" class="ltx_p">Problem:
Let</span>
<span id="A1.Ex8" class="ltx_equation ltx_eqn_table">

<span><span class="ltx_equation ltx_eqn_row ltx_align_baseline">
<span class="ltx_eqn_cell ltx_eqn_center_padleft"></span>
<span class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex8.m1.5" class="ltx_Math" alttext="f(x)=\begin{cases}9x+4&amp;\text{if }x\text{ is an integer},\\
\lfloor{x}\rfloor+5&amp;\text{if }x\text{ is not an integer}.\end{cases}" display="block"><semantics id="A1.Ex8.m1.5a"><mrow id="A1.Ex8.m1.5.6" xref="A1.Ex8.m1.5.6.cmml"><mrow id="A1.Ex8.m1.5.6.2" xref="A1.Ex8.m1.5.6.2.cmml"><mi id="A1.Ex8.m1.5.6.2.2" xref="A1.Ex8.m1.5.6.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A1.Ex8.m1.5.6.2.1" xref="A1.Ex8.m1.5.6.2.1.cmml">​</mo><mrow id="A1.Ex8.m1.5.6.2.3.2" xref="A1.Ex8.m1.5.6.2.cmml"><mo stretchy="false" id="A1.Ex8.m1.5.6.2.3.2.1" xref="A1.Ex8.m1.5.6.2.cmml">(</mo><mi id="A1.Ex8.m1.5.5" xref="A1.Ex8.m1.5.5.cmml">x</mi><mo stretchy="false" id="A1.Ex8.m1.5.6.2.3.2.2" xref="A1.Ex8.m1.5.6.2.cmml">)</mo></mrow></mrow><mo id="A1.Ex8.m1.5.6.1" xref="A1.Ex8.m1.5.6.1.cmml">=</mo><mrow id="A1.Ex8.m1.4.4" xref="A1.Ex8.m1.5.6.3.1.cmml"><mo id="A1.Ex8.m1.4.4.5" xref="A1.Ex8.m1.5.6.3.1.1.cmml">{</mo><mtable columnspacing="5pt" displaystyle="true" rowspacing="0pt" id="A1.Ex8.m1.4.4.4" xref="A1.Ex8.m1.5.6.3.1.cmml"><mtr id="A1.Ex8.m1.4.4.4a" xref="A1.Ex8.m1.5.6.3.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="A1.Ex8.m1.4.4.4b" xref="A1.Ex8.m1.5.6.3.1.cmml"><mrow id="A1.Ex8.m1.1.1.1.1.1.1" xref="A1.Ex8.m1.1.1.1.1.1.1.cmml"><mrow id="A1.Ex8.m1.1.1.1.1.1.1.2" xref="A1.Ex8.m1.1.1.1.1.1.1.2.cmml"><mn id="A1.Ex8.m1.1.1.1.1.1.1.2.2" xref="A1.Ex8.m1.1.1.1.1.1.1.2.2.cmml">9</mn><mo lspace="0em" rspace="0em" id="A1.Ex8.m1.1.1.1.1.1.1.2.1" xref="A1.Ex8.m1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.Ex8.m1.1.1.1.1.1.1.2.3" xref="A1.Ex8.m1.1.1.1.1.1.1.2.3.cmml">x</mi></mrow><mo id="A1.Ex8.m1.1.1.1.1.1.1.1" xref="A1.Ex8.m1.1.1.1.1.1.1.1.cmml">+</mo><mn id="A1.Ex8.m1.1.1.1.1.1.1.3" xref="A1.Ex8.m1.1.1.1.1.1.1.3.cmml">4</mn></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="A1.Ex8.m1.4.4.4c" xref="A1.Ex8.m1.5.6.3.1.cmml"><mrow id="A1.Ex8.m1.2.2.2.2.2.1.1" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.cmml"><mrow id="A1.Ex8.m1.2.2.2.2.2.1.1.1" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.cmml"><mtext id="A1.Ex8.m1.2.2.2.2.2.1.1.1.2" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.2a.cmml">if </mtext><mo lspace="0em" rspace="0em" id="A1.Ex8.m1.2.2.2.2.2.1.1.1.1" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="A1.Ex8.m1.2.2.2.2.2.1.1.1.3" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.3.cmml">x</mi><mo lspace="0em" rspace="0em" id="A1.Ex8.m1.2.2.2.2.2.1.1.1.1a" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.1.cmml">​</mo><mtext id="A1.Ex8.m1.2.2.2.2.2.1.1.1.4" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.4a.cmml"> is an integer</mtext></mrow><mo id="A1.Ex8.m1.2.2.2.2.2.1.1.2" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.cmml">,</mo></mrow></mtd></mtr><mtr id="A1.Ex8.m1.4.4.4d" xref="A1.Ex8.m1.5.6.3.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="A1.Ex8.m1.4.4.4e" xref="A1.Ex8.m1.5.6.3.1.cmml"><mrow id="A1.Ex8.m1.3.3.3.3.1.1" xref="A1.Ex8.m1.3.3.3.3.1.1.cmml"><mrow id="A1.Ex8.m1.3.3.3.3.1.1.3.2" xref="A1.Ex8.m1.3.3.3.3.1.1.3.1.cmml"><mo stretchy="false" id="A1.Ex8.m1.3.3.3.3.1.1.3.2.1" xref="A1.Ex8.m1.3.3.3.3.1.1.3.1.1.cmml">⌊</mo><mi id="A1.Ex8.m1.3.3.3.3.1.1.1" xref="A1.Ex8.m1.3.3.3.3.1.1.1.cmml">x</mi><mo stretchy="false" id="A1.Ex8.m1.3.3.3.3.1.1.3.2.2" xref="A1.Ex8.m1.3.3.3.3.1.1.3.1.1.cmml">⌋</mo></mrow><mo id="A1.Ex8.m1.3.3.3.3.1.1.2" xref="A1.Ex8.m1.3.3.3.3.1.1.2.cmml">+</mo><mn id="A1.Ex8.m1.3.3.3.3.1.1.4" xref="A1.Ex8.m1.3.3.3.3.1.1.4.cmml">5</mn></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="A1.Ex8.m1.4.4.4f" xref="A1.Ex8.m1.5.6.3.1.cmml"><mrow id="A1.Ex8.m1.4.4.4.4.2.1.1" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.cmml"><mrow id="A1.Ex8.m1.4.4.4.4.2.1.1.1" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.cmml"><mtext id="A1.Ex8.m1.4.4.4.4.2.1.1.1.2" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.2a.cmml">if </mtext><mo lspace="0em" rspace="0em" id="A1.Ex8.m1.4.4.4.4.2.1.1.1.1" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.1.cmml">​</mo><mi id="A1.Ex8.m1.4.4.4.4.2.1.1.1.3" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.3.cmml">x</mi><mo lspace="0em" rspace="0em" id="A1.Ex8.m1.4.4.4.4.2.1.1.1.1a" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.1.cmml">​</mo><mtext id="A1.Ex8.m1.4.4.4.4.2.1.1.1.4" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.4a.cmml"> is not an integer</mtext></mrow><mo lspace="0em" id="A1.Ex8.m1.4.4.4.4.2.1.1.2" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.cmml">.</mo></mrow></mtd></mtr></mtable></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex8.m1.5b"><apply id="A1.Ex8.m1.5.6.cmml" xref="A1.Ex8.m1.5.6"><eq id="A1.Ex8.m1.5.6.1.cmml" xref="A1.Ex8.m1.5.6.1"></eq><apply id="A1.Ex8.m1.5.6.2.cmml" xref="A1.Ex8.m1.5.6.2"><times id="A1.Ex8.m1.5.6.2.1.cmml" xref="A1.Ex8.m1.5.6.2.1"></times><ci id="A1.Ex8.m1.5.6.2.2.cmml" xref="A1.Ex8.m1.5.6.2.2">𝑓</ci><ci id="A1.Ex8.m1.5.5.cmml" xref="A1.Ex8.m1.5.5">𝑥</ci></apply><apply id="A1.Ex8.m1.5.6.3.1.cmml" xref="A1.Ex8.m1.4.4"><csymbol cd="latexml" id="A1.Ex8.m1.5.6.3.1.1.cmml" xref="A1.Ex8.m1.4.4.5">cases</csymbol><apply id="A1.Ex8.m1.1.1.1.1.1.1.cmml" xref="A1.Ex8.m1.1.1.1.1.1.1"><plus id="A1.Ex8.m1.1.1.1.1.1.1.1.cmml" xref="A1.Ex8.m1.1.1.1.1.1.1.1"></plus><apply id="A1.Ex8.m1.1.1.1.1.1.1.2.cmml" xref="A1.Ex8.m1.1.1.1.1.1.1.2"><times id="A1.Ex8.m1.1.1.1.1.1.1.2.1.cmml" xref="A1.Ex8.m1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="A1.Ex8.m1.1.1.1.1.1.1.2.2.cmml" xref="A1.Ex8.m1.1.1.1.1.1.1.2.2">9</cn><ci id="A1.Ex8.m1.1.1.1.1.1.1.2.3.cmml" xref="A1.Ex8.m1.1.1.1.1.1.1.2.3">𝑥</ci></apply><cn type="integer" id="A1.Ex8.m1.1.1.1.1.1.1.3.cmml" xref="A1.Ex8.m1.1.1.1.1.1.1.3">4</cn></apply><apply id="A1.Ex8.m1.2.2.2.2.2.1.1.1.cmml" xref="A1.Ex8.m1.2.2.2.2.2.1.1"><times id="A1.Ex8.m1.2.2.2.2.2.1.1.1.1.cmml" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.1"></times><ci id="A1.Ex8.m1.2.2.2.2.2.1.1.1.2a.cmml" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.2"><mtext id="A1.Ex8.m1.2.2.2.2.2.1.1.1.2.cmml" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.2">if </mtext></ci><ci id="A1.Ex8.m1.2.2.2.2.2.1.1.1.3.cmml" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.3">𝑥</ci><ci id="A1.Ex8.m1.2.2.2.2.2.1.1.1.4a.cmml" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.4"><mtext id="A1.Ex8.m1.2.2.2.2.2.1.1.1.4.cmml" xref="A1.Ex8.m1.2.2.2.2.2.1.1.1.4"> is an integer</mtext></ci></apply><apply id="A1.Ex8.m1.3.3.3.3.1.1.cmml" xref="A1.Ex8.m1.3.3.3.3.1.1"><plus id="A1.Ex8.m1.3.3.3.3.1.1.2.cmml" xref="A1.Ex8.m1.3.3.3.3.1.1.2"></plus><apply id="A1.Ex8.m1.3.3.3.3.1.1.3.1.cmml" xref="A1.Ex8.m1.3.3.3.3.1.1.3.2"><floor id="A1.Ex8.m1.3.3.3.3.1.1.3.1.1.cmml" xref="A1.Ex8.m1.3.3.3.3.1.1.3.2.1"></floor><ci id="A1.Ex8.m1.3.3.3.3.1.1.1.cmml" xref="A1.Ex8.m1.3.3.3.3.1.1.1">𝑥</ci></apply><cn type="integer" id="A1.Ex8.m1.3.3.3.3.1.1.4.cmml" xref="A1.Ex8.m1.3.3.3.3.1.1.4">5</cn></apply><apply id="A1.Ex8.m1.4.4.4.4.2.1.1.1.cmml" xref="A1.Ex8.m1.4.4.4.4.2.1.1"><times id="A1.Ex8.m1.4.4.4.4.2.1.1.1.1.cmml" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.1"></times><ci id="A1.Ex8.m1.4.4.4.4.2.1.1.1.2a.cmml" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.2"><mtext id="A1.Ex8.m1.4.4.4.4.2.1.1.1.2.cmml" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.2">if </mtext></ci><ci id="A1.Ex8.m1.4.4.4.4.2.1.1.1.3.cmml" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.3">𝑥</ci><ci id="A1.Ex8.m1.4.4.4.4.2.1.1.1.4a.cmml" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.4"><mtext id="A1.Ex8.m1.4.4.4.4.2.1.1.1.4.cmml" xref="A1.Ex8.m1.4.4.4.4.2.1.1.1.4"> is not an integer</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex8.m1.5c">f(x)=\begin{cases}9x+4&amp;\text{if }x\text{ is an integer},\\
\lfloor{x}\rfloor+5&amp;\text{if }x\text{ is not an integer}.\end{cases}</annotation></semantics></math></span>
<span class="ltx_eqn_cell ltx_eqn_center_padright"></span></span></span>
</span>
<span id="A1.F25.10.10.10.10" class="ltx_p">Find <math id="A1.F25.10.10.10.10.m1.1" class="ltx_Math" alttext="f(\sqrt{29})" display="inline"><semantics id="A1.F25.10.10.10.10.m1.1a"><mrow id="A1.F25.10.10.10.10.m1.1.2" xref="A1.F25.10.10.10.10.m1.1.2.cmml"><mi id="A1.F25.10.10.10.10.m1.1.2.2" xref="A1.F25.10.10.10.10.m1.1.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A1.F25.10.10.10.10.m1.1.2.1" xref="A1.F25.10.10.10.10.m1.1.2.1.cmml">​</mo><mrow id="A1.F25.10.10.10.10.m1.1.2.3.2" xref="A1.F25.10.10.10.10.m1.1.1.cmml"><mo stretchy="false" id="A1.F25.10.10.10.10.m1.1.2.3.2.1" xref="A1.F25.10.10.10.10.m1.1.1.cmml">(</mo><msqrt id="A1.F25.10.10.10.10.m1.1.1" xref="A1.F25.10.10.10.10.m1.1.1.cmml"><mn id="A1.F25.10.10.10.10.m1.1.1.2" xref="A1.F25.10.10.10.10.m1.1.1.2.cmml">29</mn></msqrt><mo stretchy="false" id="A1.F25.10.10.10.10.m1.1.2.3.2.2" xref="A1.F25.10.10.10.10.m1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.10.10.10.10.m1.1b"><apply id="A1.F25.10.10.10.10.m1.1.2.cmml" xref="A1.F25.10.10.10.10.m1.1.2"><times id="A1.F25.10.10.10.10.m1.1.2.1.cmml" xref="A1.F25.10.10.10.10.m1.1.2.1"></times><ci id="A1.F25.10.10.10.10.m1.1.2.2.cmml" xref="A1.F25.10.10.10.10.m1.1.2.2">𝑓</ci><apply id="A1.F25.10.10.10.10.m1.1.1.cmml" xref="A1.F25.10.10.10.10.m1.1.2.3.2"><root id="A1.F25.10.10.10.10.m1.1.1a.cmml" xref="A1.F25.10.10.10.10.m1.1.2.3.2"></root><cn type="integer" id="A1.F25.10.10.10.10.m1.1.1.2.cmml" xref="A1.F25.10.10.10.10.m1.1.1.2">29</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.10.10.10.10.m1.1c">f(\sqrt{29})</annotation></semantics></math>.</span>
<span id="A1.F25.29.29.29.34" class="ltx_p">—</span>
<span id="A1.F25.29.29.29.35" class="ltx_p">Now, I am going to give you a series of demonstrations of math Problems and Solutions. When you respond, respond only with the Solution of the final Problem, thinking step by step.”</span>
<span id="A1.F25.29.29.29.36" class="ltx_p">—</span>
<span id="A1.F25.11.11.11.11" class="ltx_p">Problem:
Find the domain of the expression <math id="A1.F25.11.11.11.11.m1.1" class="ltx_Math" alttext="\frac{\sqrt{x-2}}{\sqrt{5-x}}" display="inline"><semantics id="A1.F25.11.11.11.11.m1.1a"><mfrac id="A1.F25.11.11.11.11.m1.1.1" xref="A1.F25.11.11.11.11.m1.1.1.cmml"><msqrt id="A1.F25.11.11.11.11.m1.1.1.2" xref="A1.F25.11.11.11.11.m1.1.1.2.cmml"><mrow id="A1.F25.11.11.11.11.m1.1.1.2.2" xref="A1.F25.11.11.11.11.m1.1.1.2.2.cmml"><mi id="A1.F25.11.11.11.11.m1.1.1.2.2.2" xref="A1.F25.11.11.11.11.m1.1.1.2.2.2.cmml">x</mi><mo id="A1.F25.11.11.11.11.m1.1.1.2.2.1" xref="A1.F25.11.11.11.11.m1.1.1.2.2.1.cmml">−</mo><mn id="A1.F25.11.11.11.11.m1.1.1.2.2.3" xref="A1.F25.11.11.11.11.m1.1.1.2.2.3.cmml">2</mn></mrow></msqrt><msqrt id="A1.F25.11.11.11.11.m1.1.1.3" xref="A1.F25.11.11.11.11.m1.1.1.3.cmml"><mrow id="A1.F25.11.11.11.11.m1.1.1.3.2" xref="A1.F25.11.11.11.11.m1.1.1.3.2.cmml"><mn id="A1.F25.11.11.11.11.m1.1.1.3.2.2" xref="A1.F25.11.11.11.11.m1.1.1.3.2.2.cmml">5</mn><mo id="A1.F25.11.11.11.11.m1.1.1.3.2.1" xref="A1.F25.11.11.11.11.m1.1.1.3.2.1.cmml">−</mo><mi id="A1.F25.11.11.11.11.m1.1.1.3.2.3" xref="A1.F25.11.11.11.11.m1.1.1.3.2.3.cmml">x</mi></mrow></msqrt></mfrac><annotation-xml encoding="MathML-Content" id="A1.F25.11.11.11.11.m1.1b"><apply id="A1.F25.11.11.11.11.m1.1.1.cmml" xref="A1.F25.11.11.11.11.m1.1.1"><divide id="A1.F25.11.11.11.11.m1.1.1.1.cmml" xref="A1.F25.11.11.11.11.m1.1.1"></divide><apply id="A1.F25.11.11.11.11.m1.1.1.2.cmml" xref="A1.F25.11.11.11.11.m1.1.1.2"><root id="A1.F25.11.11.11.11.m1.1.1.2a.cmml" xref="A1.F25.11.11.11.11.m1.1.1.2"></root><apply id="A1.F25.11.11.11.11.m1.1.1.2.2.cmml" xref="A1.F25.11.11.11.11.m1.1.1.2.2"><minus id="A1.F25.11.11.11.11.m1.1.1.2.2.1.cmml" xref="A1.F25.11.11.11.11.m1.1.1.2.2.1"></minus><ci id="A1.F25.11.11.11.11.m1.1.1.2.2.2.cmml" xref="A1.F25.11.11.11.11.m1.1.1.2.2.2">𝑥</ci><cn type="integer" id="A1.F25.11.11.11.11.m1.1.1.2.2.3.cmml" xref="A1.F25.11.11.11.11.m1.1.1.2.2.3">2</cn></apply></apply><apply id="A1.F25.11.11.11.11.m1.1.1.3.cmml" xref="A1.F25.11.11.11.11.m1.1.1.3"><root id="A1.F25.11.11.11.11.m1.1.1.3a.cmml" xref="A1.F25.11.11.11.11.m1.1.1.3"></root><apply id="A1.F25.11.11.11.11.m1.1.1.3.2.cmml" xref="A1.F25.11.11.11.11.m1.1.1.3.2"><minus id="A1.F25.11.11.11.11.m1.1.1.3.2.1.cmml" xref="A1.F25.11.11.11.11.m1.1.1.3.2.1"></minus><cn type="integer" id="A1.F25.11.11.11.11.m1.1.1.3.2.2.cmml" xref="A1.F25.11.11.11.11.m1.1.1.3.2.2">5</cn><ci id="A1.F25.11.11.11.11.m1.1.1.3.2.3.cmml" xref="A1.F25.11.11.11.11.m1.1.1.3.2.3">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.11.11.11.11.m1.1c">\frac{\sqrt{x-2}}{\sqrt{5-x}}</annotation></semantics></math>.</span>
<span id="A1.F25.29.29.29.37" class="ltx_p">Solution:</span>
<span id="A1.F25.29.29.29.38" class="ltx_p">I need to find the domain of this expression.</span>
<span id="A1.F25.29.29.29.39" class="ltx_p">This expression has two square root expressions.</span>
<span id="A1.F25.29.29.29.40" class="ltx_p">I know the values under each square root must be non-negative.</span>
<span id="A1.F25.29.29.29.41" class="ltx_p">So let me start by ensuring that.</span>
<span id="A1.F25.13.13.13.13" class="ltx_p">I set <math id="A1.F25.12.12.12.12.m1.1" class="ltx_Math" alttext="x-2\geq 0" display="inline"><semantics id="A1.F25.12.12.12.12.m1.1a"><mrow id="A1.F25.12.12.12.12.m1.1.1" xref="A1.F25.12.12.12.12.m1.1.1.cmml"><mrow id="A1.F25.12.12.12.12.m1.1.1.2" xref="A1.F25.12.12.12.12.m1.1.1.2.cmml"><mi id="A1.F25.12.12.12.12.m1.1.1.2.2" xref="A1.F25.12.12.12.12.m1.1.1.2.2.cmml">x</mi><mo id="A1.F25.12.12.12.12.m1.1.1.2.1" xref="A1.F25.12.12.12.12.m1.1.1.2.1.cmml">−</mo><mn id="A1.F25.12.12.12.12.m1.1.1.2.3" xref="A1.F25.12.12.12.12.m1.1.1.2.3.cmml">2</mn></mrow><mo id="A1.F25.12.12.12.12.m1.1.1.1" xref="A1.F25.12.12.12.12.m1.1.1.1.cmml">≥</mo><mn id="A1.F25.12.12.12.12.m1.1.1.3" xref="A1.F25.12.12.12.12.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.12.12.12.12.m1.1b"><apply id="A1.F25.12.12.12.12.m1.1.1.cmml" xref="A1.F25.12.12.12.12.m1.1.1"><geq id="A1.F25.12.12.12.12.m1.1.1.1.cmml" xref="A1.F25.12.12.12.12.m1.1.1.1"></geq><apply id="A1.F25.12.12.12.12.m1.1.1.2.cmml" xref="A1.F25.12.12.12.12.m1.1.1.2"><minus id="A1.F25.12.12.12.12.m1.1.1.2.1.cmml" xref="A1.F25.12.12.12.12.m1.1.1.2.1"></minus><ci id="A1.F25.12.12.12.12.m1.1.1.2.2.cmml" xref="A1.F25.12.12.12.12.m1.1.1.2.2">𝑥</ci><cn type="integer" id="A1.F25.12.12.12.12.m1.1.1.2.3.cmml" xref="A1.F25.12.12.12.12.m1.1.1.2.3">2</cn></apply><cn type="integer" id="A1.F25.12.12.12.12.m1.1.1.3.cmml" xref="A1.F25.12.12.12.12.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.12.12.12.12.m1.1c">x-2\geq 0</annotation></semantics></math>, which simplifies to <math id="A1.F25.13.13.13.13.m2.1" class="ltx_Math" alttext="x\geq 2" display="inline"><semantics id="A1.F25.13.13.13.13.m2.1a"><mrow id="A1.F25.13.13.13.13.m2.1.1" xref="A1.F25.13.13.13.13.m2.1.1.cmml"><mi id="A1.F25.13.13.13.13.m2.1.1.2" xref="A1.F25.13.13.13.13.m2.1.1.2.cmml">x</mi><mo id="A1.F25.13.13.13.13.m2.1.1.1" xref="A1.F25.13.13.13.13.m2.1.1.1.cmml">≥</mo><mn id="A1.F25.13.13.13.13.m2.1.1.3" xref="A1.F25.13.13.13.13.m2.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.13.13.13.13.m2.1b"><apply id="A1.F25.13.13.13.13.m2.1.1.cmml" xref="A1.F25.13.13.13.13.m2.1.1"><geq id="A1.F25.13.13.13.13.m2.1.1.1.cmml" xref="A1.F25.13.13.13.13.m2.1.1.1"></geq><ci id="A1.F25.13.13.13.13.m2.1.1.2.cmml" xref="A1.F25.13.13.13.13.m2.1.1.2">𝑥</ci><cn type="integer" id="A1.F25.13.13.13.13.m2.1.1.3.cmml" xref="A1.F25.13.13.13.13.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.13.13.13.13.m2.1c">x\geq 2</annotation></semantics></math>.</span>
<span id="A1.F25.15.15.15.15" class="ltx_p">Then, I set <math id="A1.F25.14.14.14.14.m1.1" class="ltx_Math" alttext="5-x\geq 0" display="inline"><semantics id="A1.F25.14.14.14.14.m1.1a"><mrow id="A1.F25.14.14.14.14.m1.1.1" xref="A1.F25.14.14.14.14.m1.1.1.cmml"><mrow id="A1.F25.14.14.14.14.m1.1.1.2" xref="A1.F25.14.14.14.14.m1.1.1.2.cmml"><mn id="A1.F25.14.14.14.14.m1.1.1.2.2" xref="A1.F25.14.14.14.14.m1.1.1.2.2.cmml">5</mn><mo id="A1.F25.14.14.14.14.m1.1.1.2.1" xref="A1.F25.14.14.14.14.m1.1.1.2.1.cmml">−</mo><mi id="A1.F25.14.14.14.14.m1.1.1.2.3" xref="A1.F25.14.14.14.14.m1.1.1.2.3.cmml">x</mi></mrow><mo id="A1.F25.14.14.14.14.m1.1.1.1" xref="A1.F25.14.14.14.14.m1.1.1.1.cmml">≥</mo><mn id="A1.F25.14.14.14.14.m1.1.1.3" xref="A1.F25.14.14.14.14.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.14.14.14.14.m1.1b"><apply id="A1.F25.14.14.14.14.m1.1.1.cmml" xref="A1.F25.14.14.14.14.m1.1.1"><geq id="A1.F25.14.14.14.14.m1.1.1.1.cmml" xref="A1.F25.14.14.14.14.m1.1.1.1"></geq><apply id="A1.F25.14.14.14.14.m1.1.1.2.cmml" xref="A1.F25.14.14.14.14.m1.1.1.2"><minus id="A1.F25.14.14.14.14.m1.1.1.2.1.cmml" xref="A1.F25.14.14.14.14.m1.1.1.2.1"></minus><cn type="integer" id="A1.F25.14.14.14.14.m1.1.1.2.2.cmml" xref="A1.F25.14.14.14.14.m1.1.1.2.2">5</cn><ci id="A1.F25.14.14.14.14.m1.1.1.2.3.cmml" xref="A1.F25.14.14.14.14.m1.1.1.2.3">𝑥</ci></apply><cn type="integer" id="A1.F25.14.14.14.14.m1.1.1.3.cmml" xref="A1.F25.14.14.14.14.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.14.14.14.14.m1.1c">5-x\geq 0</annotation></semantics></math>, which simplifies to <math id="A1.F25.15.15.15.15.m2.1" class="ltx_Math" alttext="x\leq 5" display="inline"><semantics id="A1.F25.15.15.15.15.m2.1a"><mrow id="A1.F25.15.15.15.15.m2.1.1" xref="A1.F25.15.15.15.15.m2.1.1.cmml"><mi id="A1.F25.15.15.15.15.m2.1.1.2" xref="A1.F25.15.15.15.15.m2.1.1.2.cmml">x</mi><mo id="A1.F25.15.15.15.15.m2.1.1.1" xref="A1.F25.15.15.15.15.m2.1.1.1.cmml">≤</mo><mn id="A1.F25.15.15.15.15.m2.1.1.3" xref="A1.F25.15.15.15.15.m2.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.15.15.15.15.m2.1b"><apply id="A1.F25.15.15.15.15.m2.1.1.cmml" xref="A1.F25.15.15.15.15.m2.1.1"><leq id="A1.F25.15.15.15.15.m2.1.1.1.cmml" xref="A1.F25.15.15.15.15.m2.1.1.1"></leq><ci id="A1.F25.15.15.15.15.m2.1.1.2.cmml" xref="A1.F25.15.15.15.15.m2.1.1.2">𝑥</ci><cn type="integer" id="A1.F25.15.15.15.15.m2.1.1.3.cmml" xref="A1.F25.15.15.15.15.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.15.15.15.15.m2.1c">x\leq 5</annotation></semantics></math>.</span>
<span id="A1.F25.29.29.29.42" class="ltx_p">But I need to make sure the denominator is not zero.</span>
<span id="A1.F25.17.17.17.17" class="ltx_p">So, <math id="A1.F25.16.16.16.16.m1.1" class="ltx_Math" alttext="5-x" display="inline"><semantics id="A1.F25.16.16.16.16.m1.1a"><mrow id="A1.F25.16.16.16.16.m1.1.1" xref="A1.F25.16.16.16.16.m1.1.1.cmml"><mn id="A1.F25.16.16.16.16.m1.1.1.2" xref="A1.F25.16.16.16.16.m1.1.1.2.cmml">5</mn><mo id="A1.F25.16.16.16.16.m1.1.1.1" xref="A1.F25.16.16.16.16.m1.1.1.1.cmml">−</mo><mi id="A1.F25.16.16.16.16.m1.1.1.3" xref="A1.F25.16.16.16.16.m1.1.1.3.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.16.16.16.16.m1.1b"><apply id="A1.F25.16.16.16.16.m1.1.1.cmml" xref="A1.F25.16.16.16.16.m1.1.1"><minus id="A1.F25.16.16.16.16.m1.1.1.1.cmml" xref="A1.F25.16.16.16.16.m1.1.1.1"></minus><cn type="integer" id="A1.F25.16.16.16.16.m1.1.1.2.cmml" xref="A1.F25.16.16.16.16.m1.1.1.2">5</cn><ci id="A1.F25.16.16.16.16.m1.1.1.3.cmml" xref="A1.F25.16.16.16.16.m1.1.1.3">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.16.16.16.16.m1.1c">5-x</annotation></semantics></math> must also be greater than zero, which gives me <math id="A1.F25.17.17.17.17.m2.1" class="ltx_Math" alttext="x&lt;5" display="inline"><semantics id="A1.F25.17.17.17.17.m2.1a"><mrow id="A1.F25.17.17.17.17.m2.1.1" xref="A1.F25.17.17.17.17.m2.1.1.cmml"><mi id="A1.F25.17.17.17.17.m2.1.1.2" xref="A1.F25.17.17.17.17.m2.1.1.2.cmml">x</mi><mo id="A1.F25.17.17.17.17.m2.1.1.1" xref="A1.F25.17.17.17.17.m2.1.1.1.cmml">&lt;</mo><mn id="A1.F25.17.17.17.17.m2.1.1.3" xref="A1.F25.17.17.17.17.m2.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.17.17.17.17.m2.1b"><apply id="A1.F25.17.17.17.17.m2.1.1.cmml" xref="A1.F25.17.17.17.17.m2.1.1"><lt id="A1.F25.17.17.17.17.m2.1.1.1.cmml" xref="A1.F25.17.17.17.17.m2.1.1.1"></lt><ci id="A1.F25.17.17.17.17.m2.1.1.2.cmml" xref="A1.F25.17.17.17.17.m2.1.1.2">𝑥</ci><cn type="integer" id="A1.F25.17.17.17.17.m2.1.1.3.cmml" xref="A1.F25.17.17.17.17.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.17.17.17.17.m2.1c">x&lt;5</annotation></semantics></math>.</span>
<span id="A1.F25.18.18.18.18" class="ltx_p">Therefore, the domain of the expression is <math id="A1.F25.18.18.18.18.m1.2" class="ltx_Math" alttext="\boxed{[2,5)}" display="inline"><semantics id="A1.F25.18.18.18.18.m1.2a"><menclose notation="box" id="A1.F25.18.18.18.18.m1.2.2" xref="A1.F25.18.18.18.18.m1.2.2.3.cmml"><mrow id="A1.F25.18.18.18.18.m1.2.2.4" xref="A1.F25.18.18.18.18.m1.2.2.3.cmml"><mo stretchy="false" id="A1.F25.18.18.18.18.m1.2.2.4.1" xref="A1.F25.18.18.18.18.m1.2.2.3.cmml">[</mo><mn id="A1.F25.18.18.18.18.m1.1.1.1" xref="A1.F25.18.18.18.18.m1.1.1.1.cmml">2</mn><mo id="A1.F25.18.18.18.18.m1.2.2.4.2" xref="A1.F25.18.18.18.18.m1.2.2.3.cmml">,</mo><mn id="A1.F25.18.18.18.18.m1.2.2.2" xref="A1.F25.18.18.18.18.m1.2.2.2.cmml">5</mn><mo stretchy="false" id="A1.F25.18.18.18.18.m1.2.2.4.3" xref="A1.F25.18.18.18.18.m1.2.2.3.cmml">)</mo></mrow></menclose><annotation-xml encoding="MathML-Content" id="A1.F25.18.18.18.18.m1.2b"><interval closure="closed-open" id="A1.F25.18.18.18.18.m1.2.2.3.cmml" xref="A1.F25.18.18.18.18.m1.2.2"><cn type="integer" id="A1.F25.18.18.18.18.m1.1.1.1.cmml" xref="A1.F25.18.18.18.18.m1.1.1.1">2</cn><cn type="integer" id="A1.F25.18.18.18.18.m1.2.2.2.cmml" xref="A1.F25.18.18.18.18.m1.2.2.2">5</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.18.18.18.18.m1.2c">\boxed{[2,5)}</annotation></semantics></math>.</span>
<span id="A1.F25.19.19.19.19" class="ltx_p">Answer: <math id="A1.F25.19.19.19.19.m1.2" class="ltx_Math" alttext="[2,5)" display="inline"><semantics id="A1.F25.19.19.19.19.m1.2a"><mrow id="A1.F25.19.19.19.19.m1.2.3.2" xref="A1.F25.19.19.19.19.m1.2.3.1.cmml"><mo stretchy="false" id="A1.F25.19.19.19.19.m1.2.3.2.1" xref="A1.F25.19.19.19.19.m1.2.3.1.cmml">[</mo><mn id="A1.F25.19.19.19.19.m1.1.1" xref="A1.F25.19.19.19.19.m1.1.1.cmml">2</mn><mo id="A1.F25.19.19.19.19.m1.2.3.2.2" xref="A1.F25.19.19.19.19.m1.2.3.1.cmml">,</mo><mn id="A1.F25.19.19.19.19.m1.2.2" xref="A1.F25.19.19.19.19.m1.2.2.cmml">5</mn><mo stretchy="false" id="A1.F25.19.19.19.19.m1.2.3.2.3" xref="A1.F25.19.19.19.19.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.19.19.19.19.m1.2b"><interval closure="closed-open" id="A1.F25.19.19.19.19.m1.2.3.1.cmml" xref="A1.F25.19.19.19.19.m1.2.3.2"><cn type="integer" id="A1.F25.19.19.19.19.m1.1.1.cmml" xref="A1.F25.19.19.19.19.m1.1.1">2</cn><cn type="integer" id="A1.F25.19.19.19.19.m1.2.2.cmml" xref="A1.F25.19.19.19.19.m1.2.2">5</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.19.19.19.19.m1.2c">[2,5)</annotation></semantics></math>.</span>
<span id="A1.F25.20.20.20.20" class="ltx_p">Final Answer: The final answer is <math id="A1.F25.20.20.20.20.m1.2" class="ltx_Math" alttext="[2,5)" display="inline"><semantics id="A1.F25.20.20.20.20.m1.2a"><mrow id="A1.F25.20.20.20.20.m1.2.3.2" xref="A1.F25.20.20.20.20.m1.2.3.1.cmml"><mo stretchy="false" id="A1.F25.20.20.20.20.m1.2.3.2.1" xref="A1.F25.20.20.20.20.m1.2.3.1.cmml">[</mo><mn id="A1.F25.20.20.20.20.m1.1.1" xref="A1.F25.20.20.20.20.m1.1.1.cmml">2</mn><mo id="A1.F25.20.20.20.20.m1.2.3.2.2" xref="A1.F25.20.20.20.20.m1.2.3.1.cmml">,</mo><mn id="A1.F25.20.20.20.20.m1.2.2" xref="A1.F25.20.20.20.20.m1.2.2.cmml">5</mn><mo stretchy="false" id="A1.F25.20.20.20.20.m1.2.3.2.3" xref="A1.F25.20.20.20.20.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.20.20.20.20.m1.2b"><interval closure="closed-open" id="A1.F25.20.20.20.20.m1.2.3.1.cmml" xref="A1.F25.20.20.20.20.m1.2.3.2"><cn type="integer" id="A1.F25.20.20.20.20.m1.1.1.cmml" xref="A1.F25.20.20.20.20.m1.1.1">2</cn><cn type="integer" id="A1.F25.20.20.20.20.m1.2.2.cmml" xref="A1.F25.20.20.20.20.m1.2.2">5</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.20.20.20.20.m1.2c">[2,5)</annotation></semantics></math>. I hope it is correct.</span>
<span id="A1.F25.29.29.29.43" class="ltx_p">—</span>
<span id="A1.F25.23.23.23.23" class="ltx_p">Problem:
If <math id="A1.F25.21.21.21.21.m1.1" class="ltx_Math" alttext="\det\mathbf{A}=2" display="inline"><semantics id="A1.F25.21.21.21.21.m1.1a"><mrow id="A1.F25.21.21.21.21.m1.1.1" xref="A1.F25.21.21.21.21.m1.1.1.cmml"><mrow id="A1.F25.21.21.21.21.m1.1.1.2" xref="A1.F25.21.21.21.21.m1.1.1.2.cmml"><mo rspace="0.167em" id="A1.F25.21.21.21.21.m1.1.1.2.1" xref="A1.F25.21.21.21.21.m1.1.1.2.1.cmml">det</mo><mi id="A1.F25.21.21.21.21.m1.1.1.2.2" xref="A1.F25.21.21.21.21.m1.1.1.2.2.cmml">𝐀</mi></mrow><mo id="A1.F25.21.21.21.21.m1.1.1.1" xref="A1.F25.21.21.21.21.m1.1.1.1.cmml">=</mo><mn id="A1.F25.21.21.21.21.m1.1.1.3" xref="A1.F25.21.21.21.21.m1.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.21.21.21.21.m1.1b"><apply id="A1.F25.21.21.21.21.m1.1.1.cmml" xref="A1.F25.21.21.21.21.m1.1.1"><eq id="A1.F25.21.21.21.21.m1.1.1.1.cmml" xref="A1.F25.21.21.21.21.m1.1.1.1"></eq><apply id="A1.F25.21.21.21.21.m1.1.1.2.cmml" xref="A1.F25.21.21.21.21.m1.1.1.2"><determinant id="A1.F25.21.21.21.21.m1.1.1.2.1.cmml" xref="A1.F25.21.21.21.21.m1.1.1.2.1"></determinant><ci id="A1.F25.21.21.21.21.m1.1.1.2.2.cmml" xref="A1.F25.21.21.21.21.m1.1.1.2.2">𝐀</ci></apply><cn type="integer" id="A1.F25.21.21.21.21.m1.1.1.3.cmml" xref="A1.F25.21.21.21.21.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.21.21.21.21.m1.1c">\det\mathbf{A}=2</annotation></semantics></math> and <math id="A1.F25.22.22.22.22.m2.1" class="ltx_Math" alttext="\det\mathbf{B}=12," display="inline"><semantics id="A1.F25.22.22.22.22.m2.1a"><mrow id="A1.F25.22.22.22.22.m2.1.1.1" xref="A1.F25.22.22.22.22.m2.1.1.1.1.cmml"><mrow id="A1.F25.22.22.22.22.m2.1.1.1.1" xref="A1.F25.22.22.22.22.m2.1.1.1.1.cmml"><mrow id="A1.F25.22.22.22.22.m2.1.1.1.1.2" xref="A1.F25.22.22.22.22.m2.1.1.1.1.2.cmml"><mo rspace="0.167em" id="A1.F25.22.22.22.22.m2.1.1.1.1.2.1" xref="A1.F25.22.22.22.22.m2.1.1.1.1.2.1.cmml">det</mo><mi id="A1.F25.22.22.22.22.m2.1.1.1.1.2.2" xref="A1.F25.22.22.22.22.m2.1.1.1.1.2.2.cmml">𝐁</mi></mrow><mo id="A1.F25.22.22.22.22.m2.1.1.1.1.1" xref="A1.F25.22.22.22.22.m2.1.1.1.1.1.cmml">=</mo><mn id="A1.F25.22.22.22.22.m2.1.1.1.1.3" xref="A1.F25.22.22.22.22.m2.1.1.1.1.3.cmml">12</mn></mrow><mo id="A1.F25.22.22.22.22.m2.1.1.1.2" xref="A1.F25.22.22.22.22.m2.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.22.22.22.22.m2.1b"><apply id="A1.F25.22.22.22.22.m2.1.1.1.1.cmml" xref="A1.F25.22.22.22.22.m2.1.1.1"><eq id="A1.F25.22.22.22.22.m2.1.1.1.1.1.cmml" xref="A1.F25.22.22.22.22.m2.1.1.1.1.1"></eq><apply id="A1.F25.22.22.22.22.m2.1.1.1.1.2.cmml" xref="A1.F25.22.22.22.22.m2.1.1.1.1.2"><determinant id="A1.F25.22.22.22.22.m2.1.1.1.1.2.1.cmml" xref="A1.F25.22.22.22.22.m2.1.1.1.1.2.1"></determinant><ci id="A1.F25.22.22.22.22.m2.1.1.1.1.2.2.cmml" xref="A1.F25.22.22.22.22.m2.1.1.1.1.2.2">𝐁</ci></apply><cn type="integer" id="A1.F25.22.22.22.22.m2.1.1.1.1.3.cmml" xref="A1.F25.22.22.22.22.m2.1.1.1.1.3">12</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.22.22.22.22.m2.1c">\det\mathbf{B}=12,</annotation></semantics></math> then find <math id="A1.F25.23.23.23.23.m3.2" class="ltx_Math" alttext="\det(\mathbf{A}\mathbf{B})." display="inline"><semantics id="A1.F25.23.23.23.23.m3.2a"><mrow id="A1.F25.23.23.23.23.m3.2.2.1" xref="A1.F25.23.23.23.23.m3.2.2.1.1.cmml"><mrow id="A1.F25.23.23.23.23.m3.2.2.1.1" xref="A1.F25.23.23.23.23.m3.2.2.1.1.cmml"><mo rspace="0em" id="A1.F25.23.23.23.23.m3.2.2.1.1.1" xref="A1.F25.23.23.23.23.m3.2.2.1.1.1.cmml">det</mo><mrow id="A1.F25.23.23.23.23.m3.2.2.1.1.2.2" xref="A1.F25.23.23.23.23.m3.2.2.1.1.cmml"><mo stretchy="false" id="A1.F25.23.23.23.23.m3.2.2.1.1.2.2.1" xref="A1.F25.23.23.23.23.m3.2.2.1.1.cmml">(</mo><mi id="A1.F25.23.23.23.23.m3.1.1" xref="A1.F25.23.23.23.23.m3.1.1.cmml">𝐀𝐁</mi><mo stretchy="false" id="A1.F25.23.23.23.23.m3.2.2.1.1.2.2.2" xref="A1.F25.23.23.23.23.m3.2.2.1.1.cmml">)</mo></mrow></mrow><mo lspace="0em" id="A1.F25.23.23.23.23.m3.2.2.1.2" xref="A1.F25.23.23.23.23.m3.2.2.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.23.23.23.23.m3.2b"><apply id="A1.F25.23.23.23.23.m3.2.2.1.1.cmml" xref="A1.F25.23.23.23.23.m3.2.2.1"><determinant id="A1.F25.23.23.23.23.m3.2.2.1.1.1.cmml" xref="A1.F25.23.23.23.23.m3.2.2.1.1.1"></determinant><ci id="A1.F25.23.23.23.23.m3.1.1.cmml" xref="A1.F25.23.23.23.23.m3.1.1">𝐀𝐁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.23.23.23.23.m3.2c">\det(\mathbf{A}\mathbf{B}).</annotation></semantics></math></span>
<span id="A1.F25.29.29.29.44" class="ltx_p">Solution:</span>
<span id="A1.F25.29.29.29.45" class="ltx_p">I need to find the determinant of the product of matrices A and B.</span>
<span id="A1.F25.29.29.29.46" class="ltx_p">I remember that the determinant of the product of two matrices equals the product of their determinants.</span>
<span id="A1.F25.24.24.24.24" class="ltx_p">So, <math id="A1.F25.24.24.24.24.m1.5" class="ltx_Math" alttext="\det(\mathbf{A}\mathbf{B})=(\det\mathbf{A})(\det\mathbf{B})=(2)(12)=\boxed{24}" display="inline"><semantics id="A1.F25.24.24.24.24.m1.5a"><mrow id="A1.F25.24.24.24.24.m1.5.5" xref="A1.F25.24.24.24.24.m1.5.5.cmml"><mrow id="A1.F25.24.24.24.24.m1.5.5.4" xref="A1.F25.24.24.24.24.m1.5.5.4.cmml"><mo rspace="0em" id="A1.F25.24.24.24.24.m1.5.5.4.1" xref="A1.F25.24.24.24.24.m1.5.5.4.1.cmml">det</mo><mrow id="A1.F25.24.24.24.24.m1.5.5.4.2.2" xref="A1.F25.24.24.24.24.m1.5.5.4.cmml"><mo stretchy="false" id="A1.F25.24.24.24.24.m1.5.5.4.2.2.1" xref="A1.F25.24.24.24.24.m1.5.5.4.cmml">(</mo><mi id="A1.F25.24.24.24.24.m1.1.1" xref="A1.F25.24.24.24.24.m1.1.1.cmml">𝐀𝐁</mi><mo stretchy="false" id="A1.F25.24.24.24.24.m1.5.5.4.2.2.2" xref="A1.F25.24.24.24.24.m1.5.5.4.cmml">)</mo></mrow></mrow><mo id="A1.F25.24.24.24.24.m1.5.5.5" xref="A1.F25.24.24.24.24.m1.5.5.5.cmml">=</mo><mrow id="A1.F25.24.24.24.24.m1.5.5.2" xref="A1.F25.24.24.24.24.m1.5.5.2.cmml"><mrow id="A1.F25.24.24.24.24.m1.4.4.1.1.1" xref="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.cmml"><mo stretchy="false" id="A1.F25.24.24.24.24.m1.4.4.1.1.1.2" xref="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.cmml">(</mo><mrow id="A1.F25.24.24.24.24.m1.4.4.1.1.1.1" xref="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.cmml"><mo lspace="0em" rspace="0.167em" id="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.1" xref="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.1.cmml">det</mo><mi id="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.2" xref="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.2.cmml">𝐀</mi></mrow><mo stretchy="false" id="A1.F25.24.24.24.24.m1.4.4.1.1.1.3" xref="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.F25.24.24.24.24.m1.5.5.2.3" xref="A1.F25.24.24.24.24.m1.5.5.2.3.cmml">​</mo><mrow id="A1.F25.24.24.24.24.m1.5.5.2.2.1" xref="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.cmml"><mo stretchy="false" id="A1.F25.24.24.24.24.m1.5.5.2.2.1.2" xref="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.cmml">(</mo><mrow id="A1.F25.24.24.24.24.m1.5.5.2.2.1.1" xref="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.cmml"><mo lspace="0em" rspace="0.167em" id="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.1" xref="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.1.cmml">det</mo><mi id="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.2" xref="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.2.cmml">𝐁</mi></mrow><mo stretchy="false" id="A1.F25.24.24.24.24.m1.5.5.2.2.1.3" xref="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.cmml">)</mo></mrow></mrow><mo id="A1.F25.24.24.24.24.m1.5.5.6" xref="A1.F25.24.24.24.24.m1.5.5.6.cmml">=</mo><mrow id="A1.F25.24.24.24.24.m1.5.5.7" xref="A1.F25.24.24.24.24.m1.5.5.7.cmml"><mrow id="A1.F25.24.24.24.24.m1.5.5.7.2.2" xref="A1.F25.24.24.24.24.m1.5.5.7.cmml"><mo stretchy="false" id="A1.F25.24.24.24.24.m1.5.5.7.2.2.1" xref="A1.F25.24.24.24.24.m1.5.5.7.cmml">(</mo><mn id="A1.F25.24.24.24.24.m1.2.2" xref="A1.F25.24.24.24.24.m1.2.2.cmml">2</mn><mo stretchy="false" id="A1.F25.24.24.24.24.m1.5.5.7.2.2.2" xref="A1.F25.24.24.24.24.m1.5.5.7.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.F25.24.24.24.24.m1.5.5.7.1" xref="A1.F25.24.24.24.24.m1.5.5.7.1.cmml">​</mo><mrow id="A1.F25.24.24.24.24.m1.5.5.7.3.2" xref="A1.F25.24.24.24.24.m1.5.5.7.cmml"><mo stretchy="false" id="A1.F25.24.24.24.24.m1.5.5.7.3.2.1" xref="A1.F25.24.24.24.24.m1.5.5.7.cmml">(</mo><mn id="A1.F25.24.24.24.24.m1.3.3" xref="A1.F25.24.24.24.24.m1.3.3.cmml">12</mn><mo stretchy="false" id="A1.F25.24.24.24.24.m1.5.5.7.3.2.2" xref="A1.F25.24.24.24.24.m1.5.5.7.cmml">)</mo></mrow></mrow><mo id="A1.F25.24.24.24.24.m1.5.5.8" xref="A1.F25.24.24.24.24.m1.5.5.8.cmml">=</mo><menclose notation="box" id="A1.F25.24.24.24.24.m1.5.5.9" xref="A1.F25.24.24.24.24.m1.5.5.9.cmml"><mn id="A1.F25.24.24.24.24.m1.5.5.9a" xref="A1.F25.24.24.24.24.m1.5.5.9.cmml">24</mn></menclose></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.24.24.24.24.m1.5b"><apply id="A1.F25.24.24.24.24.m1.5.5.cmml" xref="A1.F25.24.24.24.24.m1.5.5"><and id="A1.F25.24.24.24.24.m1.5.5a.cmml" xref="A1.F25.24.24.24.24.m1.5.5"></and><apply id="A1.F25.24.24.24.24.m1.5.5b.cmml" xref="A1.F25.24.24.24.24.m1.5.5"><eq id="A1.F25.24.24.24.24.m1.5.5.5.cmml" xref="A1.F25.24.24.24.24.m1.5.5.5"></eq><apply id="A1.F25.24.24.24.24.m1.5.5.4.cmml" xref="A1.F25.24.24.24.24.m1.5.5.4"><determinant id="A1.F25.24.24.24.24.m1.5.5.4.1.cmml" xref="A1.F25.24.24.24.24.m1.5.5.4.1"></determinant><ci id="A1.F25.24.24.24.24.m1.1.1.cmml" xref="A1.F25.24.24.24.24.m1.1.1">𝐀𝐁</ci></apply><apply id="A1.F25.24.24.24.24.m1.5.5.2.cmml" xref="A1.F25.24.24.24.24.m1.5.5.2"><times id="A1.F25.24.24.24.24.m1.5.5.2.3.cmml" xref="A1.F25.24.24.24.24.m1.5.5.2.3"></times><apply id="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.cmml" xref="A1.F25.24.24.24.24.m1.4.4.1.1.1"><determinant id="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.1.cmml" xref="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.1"></determinant><ci id="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.2.cmml" xref="A1.F25.24.24.24.24.m1.4.4.1.1.1.1.2">𝐀</ci></apply><apply id="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.cmml" xref="A1.F25.24.24.24.24.m1.5.5.2.2.1"><determinant id="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.1.cmml" xref="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.1"></determinant><ci id="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.2.cmml" xref="A1.F25.24.24.24.24.m1.5.5.2.2.1.1.2">𝐁</ci></apply></apply></apply><apply id="A1.F25.24.24.24.24.m1.5.5c.cmml" xref="A1.F25.24.24.24.24.m1.5.5"><eq id="A1.F25.24.24.24.24.m1.5.5.6.cmml" xref="A1.F25.24.24.24.24.m1.5.5.6"></eq><share href="#A1.F25.24.24.24.24.m1.5.5.2.cmml" id="A1.F25.24.24.24.24.m1.5.5d.cmml" xref="A1.F25.24.24.24.24.m1.5.5"></share><apply id="A1.F25.24.24.24.24.m1.5.5.7.cmml" xref="A1.F25.24.24.24.24.m1.5.5.7"><times id="A1.F25.24.24.24.24.m1.5.5.7.1.cmml" xref="A1.F25.24.24.24.24.m1.5.5.7.1"></times><cn type="integer" id="A1.F25.24.24.24.24.m1.2.2.cmml" xref="A1.F25.24.24.24.24.m1.2.2">2</cn><cn type="integer" id="A1.F25.24.24.24.24.m1.3.3.cmml" xref="A1.F25.24.24.24.24.m1.3.3">12</cn></apply></apply><apply id="A1.F25.24.24.24.24.m1.5.5e.cmml" xref="A1.F25.24.24.24.24.m1.5.5"><eq id="A1.F25.24.24.24.24.m1.5.5.8.cmml" xref="A1.F25.24.24.24.24.m1.5.5.8"></eq><share href="#A1.F25.24.24.24.24.m1.5.5.7.cmml" id="A1.F25.24.24.24.24.m1.5.5f.cmml" xref="A1.F25.24.24.24.24.m1.5.5"></share><cn type="integer" id="A1.F25.24.24.24.24.m1.5.5.9.cmml" xref="A1.F25.24.24.24.24.m1.5.5.9">24</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.24.24.24.24.m1.5c">\det(\mathbf{A}\mathbf{B})=(\det\mathbf{A})(\det\mathbf{B})=(2)(12)=\boxed{24}</annotation></semantics></math>.</span>
<span id="A1.F25.25.25.25.25" class="ltx_p">Answer: <math id="A1.F25.25.25.25.25.m1.1" class="ltx_Math" alttext="24" display="inline"><semantics id="A1.F25.25.25.25.25.m1.1a"><mn id="A1.F25.25.25.25.25.m1.1.1" xref="A1.F25.25.25.25.25.m1.1.1.cmml">24</mn><annotation-xml encoding="MathML-Content" id="A1.F25.25.25.25.25.m1.1b"><cn type="integer" id="A1.F25.25.25.25.25.m1.1.1.cmml" xref="A1.F25.25.25.25.25.m1.1.1">24</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.25.25.25.25.m1.1c">24</annotation></semantics></math>.</span>
<span id="A1.F25.26.26.26.26" class="ltx_p">Final Answer: The final answer is <math id="A1.F25.26.26.26.26.m1.1" class="ltx_Math" alttext="24" display="inline"><semantics id="A1.F25.26.26.26.26.m1.1a"><mn id="A1.F25.26.26.26.26.m1.1.1" xref="A1.F25.26.26.26.26.m1.1.1.cmml">24</mn><annotation-xml encoding="MathML-Content" id="A1.F25.26.26.26.26.m1.1b"><cn type="integer" id="A1.F25.26.26.26.26.m1.1.1.cmml" xref="A1.F25.26.26.26.26.m1.1.1">24</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.26.26.26.26.m1.1c">24</annotation></semantics></math>. I hope it is correct.</span>
<span id="A1.F25.29.29.29.47" class="ltx_p">—</span>
<span id="A1.F25.29.29.29.29" class="ltx_p">Problem:
Evaluate <math id="A1.F25.27.27.27.27.m1.2" class="ltx_Math" alttext="(x+y)(x-y)" display="inline"><semantics id="A1.F25.27.27.27.27.m1.2a"><mrow id="A1.F25.27.27.27.27.m1.2.2" xref="A1.F25.27.27.27.27.m1.2.2.cmml"><mrow id="A1.F25.27.27.27.27.m1.1.1.1.1" xref="A1.F25.27.27.27.27.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.F25.27.27.27.27.m1.1.1.1.1.2" xref="A1.F25.27.27.27.27.m1.1.1.1.1.1.cmml">(</mo><mrow id="A1.F25.27.27.27.27.m1.1.1.1.1.1" xref="A1.F25.27.27.27.27.m1.1.1.1.1.1.cmml"><mi id="A1.F25.27.27.27.27.m1.1.1.1.1.1.2" xref="A1.F25.27.27.27.27.m1.1.1.1.1.1.2.cmml">x</mi><mo id="A1.F25.27.27.27.27.m1.1.1.1.1.1.1" xref="A1.F25.27.27.27.27.m1.1.1.1.1.1.1.cmml">+</mo><mi id="A1.F25.27.27.27.27.m1.1.1.1.1.1.3" xref="A1.F25.27.27.27.27.m1.1.1.1.1.1.3.cmml">y</mi></mrow><mo stretchy="false" id="A1.F25.27.27.27.27.m1.1.1.1.1.3" xref="A1.F25.27.27.27.27.m1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.F25.27.27.27.27.m1.2.2.3" xref="A1.F25.27.27.27.27.m1.2.2.3.cmml">​</mo><mrow id="A1.F25.27.27.27.27.m1.2.2.2.1" xref="A1.F25.27.27.27.27.m1.2.2.2.1.1.cmml"><mo stretchy="false" id="A1.F25.27.27.27.27.m1.2.2.2.1.2" xref="A1.F25.27.27.27.27.m1.2.2.2.1.1.cmml">(</mo><mrow id="A1.F25.27.27.27.27.m1.2.2.2.1.1" xref="A1.F25.27.27.27.27.m1.2.2.2.1.1.cmml"><mi id="A1.F25.27.27.27.27.m1.2.2.2.1.1.2" xref="A1.F25.27.27.27.27.m1.2.2.2.1.1.2.cmml">x</mi><mo id="A1.F25.27.27.27.27.m1.2.2.2.1.1.1" xref="A1.F25.27.27.27.27.m1.2.2.2.1.1.1.cmml">−</mo><mi id="A1.F25.27.27.27.27.m1.2.2.2.1.1.3" xref="A1.F25.27.27.27.27.m1.2.2.2.1.1.3.cmml">y</mi></mrow><mo stretchy="false" id="A1.F25.27.27.27.27.m1.2.2.2.1.3" xref="A1.F25.27.27.27.27.m1.2.2.2.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.27.27.27.27.m1.2b"><apply id="A1.F25.27.27.27.27.m1.2.2.cmml" xref="A1.F25.27.27.27.27.m1.2.2"><times id="A1.F25.27.27.27.27.m1.2.2.3.cmml" xref="A1.F25.27.27.27.27.m1.2.2.3"></times><apply id="A1.F25.27.27.27.27.m1.1.1.1.1.1.cmml" xref="A1.F25.27.27.27.27.m1.1.1.1.1"><plus id="A1.F25.27.27.27.27.m1.1.1.1.1.1.1.cmml" xref="A1.F25.27.27.27.27.m1.1.1.1.1.1.1"></plus><ci id="A1.F25.27.27.27.27.m1.1.1.1.1.1.2.cmml" xref="A1.F25.27.27.27.27.m1.1.1.1.1.1.2">𝑥</ci><ci id="A1.F25.27.27.27.27.m1.1.1.1.1.1.3.cmml" xref="A1.F25.27.27.27.27.m1.1.1.1.1.1.3">𝑦</ci></apply><apply id="A1.F25.27.27.27.27.m1.2.2.2.1.1.cmml" xref="A1.F25.27.27.27.27.m1.2.2.2.1"><minus id="A1.F25.27.27.27.27.m1.2.2.2.1.1.1.cmml" xref="A1.F25.27.27.27.27.m1.2.2.2.1.1.1"></minus><ci id="A1.F25.27.27.27.27.m1.2.2.2.1.1.2.cmml" xref="A1.F25.27.27.27.27.m1.2.2.2.1.1.2">𝑥</ci><ci id="A1.F25.27.27.27.27.m1.2.2.2.1.1.3.cmml" xref="A1.F25.27.27.27.27.m1.2.2.2.1.1.3">𝑦</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.27.27.27.27.m1.2c">(x+y)(x-y)</annotation></semantics></math> when <math id="A1.F25.28.28.28.28.m2.1" class="ltx_Math" alttext="x=13" display="inline"><semantics id="A1.F25.28.28.28.28.m2.1a"><mrow id="A1.F25.28.28.28.28.m2.1.1" xref="A1.F25.28.28.28.28.m2.1.1.cmml"><mi id="A1.F25.28.28.28.28.m2.1.1.2" xref="A1.F25.28.28.28.28.m2.1.1.2.cmml">x</mi><mo id="A1.F25.28.28.28.28.m2.1.1.1" xref="A1.F25.28.28.28.28.m2.1.1.1.cmml">=</mo><mn id="A1.F25.28.28.28.28.m2.1.1.3" xref="A1.F25.28.28.28.28.m2.1.1.3.cmml">13</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.28.28.28.28.m2.1b"><apply id="A1.F25.28.28.28.28.m2.1.1.cmml" xref="A1.F25.28.28.28.28.m2.1.1"><eq id="A1.F25.28.28.28.28.m2.1.1.1.cmml" xref="A1.F25.28.28.28.28.m2.1.1.1"></eq><ci id="A1.F25.28.28.28.28.m2.1.1.2.cmml" xref="A1.F25.28.28.28.28.m2.1.1.2">𝑥</ci><cn type="integer" id="A1.F25.28.28.28.28.m2.1.1.3.cmml" xref="A1.F25.28.28.28.28.m2.1.1.3">13</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.28.28.28.28.m2.1c">x=13</annotation></semantics></math> and <math id="A1.F25.29.29.29.29.m3.1" class="ltx_Math" alttext="y=5" display="inline"><semantics id="A1.F25.29.29.29.29.m3.1a"><mrow id="A1.F25.29.29.29.29.m3.1.1" xref="A1.F25.29.29.29.29.m3.1.1.cmml"><mi id="A1.F25.29.29.29.29.m3.1.1.2" xref="A1.F25.29.29.29.29.m3.1.1.2.cmml">y</mi><mo id="A1.F25.29.29.29.29.m3.1.1.1" xref="A1.F25.29.29.29.29.m3.1.1.1.cmml">=</mo><mn id="A1.F25.29.29.29.29.m3.1.1.3" xref="A1.F25.29.29.29.29.m3.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F25.29.29.29.29.m3.1b"><apply id="A1.F25.29.29.29.29.m3.1.1.cmml" xref="A1.F25.29.29.29.29.m3.1.1"><eq id="A1.F25.29.29.29.29.m3.1.1.1.cmml" xref="A1.F25.29.29.29.29.m3.1.1.1"></eq><ci id="A1.F25.29.29.29.29.m3.1.1.2.cmml" xref="A1.F25.29.29.29.29.m3.1.1.2">𝑦</ci><cn type="integer" id="A1.F25.29.29.29.29.m3.1.1.3.cmml" xref="A1.F25.29.29.29.29.m3.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F25.29.29.29.29.m3.1c">y=5</annotation></semantics></math>.</span>
</span>
</span><span id="A1.F25.29.30" class="ltx_text" style="font-size:80%;"></span></p>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_figure">Figure A.25: </span>Prompt used for Unsupervised ICL with MATH and GSM8K. We first start with a preamble saying that we are going to list a number of problems, and then we list the problems. We then give another pre-amble to specify the output format, and include up to 4 examples to fully describe this output format. As we go to the many-shot setting with hundreds of examples, we only increase the number of problems in the prompt, not the problem-solution pairs at the end.</figcaption>
</figure>
<figure id="A1.F26" class="ltx_figure">
<p id="A1.F26.2" class="ltx_p ltx_align_center">

<span id="A1.F26.2.2.2" class="ltx_tabular ltx_align_middle ltx_framed ltx_framed_rectangle">
<span class="ltx_tbody">
<span id="A1.F26.2.2.2.3.1" class="ltx_tr">
<span id="A1.F26.2.2.2.3.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.3.1.1.1" class="ltx_text" style="font-size:80%;">You will be provided questions similar to the ones below:</span></span></span>
<span id="A1.F26.2.2.2.4.2" class="ltx_tr">
<span id="A1.F26.2.2.2.4.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.4.2.1.1" class="ltx_text" style="font-size:80%;">Question:</span></span></span>
<span id="A1.F26.2.2.2.5.3" class="ltx_tr">
<span id="A1.F26.2.2.2.5.3.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.5.3.1.1" class="ltx_text" style="font-size:80%;">A large gene has dozens of exons, of which the central ones code for folded triple helical repeats that connect the cytoskeleton</span></span></span>
<span id="A1.F26.2.2.2.6.4" class="ltx_tr">
<span id="A1.F26.2.2.2.6.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.6.4.1.1" class="ltx_text" style="font-size:80%;">with sarcolemma and extracellular space. Each exon usually codes for one folded triple alpha helix. The most common mutations</span></span></span>
<span id="A1.F26.2.2.2.7.5" class="ltx_tr">
<span id="A1.F26.2.2.2.7.5.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.7.5.1.1" class="ltx_text" style="font-size:80%;">of the gene are central exon deletions that create out-of-frame peptides and progressive degenerative organ waste. A solution is</span></span></span>
<span id="A1.F26.2.2.2.8.6" class="ltx_tr">
<span id="A1.F26.2.2.2.8.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.8.6.1.1" class="ltx_text" style="font-size:80%;">to deliver a Morpholino that recognizes the 5’ end of the out-of-frame exon in pre-mRNA. The molecule prevents binding of the</span></span></span>
<span id="A1.F26.2.2.2.9.7" class="ltx_tr">
<span id="A1.F26.2.2.2.9.7.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.9.7.1.1" class="ltx_text" style="font-size:80%;">spliceosome and creates exon skipping and in-frame joining. Several missing exons are well tolerated by an organism. Which</span></span></span>
<span id="A1.F26.2.2.2.10.8" class="ltx_tr">
<span id="A1.F26.2.2.2.10.8.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.10.8.1.1" class="ltx_text" style="font-size:80%;">structure below is not involved in the proposed therapy?</span></span></span>
<span id="A1.F26.2.2.2.11.9" class="ltx_tr">
<span id="A1.F26.2.2.2.11.9.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.11.9.1.1" class="ltx_text" style="font-size:80%;">(A) antisense</span></span></span>
<span id="A1.F26.2.2.2.12.10" class="ltx_tr">
<span id="A1.F26.2.2.2.12.10.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.12.10.1.1" class="ltx_text" style="font-size:80%;">(B) polyA tail</span></span></span>
<span id="A1.F26.2.2.2.13.11" class="ltx_tr">
<span id="A1.F26.2.2.2.13.11.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.13.11.1.1" class="ltx_text" style="font-size:80%;">(C) R-loops</span></span></span>
<span id="A1.F26.2.2.2.14.12" class="ltx_tr">
<span id="A1.F26.2.2.2.14.12.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.14.12.1.1" class="ltx_text" style="font-size:80%;">(D) lariat</span></span></span>
<span id="A1.F26.2.2.2.15.13" class="ltx_tr">
<span id="A1.F26.2.2.2.15.13.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.15.13.1.1" class="ltx_text" style="font-size:80%;">Question:</span></span></span>
<span id="A1.F26.1.1.1.1" class="ltx_tr">
<span id="A1.F26.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><math id="A1.F26.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="A1.F26.1.1.1.1.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="A1.F26.1.1.1.1.1.m1.1.1" xref="A1.F26.1.1.1.1.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="A1.F26.1.1.1.1.1.m1.1b"><ci id="A1.F26.1.1.1.1.1.m1.1.1.cmml" xref="A1.F26.1.1.1.1.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F26.1.1.1.1.1.m1.1c">\cdots</annotation></semantics></math></span></span>
<span id="A1.F26.2.2.2.2" class="ltx_tr">
<span id="A1.F26.2.2.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><math id="A1.F26.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="A1.F26.2.2.2.2.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="A1.F26.2.2.2.2.1.m1.1.1" xref="A1.F26.2.2.2.2.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="A1.F26.2.2.2.2.1.m1.1b"><ci id="A1.F26.2.2.2.2.1.m1.1.1.cmml" xref="A1.F26.2.2.2.2.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F26.2.2.2.2.1.m1.1c">\cdots</annotation></semantics></math></span></span>
<span id="A1.F26.2.2.2.16.14" class="ltx_tr">
<span id="A1.F26.2.2.2.16.14.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.16.14.1.1" class="ltx_text" style="font-size:80%;">You will be given a multiple choice question with different choices such as (A), (B), (C), (D). Think step by step</span></span></span>
<span id="A1.F26.2.2.2.17.15" class="ltx_tr">
<span id="A1.F26.2.2.2.17.15.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.17.15.1.1" class="ltx_text" style="font-size:80%;">before giving a final answer to this question. Always finish your answer with ’Final Answer: (X)’, where X is the</span></span></span>
<span id="A1.F26.2.2.2.18.16" class="ltx_tr">
<span id="A1.F26.2.2.2.18.16.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F26.2.2.2.18.16.1.1" class="ltx_text" style="font-size:80%;">correct answer choice. If none of the options match, choose the closest option as the final answer.</span></span></span>
</span>
</span>
<span id="A1.F26.2.3" class="ltx_text" style="font-size:80%;"></span></p>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_figure">Figure A.26: </span>Unsupervised ICL Prompt for GPQA. We first start with a preamble saying that we are going to list a number of questions, and then we list the questions. We then give another preamble to specify the output format. As we go to the many-shot setting with hundreds of examples, we only increase the number of questions in the prompt.
</figcaption>
</figure>
<figure id="A1.F27" class="ltx_figure">
<p id="A1.F27.3" class="ltx_p ltx_align_center">

<span id="A1.F27.3.3.3" class="ltx_tabular ltx_align_middle ltx_framed ltx_framed_rectangle">
<span class="ltx_tbody">
<span id="A1.F27.3.3.3.4.1" class="ltx_tr">
<span id="A1.F27.3.3.3.4.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.4.1.1.1" class="ltx_text" style="font-size:80%;">You will be provided source sentences in English to translate in into Kurdish similar to the ones below:</span></span></span>
<span id="A1.F27.3.3.3.5.2" class="ltx_tr">
<span id="A1.F27.3.3.3.5.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.5.2.1.1" class="ltx_text" style="font-size:80%;">English: Its remnants produced showers across most of the islands, though as of yet, no damage or flooding</span></span></span>
<span id="A1.F27.3.3.3.6.3" class="ltx_tr">
<span id="A1.F27.3.3.3.6.3.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.6.3.1.1" class="ltx_text" style="font-size:80%;">has been reported.</span></span></span>
<span id="A1.F27.1.1.1.1" class="ltx_tr">
<span id="A1.F27.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><math id="A1.F27.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="A1.F27.1.1.1.1.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="A1.F27.1.1.1.1.1.m1.1.1" xref="A1.F27.1.1.1.1.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="A1.F27.1.1.1.1.1.m1.1b"><ci id="A1.F27.1.1.1.1.1.m1.1.1.cmml" xref="A1.F27.1.1.1.1.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F27.1.1.1.1.1.m1.1c">\cdots</annotation></semantics></math></span></span>
<span id="A1.F27.2.2.2.2" class="ltx_tr">
<span id="A1.F27.2.2.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left"><math id="A1.F27.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="A1.F27.2.2.2.2.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="A1.F27.2.2.2.2.1.m1.1.1" xref="A1.F27.2.2.2.2.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="A1.F27.2.2.2.2.1.m1.1b"><ci id="A1.F27.2.2.2.2.1.m1.1.1.cmml" xref="A1.F27.2.2.2.2.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F27.2.2.2.2.1.m1.1c">\cdots</annotation></semantics></math></span></span>
<span id="A1.F27.3.3.3.7.4" class="ltx_tr">
<span id="A1.F27.3.3.3.7.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.7.4.1.1" class="ltx_text" style="font-size:80%;">You are an expert translator. I am going to give you one or more example pairs of text snippets where the</span></span></span>
<span id="A1.F27.3.3.3.8.5" class="ltx_tr">
<span id="A1.F27.3.3.3.8.5.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.8.5.1.1" class="ltx_text" style="font-size:80%;">first is in English and the second is a translation of the first snippet into Kurdish. The sentences will be</span></span></span>
<span id="A1.F27.3.3.3.9.6" class="ltx_tr">
<span id="A1.F27.3.3.3.9.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.9.6.1.1" class="ltx_text" style="font-size:80%;">written</span></span></span>
<span id="A1.F27.3.3.3.10.7" class="ltx_tr">
<span id="A1.F27.3.3.3.10.7.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.10.7.1.1" class="ltx_text" style="font-size:80%;">English: &lt;first sentence&gt;</span></span></span>
<span id="A1.F27.3.3.3.11.8" class="ltx_tr">
<span id="A1.F27.3.3.3.11.8.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.11.8.1.1" class="ltx_text" style="font-size:80%;">Kurdish: &lt;translated first sentence&gt;</span></span></span>
<span id="A1.F27.3.3.3.12.9" class="ltx_tr">
<span id="A1.F27.3.3.3.12.9.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.12.9.1.1" class="ltx_text" style="font-size:80%;">After the example pairs, I am going to provide another sentence in English and I want you to translate it</span></span></span>
<span id="A1.F27.3.3.3.13.10" class="ltx_tr">
<span id="A1.F27.3.3.3.13.10.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.13.10.1.1" class="ltx_text" style="font-size:80%;">into Kurdish. Give only the translation, and no extra commentary, formatting, or chattiness. Translate the</span></span></span>
<span id="A1.F27.3.3.3.14.11" class="ltx_tr">
<span id="A1.F27.3.3.3.14.11.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.14.11.1.1" class="ltx_text" style="font-size:80%;">text from English to Kurdish.</span></span></span>
<span id="A1.F27.3.3.3.15.12" class="ltx_tr">
<span id="A1.F27.3.3.3.15.12.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.15.12.1.1" class="ltx_text" style="font-size:80%;">English: Its remnants produced showers across most of the islands, though as of yet, no damage or flooding</span></span></span>
<span id="A1.F27.3.3.3.16.13" class="ltx_tr">
<span id="A1.F27.3.3.3.16.13.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.16.13.1.1" class="ltx_text" style="font-size:80%;">has been reported.</span></span></span>
<span id="A1.F27.3.3.3.17.14" class="ltx_tr">
<span id="A1.F27.3.3.3.17.14.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.17.14.1.1" class="ltx_text" style="font-size:80%;">Kurdish: Li herêma Serengetîyê, Parka Neteweyî ya Serengetî ya Tanzanyayê, Cihê Parastina Ngorongoro û</span></span></span>
<span id="A1.F27.3.3.3.3" class="ltx_tr">
<span id="A1.F27.3.3.3.3.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.3.1.1" class="ltx_text" style="font-size:80%;">Cihê Parastina Gîyanewerên Nêçîrê Maswa û Cihê Parastina Neteweyî ya Masaî Mara ya Kendyayê hene.
English: </span><math id="A1.F27.3.3.3.3.1.m1.1" class="ltx_Math" alttext="\cdots" display="inline"><semantics id="A1.F27.3.3.3.3.1.m1.1a"><mi mathsize="80%" mathvariant="normal" id="A1.F27.3.3.3.3.1.m1.1.1" xref="A1.F27.3.3.3.3.1.m1.1.1.cmml">⋯</mi><annotation-xml encoding="MathML-Content" id="A1.F27.3.3.3.3.1.m1.1b"><ci id="A1.F27.3.3.3.3.1.m1.1.1.cmml" xref="A1.F27.3.3.3.3.1.m1.1.1">⋯</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F27.3.3.3.3.1.m1.1c">\cdots</annotation></semantics></math></span></span>
<span id="A1.F27.3.3.3.18.15" class="ltx_tr">
<span id="A1.F27.3.3.3.18.15.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="A1.F27.3.3.3.18.15.1.1" class="ltx_text" style="font-size:80%;">Kurdish:</span></span></span>
</span>
</span>
<span id="A1.F27.3.4" class="ltx_text" style="font-size:80%;"></span></p>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_figure">Figure A.27: </span>Unsupervised ICL Prompt for the low-resource MT task. We first start with a preamble saying that we are going to list a number of source sentences, and then we list the sentences. We then give another preamble with 1 input-output example to specify the output format. As we go to the many-shot setting with hundreds of examples, we only increase the number of source sentences in the prompt.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A1.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.7 </span>Training GPT-2 from scratch on the sequential parity task</h3>

<figure id="A1.F28" class="ltx_figure"><img src="/html/2404.11018/assets/figures/parity_from_scratch.png" id="A1.F28.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="419" height="209" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure A.28: </span><span id="A1.F28.2.1" class="ltx_text ltx_font_bold">For the sequential parity task, training a transformer from scratch does not meet 8192-shot ICL performance (dashed lines) until 20<math id="A1.F28.2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A1.F28.2.1.m1.1b"><mo id="A1.F28.2.1.m1.1.1" xref="A1.F28.2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A1.F28.2.1.m1.1c"><times id="A1.F28.2.1.m1.1.1.cmml" xref="A1.F28.2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A1.F28.2.1.m1.1d">\times</annotation></semantics></math> the number of examples.</span> We trained two transformers on the sequential parity task (from Section <a href="#S5.SS2" title="5.2 Learning Non-Natural Language Tasks ‣ 5 Analyzing Many-Shot In-Context Learning ‣ Many-Shot In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.2</span></a>). The smaller model was the size of GPT-2 Small, with 12 layers and 768 embedding dimension. The larger model was the size of GPT-2 Medium, with 24 layers and 1024 embedding dimension. We trained using a linear warmup and square root decay schedule, sweeping max learning rate values [1e-5, 5e-5, 1e-4, 5e-4, 1-e3] and num warmup steps [50, 100, 500, 1000, 5000]. The best values for both models were <span id="A1.F28.5.2" class="ltx_text">max_lr</span>=1e-4, <span id="A1.F28.6.3" class="ltx_text">warmup_steps</span>=1000.
</figcaption>
</figure>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2404.11017" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2404.11018" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2404.11018">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2404.11018" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2404.11019" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Sun May  5 20:28:29 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2404.11018
Large language models (LLMs)  excel at few-shot in-context learning (ICL)  – learning from a few examples provided in context at inference, without any weight updates.
Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples – the many-shot regime.
Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks.
While promising, many-shot ICL can be bottlenecked by the available amount of human-generated outputs.
To mitigate this limitation, we explore two new settings: “Reinforced ICL” and “Unsupervised ICL”.
Reinforced ICL uses model-generated chain-of-thought rationales in place of human rationales.
Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific inputs.
We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases and can learn high-dimensional functions with numerical inputs. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream performance.
abstract
Many-Shot In-Context Learning
Abstract
2404.11018
1IntroductionFigure 1:Many-shot vs Few-Shot In-Context Learning(ICL)  across several tasks. Many-shot learning exhibits consistent performance gains over few-shot ICL. This gain is especially dramatic for difficult non-natural language tasks like sequential parity prediction and linear classification. Number of best-performing shots for many-shot ICL are shown inside the bar for each task. For few-shot ICL, we either use typical number of shots used on a benchmark, for example, 4-shot for MATH, or the longest prompt among the ones we tested with less than the GPT-3 context length of 2048 tokens. Reasoning-oriented tasks, namely MATH, GSM8K, BBH, and GPQA uses human-generated chain-of-thought rationales. For translation, we report performance FLORES-MT result on English to Kurdish, summarization uses XLSum, MATH corresponds to the MATH500 test set, and sentiment analysis results are reported with semantically-unrelated labels. See §3, §4, and §5for more details.Large language models (LLMs)  have demonstrated a remarkable ability to performin-context learning(ICL) : they can learn a new task just from input-output examples, also known asshots, which precede a test input presented within the LLM context. However, an LLM’s context window, i.e. the amount of tokenized inputs they can process for each forward inference, limits the number of shots that can be used for ICL, restricting prior research to thefew-shotlearning regime.While relatively under-explored,many-shotlearning – ICL with a large number of shots, for example, hundreds or more – can potentially make LLMs more versatile, adaptable, and ultimately more useful. For example, many shots allow clearer specification of the task, whereas doing so through few shots may be more ambiguous. Exploring many-shot ICL is now feasible, given the recent increase in context windows of publicly available LLMs by at least: from only a few thousand tokens in GPT-3and Llama 2to 1M tokens in Gemini 1.5 Pro.In this paper, we investigate how scaling the number of in-context examples (shots)  effects LLM performance across diverse downstream tasks. To gain broad insights into many-shot learning, we analyze its effects on a variety of widely-used tasks: math problem solving using MATHand GSM8K, question-answering, summarization using XSumand XLSum, algorithmic reasoning, outcome reward modeling, low-resource machine translation, planning, and sentiment analysis. Compared to few-shot learning, many-shot learning leads to significant performance gains across these tasks, using several hundreds or thousands of shots with Gemini 1.5 Pro (Figure 1) . Furthermore, maximum performance is often achieved only once the number of shots reaches up tohundreds of thousandsof tokens (Figure 2) .While many-shot learning holds significant promise, it can be constrained by the need for high-quality, human-generated outputs. This limitation is amplified in complex reasoning tasks, such as GPQA. To overcome this, we introducereinforcedICL andunsupervisedICL.
Inspired by the efficacy of model-generated solutions for fine-tuning, Reinforced ICL involves replacing human-written rationales with model-generated ones, filtered via answer correctness, for in-context learning. Inspired by task-recognition view of ICL, we also introduce Unsupervised ICL where we prompt the model with only problems instead of problem-solution pairs. On reasoning and problem-solving tasks, such as MATH and GPQA, we find that both reinforced and unsupervised ICL with many-shots can be more effective than few-shot ICL with human-generated rationales.Finally, we empirically study how the learning dynamics of in-context learning changes from few-shot to the many-shot regime. We find that with sufficient examples, ICL can overcome pre-training biases and solve high-dimensional prediction tasks with numerical inputs, namely sequential parity prediction and linear classification. This suggests the potential of many-shot learning to adapt to new tasks and domains that might be misaligned with an LLM’s training data. Surprisingly, the order of examples can significantly influence ICL performance, even in the many-shot setting. Furthermore, we discover that the widely-used next-token prediction loss may not reliably predict ICL performance on problem-solving and reasoning tasks.Our key contributions are as follows:•Scaling In-Context Learning (ICL) (§3) : We systematically evaluate LLM performance at different scales of in-context examples for a wide range of tasks. Our results indicate significant performance leaps for LLMs transitioning from few-shot to many-shot learning regime.•Reinforced and Unsupervised ICL(§4) : We find that either using model-generated rationales or only problems can reduce the dependence of many-shot learning on human-generated data.•Analysing ICL(§5) : We find that many-shot ICL can overcome pre-training biases and learn non-natural language prediction tasks, where few-shot ICL struggles.Figure 2:Context Lengthfor best-performing and the maximum number of shots tested for each task. The horizontal dashed line shows the context length of GPT-3 (2048 tokens) , which is representative of typical few-shot prompts tested in the LLM literature. For several tasks, we observed the best-performing shots correspond to the maximum number of shots we tested, which was often limited by the number of available examples for in-context learning. On some tasks (e.g., code verifier, planning) , we did observe slight performance deterioration beyond a certain number of shots.
introduction
Many-Shot In-Context Learning
1Introduction
2404.11018
2Related WorkScaling in-context learningreported improved performance as you increase the number of examples used for in-context learning in large language models, and later works corroborated this finding. However, very few works have explored using a large number of examples (1000 or above)  in the prompt. This is likely due to the fact the context lengths in large language models have been quite limited until recently. One closely related work to ours is from, who scale the number of examples for in-context learning to 2000. However,use a custom model architectureto achieve long context lengths, and only evaluate models of up to 1.3B parameters, which is several orders of magnitude smaller than state-of-the-art language models, and are ineffective for complex tasks, such as GPQA.Concurrently to our work,used many-shot prompting (upto 256 shots)  to jailbreak language models. In our work, we focus on a much wider range of tasks, use a lot more examples (up to 8192 shots)  and use models with much longer context lengths (up to 1M tokens) . Also, we explore mitigations for needing many human-generated examples with many-shot ICL. Furthermore, whileuse many-shot learning to override preferences learned during RLHF phase to elicit the biases stemming from pretraining, our results in §5.1demonstrate that we can also override pretraining biases themselves.Long-context scaling lawsPrior workshave reported smaller next-token prediction loss with longer contexts, whichalso show using theoretical analysis. Our findings confirm this trend for even longer context lengths, but our analysis reveals some of the limitations of using next-token prediction loss as a metric for evaluating long-context performance, as next-token prediction loss continues to go down even as overall performance plateaus.Learning from self-generated dataNumerous recent workspropose fine-tuning language models on self-generated data to improve performance. Their approach consists of (1)  generate samples from the model and filter them using binary feedback, (2)  fine-tune the model on these samples, and (3)  repeat this process a few times. In this work, we extend this idea to the domain of in-context learning, and study the efficacy of Reinforced ICL in reasoning and problem-solving domains.Self-generated data and in-context learningexplore performing in-context learning using self-generated data for classification problems. Their approach consists of generating demonstrations conditioned on the test input for each possible class label, and including these demonstrations in the context when performing the final prediction. In contrast, our approach generates data using the same procedure as, does not require access to the test input for data generation, and can be applied to any problem for which we can obtain reliable reward signals (not just classification problems) .Learning Input-Output Relationships with ICLNumerous workshave investigated whether LLMs truly learn input-output relationships during in-context learning.found that replacing the ground truth labels in in-context examples with random labels barely effected final performance. Further investigations byandfound that this finding does not necessarily hold across tasks and model sizes. In particular,showed that LLMs can indeed learn input-output relationships via in-context learning, but require more examples in order to do so well. In our work, we extrapolate the trend found in those works to much longer context lengths, showing that pre-training biases can be mostly overcome given enough training examples.
body
Many-Shot In-Context Learning
2Related Work
2404.11018
3Scaling In-Context LearningIn-context learning (ICL)  gives LLMs the ability to learn new tasks from examples provided only at inference time. During ICL, the LLM receives a prompt containing a set of input-output examples, also calledshots, that illustrate the desired task. At the end of the prompt, we append a test input and allow the LM to make a prediction just by conditioning on the prompt and predicting the next tokens auto-regressively.
Contrary to task-specific fine-tuning, ICL does not require optimizing any model parameters, allowing LLMs to perform a variety of tasks at inference.Many-shot ICLRecent increase in context windows of LLMs allow using many more shots for ICL than typically used. Compared to few-shot ICL, many-shot learning can use larger fractions of available fine-tuning datasets. This could make task-specific fine-tuning less essential or, in some cases, even unnecessary, allowing LLMs to tackle a wider range of tasks without specialization. Furthermore, exposure to diverse ICL examples can potentially lead to better generalization, the ability to handle more complex problems than what is possible with few-shot ICL, and greater control over model outputs, potentially reducing biases stemming from pre-training.Evaluation detailsWe use the Gemini 1.5 Pro111This corresponds to original version in the Gemini 1.5 Tech Report, released in February 2024. We note that the Gemini 1.5 Pro API now serves a newer version starting from April 2024.MoE model with 1 million token context length, the largest publicly available so far. Unless specified otherwise, we use greedy decoding for evaluation. For reliable results, we randomly sample in-context examples for each-shot prompt multiple times using multiple random seeds and report average performance across seeds, along with some visualization for performance on individual seeds. To ensure that using more shots provides additional information, any-shot prompt in our setup includes all in-context examples from prompts with less thanexamples.3.1Machine Translation (MT) To evaluate how performance scales as we increase the number of in-context examples, we consider machine translation from English to a low-resource target language, where knowledge learned during pre-training may not be sufficient. Following the analysis of, we use the target languages with the largest gap reported between LLMs and Google Translate, namely Tamil and Kurdish, from FLORES-200 MT benchmark. We modify the default 1-shot MT prompt fromto include multiple translation pairs as shots from the FLORES dev split (containing 997 examples) . We evaluate performance on the first 150 sentences from the test set using chRF, a standard MT metric based on character and word-gram overlap between model-generated and reference translation. SeeFigure 3for results.Similar to, we observed marginal gains in the few-shot regime from 1-shot to 10-shot. However, when using the entire dev set for many-shot ICL, we observe improvements of 4.5% on Kurdish and 1.5% on Tamil, relative to the 1-shot Gemini prompt. Notably, these many-shot results establish the new-state-of-art for these language pairs, outperforming Google Translate.Figure 3:Machine Translation(MT) . Test Performance improves almost monotonically as we increase the number of MT pairs provided as in-context examples during inference. In both cases, Gemini 1.5 Prooutperforms Google Translate, which obtains a chRF of 40% on Kurdish and 56% on Tamil. In terms of context length, 997-shot prompt corresponds to 85.3K and 95.3K tokens for translation to Kurdish and Tamil respectively. See an example prompt inFigure A.18. These results complement the Kalamang translation with Gemini 1.5from a single book.3.2SummarizationWe now consider abstractive summarization, which tests the comprehension ability of LLMs to capture essence of the text. Scaling examples for ICL reveals how effectively the model can generalize this ability across different kinds of topics, a realistic scenario where adaptability is crucial. Our evaluation leverages the XSum task from the GEM benchmark. Using XSum development set examples (news articles and summaries)  for many-shot ICL, we also investigate generalization capabilities to XLSum. To assess performance, we report performance on the first 150 articles in the test set using ROUGE-L, which measures the longest common subsequence between reference and generated summaries.As depicted inFigure 4, many-shot ICL achieves performance remarkably close to specialized summarization models fine-tuned on XSum (PEGASUS)  and XLSum (mT5) . Notably, XSum performance declines with more than 50 in-context examples. We surprisingly observed the many-shot prompted model occasionally generating summaries with fabricated dates and times (§A.2) , despite the absence of such data in the in-context summaries. Nonetheless, performance on XLSum monotonically improves with more shots, demonstrating positive transfer from many-shot learning to a related task.Figure 4:Summarization. As we increase the number of (news article, summary)  pairs from XSum dev set as in-context examples, XSum performance improves up to 50 shots and then deteriorates. In contrast, XLSum performance typically improves with more shots from XSum. The 500-shot prompt corresponds to 205K tokens. PEGASUSand mT5are specialized models fine-tuned for summarization. See an example prompt inFigure A.19.3.3Planning: Logistics DomainFigure 5:In-context Planning.Success rate quickly improves with up to 10 shots (37K tokens) , followed by saturation up to 400 shots and a sudden performance jump at 800 shots.
As a reference, we report 1-shot GPT-4 results from.While LLMs have demonstrated remarkable reasoning abilities, their ability to perform commonsense planning seems to be limited. To this end, we verify whether many-shot ICL can improve their ability to generate simple plans on the Logistics domain, a widely used benchmark. The objective in this domain is to transport packages within cities via trucks, and between cities via airplanes.
We generate a set of planning problems with 2-3 cities, 1-2 packages, 1 truck and airplane per city using a formal planning language (PDDL)  generator, resulting in 1.3K problems for learning and 600 for evaluation.
To compute optimal solutions for each problem, we use the Fast-Downward planner. SeeFigure A.20for an example 1-shot prompt.As shown inFigure 5, in-context learning achieves significant improvement in success rate with increasing numbers of shots. While far from state-of-the-art planning approaches (e.g., Fast-Downward) , our results demonstrate the potential of many-shot ICL to improve the commonsense planning abilities of LLMs.3.4Reward Modelling: Learning Code Verifiers In-ContextA standard approach to improve LLM reasoning is to use test-time verification. Specifically, an LLM generates multiple candidate solutions for a given problem and a learned verifier, also known as anoutcome rewardmodel, ranks these solutions and selects the best one. Such verifiers are typically obtained by fine-tuning LLMs to predict solution correctness via binary classification. Here, we focus on learning such verifiers in-context.To create in-context examples for code verification, we utilize correct and incorrect code-based solutions sampled from Gemini 1.0 Proon the GSM8K train set. In the prompt, each (problem, solution)  pair is appended with the question “Is the solution correct?” followed by theYesorNotoken according to ground truth correctness. At inference, we modify each test (problem, solution)  pair in the same way and record the logit of theYesandNotokens (denoted by,) . To compute the verifier score, we use thenormalizedprobability of theYestoken:. To evaluate verifier performance, we perform best-of-4 selection using the verifier score on 200 problems from GSM8K test set with Gemini 1.0 solutions.Figure 6:Learning Verifiers In-Contextfor checking correctness of GSM8K code solutions. Error bars denotes standard error of mean over 3 seeds. See FigureA.22for a 2-shot prompt example.Best-of-N accuracy. (Left)  Average accuracy of top-ranked code solutions (among 4 solutions)  based on the verifier score on 200 GSM8K test problems. Best-of-4 selection with 128-shot bridges the gap between Pass@1 accuracy of 77.25% and Pass@4 accuracy of 90% with Gemini 1.0 Pro model.Verifier Confidence. (Right)  Conditional Probabilities of theYestokenfrom the verifier, averaged over all correct and incorrect solutions on test problems.As shown inFigure 6(left) , best-of-4 accuracy with the few-shot prompted verifier significantly improves above pass@1 accuracy with 16 or more in-context examples. Along with an accuracy improvement, the probabilities of theYestoken conditioned on ground-truth correct and incorrect solutions separate with increasing the number of shots up to 256, as shown inFigure 6(right) . Overall, these results show a proof-of-concept that the Gemini model becomes better at verifying correctness of solutions with many-shot ICL.
body
Many-Shot In-Context Learning
3Scaling In-Context Learning
2404.11018
4Many-shot Learning without Human-Written RationalesMany-shot ICL could potentially be limited by the availability of high-quality human-generated rationales or demonstrations. This is particularly challenging for complex reasoning tasks, such as GPQA, where human-generated rationales require significant resources and expert knowledge. In this work, we explore two approaches for addressing this issue.Reinforced ICLRecent workhas introduced Reinforced Self-Training (ReST) . In particular,demonstrated that fine-tuning using model-generated rationales can be more effective than human-generated rationales for problem-solving tasks, and can be viewed as applying expectation-maximization for reinforcement
learning.
We introduce Reinforced ICL, where we use model-generated rationales for in-context learning.
To obtain model-generated rationales, we use a few-shot or zero-shot chain-of-thought prompt as a starting point to sample multiple rationales for each training problem.
Then, we select rationales that obtain the correct final answer (we assume access to ground truth final answers) , and arrange them into in-context examples containing (problem, rationale)  pairs.One potential issue with model-generated rationales is that of false positives: it is possible for an incorrect reasoning chain to lead to the correct final answer, and fine-tuning or prompting using such a reasoning chain would typically harm performance.
Nevertheless, as we discuss in later sections, we often find model-generated rationales to be at least as effective human-written rationales.Unsupervised ICLMotivated by encouraging results with Reinforced ICL, we go one step further: what if we removed rationales from the many-shot prompt altogether, and prompt the model only with inputs (e.g., problems) ?
Specifically, the Unsupervised ICL prompt consists of: 1)  a preamble, such as, “You will be provided questions similar to the ones below:”, 2)  a list of unsolved inputs or problems, and 3)  a zero-shot instruction or a few-shot prompt with outputs for the desired output format. See AppendixA.6for the exact prompts we use.One hypothesis for how many-shot unsupervised ICL might surpass few-shot learning with human demonstrations is that, when the LLM already possesses the required knowledge to solve a task, any information inserted in the prompt that can narrow down what knowledge is needed for the task becomes helpful. This would be consistent with the view that ICL simply “locates” latent concepts (e.g., math problem-solving)  the LLM acquired during pre-training. As such, any of the prompt components – inputs, outputs, and their mapping – can help locate such concepts. While Unsupervised ICL is broadly applicable, it may not perform well when the outputs are critical for specifying the task, for example, seeFigure A.17for negative results low-resource machine translation.4.1Problem-solving: Hendrycks MATH & GSM8KFigure 7:Many-shot Reinforced and Unsupervised ICL for problem-solvinggenerally outperform ICL with ground-truth MATH solutions.MATH. (Left)  The bar plots depict the average performance across five random seeds on the MATH500 test set. Each random seed (denoted by the dots)  corresponds to a different subset of problems along with ground truth or model-generated solutions (if any)  in the prompt.Transfer to GSM8K. (Right)  We see that the prompt obtained from MATH transfers well to the GSM8K test split containing 500 problems. Our results with many-shot ICL outperform the 4-shot Minerva prompt, which obtains a test accuracy of 55.7% on MATH500 and 90.6% on GSM8K.We evaluate Reinforced and Unsupervised ICL on the Hendrycks MATH dataset, which consists of challenging high school competition-level mathematics problems. We use the MATH500 test set fromto report performance, and our 4-shot MATH prompt for data generation can be found in the Appendix (FigureA.23) . For Unsupervised ICL, we append this 4-shot prompt after the unsolved problems (see FigureA.25for complete prompt) . For comparison, we also evaluate ICL with human-written solutions (ground-truth)  from the MATH training set.Our results are shown in the Figure7(left) . On MATH500, both Reinforced and Unsupervised ICL outperforms ICL with ground-truth solutions in both the few-shot and many-shot regime. For ICL, we observe that the performance improves with more examples in the prompt up to a point, and then declines (with the peak being at about 125 examples) . Performance for Reinforced ICL also improves with the number of examples, and reaches a plateau at around 25 examples (while being about 5% higher than ICL) , and unlike ICL, we don’t see a significant drop in performance even for a very large number of examples in the context. Notably, many-shot ICL achieves comparable or superior performance when using only problems compared to using problems with solutions. This suggests solutions may be redundant for eliciting problem-solving via in-context learning on this domain, potentially due to the math-related data seen during pretraining.Can many-shot ICL enable out-of-distribution generalization?found that fine-tuning a model on model-generated solutions from MATH resulted in improved test performance on GSM8K, which has a different distribution of problems than MATH. Here, we investigate whether many-shot ICL also improves transfer performance on GSM8K, indicating an improvement in general problem-solving abilities from in-context learning. Our results in Figure7(right)  show that this is indeed the case – Reinforced ICL with MATH prompts excels on GSM8K, outperforming ICL with ground truth MATH solutions as well as Unsupervised ICL in the many-shot setting with at least 25 shots. This indicates that model-generated solutionscanenable better generalization than just using problems or combining them with ground-truth solutions for ICL.4.2Question Answering: Google-Proof QA (GPQA) Figure 8:Many-shot Reinforced and Unsupervised ICL for GPQA. The baseline zero-shot prompt, which is used for generating rationales for Reinforced ICL and appended to the prompt for Unsupervised ICL, obtains a performance of 38.8%.
The average test accuracy with 125-shot prompt with both ground-truth or model-generated rationales surpass the 40.4% obtained by Claude-3 Sonnet. As we vary the number of shots, while Unsupervised ICL matches or outperforms the zero-shot prompt, Reinforced ICL consistently outperforms it.GPQAis a multiple-choice QA benchmark, with difficult questions focused on graduate-level reasoning in biology, physics, and chemistry. Following Claude-3, we use the diamond split (198 problems)  for evaluation. This split focuses on questions where domain experts agree but experts in other domains struggle despite extended effort and internet access. Remaining 250 questions in non-diamond split are used for many-shot ICL with and without human-written rationales. For Reinforced ICL, we use a zero-shot prompt (Figure A.21)  to generate multiple rationales on the non-diamond split, solving 129 problems. We also append this zero-shot prompt after the GPQA problems for specifying output format for Unsupervised ICL.As shown inFigure 8, average test accuracy with ground-truth rationales improves substantially from 5 shots to 125 shots, with the best-performing 125-shot prompt nearly matching the accuracy of the state-of-the-art Claude-3 Opus. However, we do observe a performance degradation with 250 shots. Moreover, Reinforced ICL results indicate that model-generated rationales on GPQA seem to be better than ground-truth rationales up to 25 shots, while resulting in similar performance with more shots. Additionally, Unsupervised ICL does not follow any systematic trend: it sometimes performs better ICL with ground-truth rationales depending on the number of shots, but generally underperforms Reinforced ICL. As noted in, GPQA is a small evaluation dataset and has an inherent higher variance across different runs, which might explain the non-systematic trends.4.3Algorithmic and Symbolic Reasoning: Big-Bench HardFigure 9:BIG-Bench Hard. Reinforced ICL’s performance over number of shots, averaged across five random seeds. The error bars denote standard deviation. Reinforced ICL outperforms the human-written chain-of-thought prompt for all tasks. Averaged across tasks, chain-of-thought prompting using human-written prompts gets a success rate of 72.1%, while Reinforced ICL gets 83%.We now evaluate Reinforced ICL on BIG-Bench Hard, a suite of challenging algorithmic reasoning tasks. To reduce the impact of false positives, we select 8 tasks out of 23 in BIG-Bench Hard for which the likelihood of getting a false positive is low: either the answer string is long, or the number of options for each question is large (at least 6) . We use the standard 3-shot CoT prompt fromto sample 10 rationales per problem from a training set of 150 problem at a temperature of 1.0. We filter the rationales based on final answer correctness and arrange them into prompts containing 3 to 100 (problem, rationale)  pairs. We evaluate test performance on a held-out set of 100 problems.As shown in Figure9, Reinforced ICL strongly outperforms the standard 3-shot CoT prompt for almost all tasks, and the performance generally improves monotonically with the number of prompts for 7 out of 8 tasks.
For a few tasks, Reinforced ICL outperforms the human-written 3-shot prompt even in the 3-shot setting. This result suggests that model-generated rationales cansometimes outperform human-written rationales even when controlling for the amount of data, mirroring the results reported byfor fine-tuning.
body
Many-Shot In-Context Learning
4Many-shot Learning without Human-Written Rationales
2404.11018
5Analyzing Many-Shot In-Context LearningIn this section, we study how ICL behavior changes from few-shot to the many-shot regime.5.1Overcoming Pre-training Biases with Many-Shot ICLFigure 10:Overcoming Pre-Training Bias with Many-Shot ICL.(Left) Many-shot ICL overcomes label flips: Test accuracy for sentiment analysis typically improves with more training shots. Flipped and abstract labels eventually approaching the performance of default labels. (Right) Confidence shift in overcoming bias. For flipped and abstract labels, model confidence in its predicted sentiment labels initially drops, then sharply increases with more training shots to similar value, suggesting a period of overcoming pre-training bias. See §5.1for more details.While LLMs demonstrate in-context learning of novel tasks,suggest that ICL may have difficulty unlearning biases derived from pre-training data. Their experiments, however, focused mainly on few-shot ICL due to LLM context length limitations.
Here, we revisit their study using many-shot ICL on the Financial PhraseBank (FP)  sentiment analysis dataset. Like, we study label relationships that challenge pre-training preferences:•Flipped Labels: Default labels are rotated, that is, [‘negative’, ‘neutral’, ‘positive’] becomes [ ‘neutral’, ‘positive’, ‘negative’]. This conflicts with sentiment biases an LLM might have learned.•Abstract Labels: Semantically-Unrelated labels [‘A’, ‘B’, ‘C’] are used,
removing any pre-existing sentiment association.For ICL shots, we sample examples from the validation set (with replaced labels)  to exhibit the input-label relationship. We report the results inFigure 10(left) . With few shots, test accuracy with replacement labels is much lower than with default labels. This suggests that with few-shot ICL, the model struggles to overcome its pre-existing biases from pre-training. However, as the number of shots increases, performance on flipped and abstract labels dramatically improves, approaching that of default labels.We also report model confidence on predicted labels inFigure 10(right) . For default labels, confidence in predicted labels steadily increases with more shots. In contrast, for flipped labels, confidence initially drops then sharply increases before reaching a plateau, suggesting a period of overcoming pre-training bias. We posit that the initial drop in performance and confidence in the few-shot regime from 4 to 8 shots may be attributed to the “early ascent” phenomenon: a small number of shots may lead to the retrieval of an incorrect skill, which eventually diminishes as task learning takes effect with more shots. Overall, these results indicates that with sufficient in-context examples, LLMscanovercome pre-training biases.5.2Learning Non-Natural Language TasksWe now test many-shot ICL’s ability to learn abstract mathematical functions with numerical inputs.
We focus on parity functions and high-dimensional linear classification; these tasks are appealing because their synthetic nature let us test the generality of ICL and determine its applicability beyond the linguistic domain.Figure 11:In-Context Linear Classification. Test accuracy for 16, 32 and 64 dimensional linear classification problems, averaged across 5 randomly-generated datasets with 25 points per class for each dataset (250 evaluation points total) . As we increase the number of shots, the accuracy improves and approximately tracks the performance of the nearest-neighbor baseline trained from scratch on the same data. We use the default implementation of-nearest neighbours (with)  from scikit-learn. SeeFigure A.24for an example prompt.Linear Classification in High DimensionsWe first consider linear classification in high dimensions to test the limits of many-shot ICL. Following the setup from, we create datasets with-dimensional inputs vectors and their corresponding labels, where each dimension is a random integer in. For each dataset, we randomly sample another-dimensional vector as the decision boundary and a decision threshold (§A.4) . We then provideN-dimensional points above this threshold andpoints below that same
threshold as in-context exemplars, and the model must determine whether unseen N-dimensional points are above or below the threshold (we do not tell the model the equation or the threshold) . Whileused only 16 shots per class, we scale ICL up to 2048 shots per class. As shown in Figure11, many-shot ICL substantially outperforms random-chance accuracy and nearly matches the performance of a strong baseline, namely-nearest neighbors. Whileshots per class perform best when, we observe slight accuracy decrease beyondshots for higher values of(Figure 11C, R) .Sequential ParityParity is a fundamental Boolean function that determines if a binary input sequence contains an even or odd number of 1s. It’s computed by applying the XOR ()  operation to all bits in the sequence. Despite their power, transformers trained specifically for in-context learning, struggle to in-context learn the Parity function over 20-digit sequences, achieving near chance-level accuracy. In this work, we evaluate how well ICL performs with a pretrained LLM to learn the sequential parity function, whereand. We report the results inFigure 12. We see consistent improvement in test accuracy as we increase the number of shots 8192. Performance surpasses a GPT-2 Medium sized transformertrained from scratch on 20more input-output examples (with no repeated examples; details in AppendixA.7) .Input: 1 0 1 1 0 0 0 1 1 1 0 0 0 0 1 0 0 1 1 1Label: Odd Odd Even Odd Odd Odd Odd EvenOdd Even Even Even Even Even Odd Odd OddEven Odd EvenInput: 0 1 1 0 0 1 1 0 1 1 0 0 1 1 0 0 0 1 1 1Label:Figure 12:Learning Sequential Parity Function In-context. We report test accuracy over 200 unseen inputs, averaged across 3 seeds. Error bars denote standard error of the mean.Task Prompt. (Left)  Example prompt with input and output labels of the 20-digit Sequential Parity Function.Test accuracy(Right)  Many-shot ICL performance improves almost monotonically with the number of shots, surpassing performance of GPT-2 Medium sized transformer trained from scratch for 1 epoch on 20more data.5.3Is Many-Shot ICL Sensitive to Example Ordering?In few-shot in-context learning (ICL) , the order of examples within the prompt can significantly impact model performance.
Here, we investigate whether such sensitivity to prompt ordering observed in few-shot ICL persists in many-shot scenarios, which remainsFigure 13:Many-Shot Sensitivity To Example Ordering.
Each colored data point represents a different random ordering of 50 in-context examples provided to Gemini 1.5 Pro.largely unexplored. Specifically, we evaluate ten different random orderings of fixed 50 in-context examples from MATH training split and evaluate performance on the held-out MATH500 test set.AsFigure 13reveals, performance varies significantly across different subareas in MATH500. Strikingly, an ordering that that excels in one subarea may perform poorly in another, for example, the best Geometry ordering yields weak results on Number Theory. This fluctuation results in a smaller variation in average performance compared to individual subareas. Overall, these findings highlight a key challenge in ensuring reliable results with many-shot ICL for long-context models.5.4NLL May Not Be Predictive of ICL PerformanceFigure 14:Negative Log-Likelihoodas a function of number of shots We plot negative log-likelihood (NLL)  on ground truth test set solutions for GPQA, MATH and GSM8K. For GPQA and MATH, questions for Reinforced ICL and Unsupervised ICL comes from the training splits of those datasets. We study GSM8K in the transfer setting, i.e. questions for Reinforced ICL and Unsupervised ICL come from MATH. Note that the absolute NLL numbers for ICL and Reinforced ICL are not directly comparable to Unsupervised ICL, since they employ different prompt formats.Prior workshave found that the negative log-likelihood (NLL)  for ground-truth test outputs, which corresponds to next-token prediction loss, decreases predictably as the context length increases. We confirm this finding for GPQA, Hendrycks MATH and GSM8K with many-shot in-context learning, and report our results in Figure14. However, we note that NLL trends are not a strong predictor for downstream task performance. For example, the success rate for both MATH and GPQA with ICL decreases after 125 shots (Figure7,8) , but we do not observe a corresponding increase in the negative log-likelihood in Figure14.We also plot negative log-likelihood curves for Reinforced and Unsupervised ICL, and find them to generally have a smaller slope when compared to supervised ICL. Furthermore, we observe that NLL for ICL with ground-truth outputs is much lower compared to model-generated outputs, depsite model-generated outputs often resulting in better performance than ground-truth outputs. These results imply that NLL may not be predictive of downstream ICL performance when using prompts that are out-of-distribution with respect to the test set.In the GSM8K transfer setting (using MATH problems and solutions to score GSM8K solutions) , we see that the change in NLL is close to nil. Once again, these trends do not necessarily reflect the final performance observed in Figure7, as GSM8K performance continues to increase with more examples in the transfer setting. Finally, while MATH performance for Reinforced and Unsupervised ICL saturates around 25-shot prompts, the NLL continues to decrease with more shots.Our results lead us to conclude that negative log-likelihood is not a reliable proxy when attempting to predict final performance for problem-solving domains. This makes intuitive sense: for any given problem, there are a large number of potentially correct chain-of-thought paths that the model can take, and calculating the log-likelihood on only one such path may not provide a clear picture for overall model capability. We also explore computing negative log-likelihood on a diverse set of model-generated outputs on MATH, and our findings are presented in AppendixA.1.
body
Many-Shot In-Context Learning
5Analyzing Many-Shot In-Context Learning
2404.11018
6ConclusionWe investigated many-shot in-context learning.
We found significant gains in performance when going from few-shot to many-shot in a wide range of tasks, including translation, summarization, planning, reward modeling, mathematical problem solving, scientific question-answering, and algorithmic reasoning.
To overcome the challenges of obtaining a large number of high-quality human-written rationales for many-shot learning, we introduced two new in-context learning regimes: Reinforced ICL and Unsupervised ICL.
We found that, for problem-solving domains where human-generated rationales are expensive to obtain, Reinforced and Unsupervised ICL can obtain strong performance when compared to ICL with human data. We concluded with a set of analysis experiments showing that many-shot learning can overcome pre-training biases, allows learning non-natural language tasks typically difficult for LLMs with few-shot learning, and explored negative log-likelihood trends with respect to context length in the many-shot setting.LimitationsAn important limitation of our work is that it uses a single model for all its experiments, the Gemini 1.5 Pro.
Future work should focus on evaluating the many-shot in-context learning abilities of a wide range of long context models, as they become available.
Many-shot performance can likely serve as a valuable metric for evaluating the quality of long-context models, going beyond the needle-in-a-haystack testthat is currently popular for this purpose.Another limitation of our work is that we don’t completely understand why performance can sometimes degrades with more examples in the prompt (for example, for MATH) .
Our analysis found that negative log-likelihood trends are insufficient to explain this degradation, and future work should focus on investigating new research directions to shed light on the matter.
body
Many-Shot In-Context Learning
6Conclusion
2404.11018
AcknowledgementsWe would like to thank Gheorghe Comanici for reviewing an early draft of this work. We are also grateful to Doina Precup, Aviral Kumar, Dale Schuurmans, Ankit Anand, Ross Goroshin, Urvashi Singh, and Daniel Toyoma for helpful discussions.
body
Many-Shot In-Context Learning
Acknowledgements
2404.11018
Contribution StatementRA initiated the project, ran majority of the many-shot experiments and analysis, came up with reinforced ICL, on-boarded collaborators, wrote the initial draft. AS contributed initial infra for experiments on MATH and GSM8K, ran BBH experiments, conducted NLL analysis on problem-solving tasks, and wrote several sections of the paper.LZ contributed results for in-context verifier. BB contributed the planning logistics task. AA helped with GPQA, SC contributed the baseline for parity task and both helped edit the paper. AF and HL provided feedback on an early draft. HL also suggested the unsupervised ICL experiments. Others were involved in project discussions and minor edits to the paper.
conclusion
Many-Shot In-Context Learning
Contribution Statement