<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.06023] Distilling System 2 into System 1</title><meta property="og:description" content="Large language models (LLMs) can spend extra compute during inference to generate intermediate thoughts, which helps to produce better final responses.
Since Chain-of-Thought (Wei et al., 2022),
many such System 2 tech…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Distilling System 2 into System 1">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Distilling System 2 into System 1">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.06023">

<!--Generated on Mon Aug  5 18:47:15 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">Distilling System 2 into System 1</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ping Yu        Jing Xu        Jason Weston        Ilia Kulikov
<br class="ltx_break">
<br class="ltx_break">Meta FAIR
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id1.id1" class="ltx_p">Large language models (LLMs) can spend extra compute during inference to generate intermediate thoughts, which helps to produce better final responses.
Since Chain-of-Thought <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite>,
many such <span id="id1.id1.1" class="ltx_text ltx_font_italic">System 2</span> techniques have been proposed such as Rephrase and Respond <cite class="ltx_cite ltx_citemacro_citep">(Deng et al., <a href="#bib.bib12" title="" class="ltx_ref">2023a</a>)</cite>, System 2 Attention <cite class="ltx_cite ltx_citemacro_citep">(Weston and Sukhbaatar, <a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite> and Branch-Solve-Merge <cite class="ltx_cite ltx_citemacro_citep">(Saha et al., <a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite>. In this work we investigate self-supervised methods to “compile” (distill) higher quality outputs from System 2 techniques back into LLM generations <span id="id1.id1.2" class="ltx_text ltx_font_italic">without</span> intermediate reasoning token sequences, as this reasoning has been distilled into <span id="id1.id1.3" class="ltx_text ltx_font_italic">System 1</span>. We show that several such techniques can be successfully distilled, resulting in improved results compared to the original System 1 performance, and with less inference cost than System 2. We posit that System 2 distillation will be an important feature of future continually learning AI systems, enabling them to focus System 2 capabilities
on the reasoning tasks that they cannot yet do well.</p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<div id="p1.1" class="ltx_block ltx_align_bottom">
<p id="p1.1.1" class="ltx_p"><span id="p1.1.1.1" class="ltx_text ltx_font_bold">Distilling System 2 into System 1</span></p>
<br class="ltx_break ltx_centering">
<p id="p1.1.2" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p1.1.2.1" class="ltx_text ltx_inline-block" style="width:0.0pt;">
<span id="p1.1.2.1.1" class="ltx_tabular ltx_align_top">
<span id="p1.1.2.1.1.1" class="ltx_tr">
<span id="p1.1.2.1.1.1.1" class="ltx_td ltx_align_center"><span id="p1.1.2.1.1.1.1.1" class="ltx_text ltx_font_bold">Ping Yu        Jing Xu        Jason Weston        Ilia Kulikov</span></span></span>
<span id="p1.1.2.1.1.2" class="ltx_tr">
<span id="p1.1.2.1.1.2.1" class="ltx_td ltx_align_center">Meta FAIR</span></span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Generating intermediate thoughts allows a model (or human!) to reason and plan in order to successfully complete a task or respond to an instruction.
We refer to such deliberate thinking as System 2 reasoning, following its description for humans in <cite class="ltx_cite ltx_citemacro_citet">Sloman (<a href="#bib.bib33" title="" class="ltx_ref">1996</a>); Kahneman (<a href="#bib.bib18" title="" class="ltx_ref">2011</a>)</cite> and later for AI models <cite class="ltx_cite ltx_citemacro_citep">(Bengio, <a href="#bib.bib3" title="" class="ltx_ref">2017</a>; LeCun, <a href="#bib.bib21" title="" class="ltx_ref">2022</a>; Weston and Sukhbaatar, <a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite>. In System 2 reasoning effortful mental activity is exerted, especially in situations where System 1 – more automatic thinking – is likely to make errors.
In standard Large Language Models (LLMs) we thus define <span id="S1.p1.1.1" class="ltx_text ltx_font_italic">System 1</span> as application of the Transformer <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib35" title="" class="ltx_ref">2017</a>)</cite> to directly produce a response given an input, without generation of intermediate tokens.
We define <span id="S1.p1.1.2" class="ltx_text ltx_font_italic">System 2</span> as any approach which generates intermediate tokens, including methods that perform search, or prompt multiple times, before finally generating a response.
A battery of such <span id="S1.p1.1.3" class="ltx_text ltx_font_italic">System 2</span> techniques have been proposed, among them Chain-of-Thought <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite>, Tree-of-Thoughts <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib41" title="" class="ltx_ref">2024</a>)</cite>, Graph-of-Thoughts <cite class="ltx_cite ltx_citemacro_citep">(Besta et al., <a href="#bib.bib4" title="" class="ltx_ref">2024</a>)</cite>, Branch-Solve-Merge <cite class="ltx_cite ltx_citemacro_citep">(Saha et al., <a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite>, System 2 Attention <cite class="ltx_cite ltx_citemacro_citep">(Weston and Sukhbaatar, <a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite>, Rephrase and Respond <cite class="ltx_cite ltx_citemacro_citep">(Deng et al., <a href="#bib.bib12" title="" class="ltx_ref">2023a</a>)</cite> and more.
Many of these methods are shown to produce more accurate results due to this explicit reasoning, but typically do so at much higher inference cost and latency for a response.
Due to the latter, many of these approaches are not used in production systems, which mostly use <span id="S1.p1.1.4" class="ltx_text ltx_font_italic">System 1</span> generations.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.06023/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="352" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span><span id="S1.F1.2.1" class="ltx_text ltx_font_bold">Overview of System 2 Distillation.</span> Filtered training examples are collected by running System 2 approaches
such as Branch-Solve-Merge (BSM)
on unlabeled data, which uses extra compute to produce higher quality outputs. These targets are then distilled into the standard (System 1) LLM.</figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">For a human, the process of learning to transfer a skill from deliberate (System 2)
to automatic (System 1) in psychology
is referred to as <span id="S1.p2.1.1" class="ltx_text ltx_font_italic">automaticity</span>, and the use of <span id="S1.p2.1.2" class="ltx_text ltx_font_italic">procedural memory</span> <cite class="ltx_cite ltx_citemacro_cite">Cohen and Squire (<a href="#bib.bib10" title="" class="ltx_ref">1980</a>)</cite>.
For example, when driving to work for the first time one might typically
expend conscious effort planning and making decisions to get there.
After a driver repeats this route, the driving process becomes “compiled” into the subconscious <cite class="ltx_cite ltx_citemacro_cite">Charlton and Starkey (<a href="#bib.bib6" title="" class="ltx_ref">2013</a>)</cite>. Similarly, playing a sport such as tennis can become “second nature”.
In this work, we explore an analogous technique for AI models.
Our approach performs this compilation, which we refer to as <span id="S1.p2.1.3" class="ltx_text ltx_font_italic">System 2 distillation</span>, in an unsupervised manner given a set of unlabeled examples.
For each example we apply the given System 2 method, and then measure the quality of the prediction in an unsupervised manner. For example, for tasks with unique answers we apply self-consistency <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib37" title="" class="ltx_ref">2022</a>)</cite>, sampling multiple times. For examples where System 2 is consistent enough, we assume this result should be distilled, and add it to the distillation pool.
We then fine-tune System 1 to match the predictions of the System 2 method on the collected pool of examples, but <span id="S1.p2.1.4" class="ltx_text ltx_font_italic">without</span> generating the intermediate steps. Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Distilling System 2 into System 1" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> illustrates the overall process of distilling System 2 into System 1.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">We conduct experiments across 4 different System 2 LLM approaches and 5 different tasks. We find our approach can distill System 2 reasoning into System 1 in a diverse array of settings, sometimes even improving the results over the System 2 teacher.
Moreover, these predictions are now produced at a fraction of the computational cost.
For example, we see successful distillation for tasks involving dealing with biased opinions or irrelevant information (System 2 Attention), clarifying and improving responses in some reasoning tasks (Rephrase and Respond), and for fine-grained evaluation of LLMs (Branch-Solve-Merge). However, we also show that not all tasks can be distilled into System 1, particularly complex math reasoning tasks requiring chain-of-thought. This is also mirrored in humans,
who cannot execute some tasks without deliberate System 2 reasoning <cite class="ltx_cite ltx_citemacro_citep">(Kahneman, <a href="#bib.bib18" title="" class="ltx_ref">2011</a>)</cite>.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related work</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>System 1 and System 2 in Humans</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">In humans, System 1 reasoning is described as being capable of recognizing patterns, making quick judgments, and understanding simple or familiar symbols. For instance, it is used to identify common traffic signs, recognize faces, or associate basic symbols with specific emotions or ideas.
However, for complex problem-solving or for example manipulation of abstract symbols (like algebraic equations or logical statements), System 2 reasoning is deemed necessary <cite class="ltx_cite ltx_citemacro_citep">(Kahneman, <a href="#bib.bib18" title="" class="ltx_ref">2011</a>)</cite>.
In psychology the concept of <span id="S2.SS1.p1.1.1" class="ltx_text ltx_font_italic">automaticity</span> describes behavior that becomes so well-practiced that it can be performed with little to no conscious thought, with an example being driving a familiar route <cite class="ltx_cite ltx_citemacro_cite">Charlton and Starkey (<a href="#bib.bib6" title="" class="ltx_ref">2013</a>)</cite>.
In general, humans are said to use <span id="S2.SS1.p1.1.2" class="ltx_text ltx_font_italic">procedural memory</span> to consolidate a specific task into memory, learning through practice, so that it can be later performed without conscious awareness <cite class="ltx_cite ltx_citemacro_citep">(Cohen and Squire, <a href="#bib.bib10" title="" class="ltx_ref">1980</a>)</cite>.
The concept of <span id="S2.SS1.p1.1.3" class="ltx_text ltx_font_italic">unconscious competence</span> is classified as a later stage of learning. Initially a person recognizes their incompetence, and consciously seeks to learn a skill until they acquire <span id="S2.SS1.p1.1.4" class="ltx_text ltx_font_italic">conscious competence</span>. Finally, the aim is to utilize it without conscious thought when it is said to become, in common language, “second nature” <cite class="ltx_cite ltx_citemacro_citep">(DePhillips et al., <a href="#bib.bib15" title="" class="ltx_ref">1960</a>)</cite>.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>System 1 and System 2 Models</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">We refer to a neural network that outputs a response directly without intermediate outputs as a <span id="S2.SS2.p1.1.1" class="ltx_text ltx_font_italic">System 1 model</span>. Such a network can nevertheless compute intermediate latent representations in its layers before it outputs a response. As these states are represented as vectors they typically encode distributed knowledge, rather than discrete decisions, and have difficulty manipulating
complex symbolic reasoning tasks directly <cite class="ltx_cite ltx_citemacro_citep">(Nye et al., <a href="#bib.bib26" title="" class="ltx_ref">2021</a>; Cobbe et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>; Yu et al., <a href="#bib.bib42" title="" class="ltx_ref">2023</a>; Li et al., <a href="#bib.bib24" title="" class="ltx_ref">2024</a>)</cite>, which is analogous to issues with System 1 reasoning in humans. Nevertheless, a vast array of tasks can be solved with success directly in this manner without intermediate generations <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib29" title="" class="ltx_ref">2019</a>)</cite>.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">Nye et al. (<a href="#bib.bib26" title="" class="ltx_ref">2021</a>)</cite> showed that the same language model that is unable to perform complex multi-step computations can perform those tasks when asked to generate intermediate steps into a “scratchpad” using either few-shot prompting or supervised training.
Chain-of-thought reasoning was shown to be elicited from LLMs even using zero-shot prompting <cite class="ltx_cite ltx_citemacro_citep">(Kojima et al., <a href="#bib.bib19" title="" class="ltx_ref">2022</a>)</cite> as well as by supervised
<cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite> or few-shot <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite> methods.
LLM pretraining allows such reasoning to be built into the model because reasoning steps in discrete symbols (text) are present in the training corpora written by humans.
Such <span id="S2.SS2.p2.1.1" class="ltx_text ltx_font_italic">System 2 model</span> approaches output discrete tokens which is good for making sequential correct logical reasoning steps – but obviously has a downside if the reasoning is generated incorrectly.
An incorrect discrete decision is difficult to recover from, unlike latent vector-based reasoning that might more easily model a distribution.</p>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.1" class="ltx_p">Recently, many approaches have been proposed to execute deeper reasoning using the LLM as part of an inner loop where it generates intermediate outputs, sometimes referred to as
<span id="S2.SS2.p3.1.1" class="ltx_text ltx_font_italic">LLM Programs</span> <cite class="ltx_cite ltx_citemacro_citep">(Schlag et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>)</cite>.
These include subquestion decomposition <cite class="ltx_cite ltx_citemacro_citep">(Perez et al., <a href="#bib.bib27" title="" class="ltx_ref">2020</a>)</cite>,
self-refinement <cite class="ltx_cite ltx_citemacro_citep">(Madaan et al., <a href="#bib.bib25" title="" class="ltx_ref">2024</a>; Weston and Sukhbaatar, <a href="#bib.bib40" title="" class="ltx_ref">2023</a>; Deng et al., <a href="#bib.bib12" title="" class="ltx_ref">2023a</a>)</cite>, self-verification and asking <cite class="ltx_cite ltx_citemacro_citep">(Press et al., <a href="#bib.bib28" title="" class="ltx_ref">2022</a>; Weng et al., <a href="#bib.bib39" title="" class="ltx_ref">2022</a>; Dhuliawala et al., <a href="#bib.bib16" title="" class="ltx_ref">2023</a>)</cite>,
and various search techniques such as Tree-of-Thoughts and others <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib41" title="" class="ltx_ref">2024</a>; Besta et al., <a href="#bib.bib4" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>(Standard) Distillation</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">The concept of distillation is usually applied to taking separate models, a powerful teacher model (or multiple teacher models) and a less powerful student model with separate parameters.
The student model is then trained to mimic the behavior of the teacher(s).
Methods of distillation include training the student to have similar
output distributions <cite class="ltx_cite ltx_citemacro_citep">(Hinton et al., <a href="#bib.bib17" title="" class="ltx_ref">2015</a>)</cite>, layer activations <cite class="ltx_cite ltx_citemacro_citep">(Adriana et al., <a href="#bib.bib1" title="" class="ltx_ref">2015</a>)</cite> or derivatives of the target teacher outputs <cite class="ltx_cite ltx_citemacro_citep">(Czarnecki et al., <a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite>.
Earlier works considered distillation from an ensemble of multiple teacher models <cite class="ltx_cite ltx_citemacro_citep">(Buciluǎ et al., <a href="#bib.bib5" title="" class="ltx_ref">2006</a>; Hinton et al., <a href="#bib.bib17" title="" class="ltx_ref">2015</a>)</cite>.
As neural networks have become larger, distilling from a larger to a smaller network has become a common paradigm <cite class="ltx_cite ltx_citemacro_citep">(Ba and Caruana, <a href="#bib.bib2" title="" class="ltx_ref">2014</a>)</cite>.
In contrast, in our work the teacher and student model are the same language model, but applied differently (either with intermediate reasoning, or not).</p>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<p id="S2.SS3.p2.1" class="ltx_p">For chain-of-thought reasoning in particular, several distillation approaches have been considered <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib36" title="" class="ltx_ref">2023</a>); Li et al. (<a href="#bib.bib22" title="" class="ltx_ref">2023a</a>); Chen et al. (<a href="#bib.bib7" title="" class="ltx_ref">2024</a>)</cite>.
These again follow the paradigm of distilling a <span id="S2.SS3.p2.1.1" class="ltx_text ltx_font_italic">separate</span> larger model’s output into a smaller model, i.e.
the student model is asked to
mimic the internal thoughts of the teacher model.
The work of <cite class="ltx_cite ltx_citemacro_citet">Zhang et al. (<a href="#bib.bib44" title="" class="ltx_ref">2024</a>)</cite>, however, considers distilling
a slower System 2 method (Tree-of-Thought) into a faster System 2 method (CoT), which can use the same model as student and teacher.
In contrast our work’s goal is to <span id="S2.SS3.p2.1.2" class="ltx_text ltx_font_italic">not</span> generate internal thoughts (to improve System 1).
Some exceptions are <cite class="ltx_cite ltx_citemacro_citet">Deng et al. (<a href="#bib.bib14" title="" class="ltx_ref">2023b</a>, <a href="#bib.bib13" title="" class="ltx_ref">2024</a>)</cite>. The former still uses a separate student and teacher model, but attempts to distill the intermediate thought tokens into the layers of the network by representing reasoning steps as vectors and then setting them as targets. The latter recent work attempts to distill CoT by gradually removing the intermediate steps, which can improve performance greatly compared to not doing so, but still does not match explicit CoT.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Distilling System 2 into System 1</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Setup: System 1 and System 2 models</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">Given an input <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">x</annotation></semantics></math>, in this work we consider the setting of a single model, in our case a large language model (LLM), that is capable of two modes of response:</p>
<ul id="S3.I1" class="ltx_itemize">
<li id="S3.I1.ix1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">(i)</span> 
<div id="S3.I1.ix1.p1" class="ltx_para">
<p id="S3.I1.ix1.p1.1" class="ltx_p"><span id="S3.I1.ix1.p1.1.1" class="ltx_text ltx_font_italic">System 1</span>: Produces the output <math id="S3.I1.ix1.p1.1.m1.1" class="ltx_Math" alttext="y" display="inline"><semantics id="S3.I1.ix1.p1.1.m1.1a"><mi id="S3.I1.ix1.p1.1.m1.1.1" xref="S3.I1.ix1.p1.1.m1.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="S3.I1.ix1.p1.1.m1.1b"><ci id="S3.I1.ix1.p1.1.m1.1.1.cmml" xref="S3.I1.ix1.p1.1.m1.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.ix1.p1.1.m1.1c">y</annotation></semantics></math> directly. This is done by forwarding through the layers of the underlying autoregressive neural network (Transformer) to produce the output tokens.</p>
</div>
</li>
<li id="S3.I1.ix2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">(ii)</span> 
<div id="S3.I1.ix2.p1" class="ltx_para">
<p id="S3.I1.ix2.p1.1" class="ltx_p"><span id="S3.I1.ix2.p1.1.1" class="ltx_text ltx_font_italic">System 2</span>: We define System 2 models as methods that use the underlying Transformer to generate intermediate output tokens <math id="S3.I1.ix2.p1.1.m1.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.I1.ix2.p1.1.m1.1a"><mi id="S3.I1.ix2.p1.1.m1.1.1" xref="S3.I1.ix2.p1.1.m1.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.I1.ix2.p1.1.m1.1b"><ci id="S3.I1.ix2.p1.1.m1.1.1.cmml" xref="S3.I1.ix2.p1.1.m1.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.ix2.p1.1.m1.1c">z</annotation></semantics></math> of any kind <span id="S3.I1.ix2.p1.1.2" class="ltx_text ltx_font_italic">before</span> generating the final response tokens. This may include multiple calls (prompts).</p>
</div>
</li>
</ul>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.5" class="ltx_p">More formally, we consider a System 2 model <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="S_{\text{II}}" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><msub id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml"><mi id="S3.SS1.p2.1.m1.1.1.2" xref="S3.SS1.p2.1.m1.1.1.2.cmml">S</mi><mtext id="S3.SS1.p2.1.m1.1.1.3" xref="S3.SS1.p2.1.m1.1.1.3a.cmml">II</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><apply id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.1.m1.1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p2.1.m1.1.1.2.cmml" xref="S3.SS1.p2.1.m1.1.1.2">𝑆</ci><ci id="S3.SS1.p2.1.m1.1.1.3a.cmml" xref="S3.SS1.p2.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS1.p2.1.m1.1.1.3.cmml" xref="S3.SS1.p2.1.m1.1.1.3">II</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">S_{\text{II}}</annotation></semantics></math> as a function that takes an LLM <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="p_{\theta}" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><msub id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml"><mi id="S3.SS1.p2.2.m2.1.1.2" xref="S3.SS1.p2.2.m2.1.1.2.cmml">p</mi><mi id="S3.SS1.p2.2.m2.1.1.3" xref="S3.SS1.p2.2.m2.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><apply id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.2.m2.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p2.2.m2.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2">𝑝</ci><ci id="S3.SS1.p2.2.m2.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">p_{\theta}</annotation></semantics></math> and input <math id="S3.SS1.p2.3.m3.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS1.p2.3.m3.1a"><mi id="S3.SS1.p2.3.m3.1.1" xref="S3.SS1.p2.3.m3.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m3.1b"><ci id="S3.SS1.p2.3.m3.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m3.1c">x</annotation></semantics></math>, and can call the LLM possibly repeatedly to generate intermediate tokens <math id="S3.SS1.p2.4.m4.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.SS1.p2.4.m4.1a"><mi id="S3.SS1.p2.4.m4.1.1" xref="S3.SS1.p2.4.m4.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.4.m4.1b"><ci id="S3.SS1.p2.4.m4.1.1.cmml" xref="S3.SS1.p2.4.m4.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.4.m4.1c">z</annotation></semantics></math> using a specific algorithm, before returning an output <math id="S3.SS1.p2.5.m5.1" class="ltx_Math" alttext="y" display="inline"><semantics id="S3.SS1.p2.5.m5.1a"><mi id="S3.SS1.p2.5.m5.1.1" xref="S3.SS1.p2.5.m5.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.5.m5.1b"><ci id="S3.SS1.p2.5.m5.1.1.cmml" xref="S3.SS1.p2.5.m5.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.5.m5.1c">y</annotation></semantics></math>:</p>
<table id="A1.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E1.m1.4" class="ltx_Math" alttext="\displaystyle S_{\text{II}}(x;p_{\theta})\rightarrow z,y." display="inline"><semantics id="S3.E1.m1.4a"><mrow id="S3.E1.m1.4.4.1" xref="S3.E1.m1.4.4.1.1.cmml"><mrow id="S3.E1.m1.4.4.1.1" xref="S3.E1.m1.4.4.1.1.cmml"><mrow id="S3.E1.m1.4.4.1.1.1" xref="S3.E1.m1.4.4.1.1.1.cmml"><msub id="S3.E1.m1.4.4.1.1.1.3" xref="S3.E1.m1.4.4.1.1.1.3.cmml"><mi id="S3.E1.m1.4.4.1.1.1.3.2" xref="S3.E1.m1.4.4.1.1.1.3.2.cmml">S</mi><mtext id="S3.E1.m1.4.4.1.1.1.3.3" xref="S3.E1.m1.4.4.1.1.1.3.3a.cmml">II</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E1.m1.4.4.1.1.1.2" xref="S3.E1.m1.4.4.1.1.1.2.cmml">​</mo><mrow id="S3.E1.m1.4.4.1.1.1.1.1" xref="S3.E1.m1.4.4.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.E1.m1.4.4.1.1.1.1.1.2" xref="S3.E1.m1.4.4.1.1.1.1.2.cmml">(</mo><mi id="S3.E1.m1.1.1" xref="S3.E1.m1.1.1.cmml">x</mi><mo id="S3.E1.m1.4.4.1.1.1.1.1.3" xref="S3.E1.m1.4.4.1.1.1.1.2.cmml">;</mo><msub id="S3.E1.m1.4.4.1.1.1.1.1.1" xref="S3.E1.m1.4.4.1.1.1.1.1.1.cmml"><mi id="S3.E1.m1.4.4.1.1.1.1.1.1.2" xref="S3.E1.m1.4.4.1.1.1.1.1.1.2.cmml">p</mi><mi id="S3.E1.m1.4.4.1.1.1.1.1.1.3" xref="S3.E1.m1.4.4.1.1.1.1.1.1.3.cmml">θ</mi></msub><mo stretchy="false" id="S3.E1.m1.4.4.1.1.1.1.1.4" xref="S3.E1.m1.4.4.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.E1.m1.4.4.1.1.2" xref="S3.E1.m1.4.4.1.1.2.cmml">→</mo><mrow id="S3.E1.m1.4.4.1.1.3.2" xref="S3.E1.m1.4.4.1.1.3.1.cmml"><mi id="S3.E1.m1.2.2" xref="S3.E1.m1.2.2.cmml">z</mi><mo id="S3.E1.m1.4.4.1.1.3.2.1" xref="S3.E1.m1.4.4.1.1.3.1.cmml">,</mo><mi id="S3.E1.m1.3.3" xref="S3.E1.m1.3.3.cmml">y</mi></mrow></mrow><mo lspace="0em" id="S3.E1.m1.4.4.1.2" xref="S3.E1.m1.4.4.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E1.m1.4b"><apply id="S3.E1.m1.4.4.1.1.cmml" xref="S3.E1.m1.4.4.1"><ci id="S3.E1.m1.4.4.1.1.2.cmml" xref="S3.E1.m1.4.4.1.1.2">→</ci><apply id="S3.E1.m1.4.4.1.1.1.cmml" xref="S3.E1.m1.4.4.1.1.1"><times id="S3.E1.m1.4.4.1.1.1.2.cmml" xref="S3.E1.m1.4.4.1.1.1.2"></times><apply id="S3.E1.m1.4.4.1.1.1.3.cmml" xref="S3.E1.m1.4.4.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m1.4.4.1.1.1.3.1.cmml" xref="S3.E1.m1.4.4.1.1.1.3">subscript</csymbol><ci id="S3.E1.m1.4.4.1.1.1.3.2.cmml" xref="S3.E1.m1.4.4.1.1.1.3.2">𝑆</ci><ci id="S3.E1.m1.4.4.1.1.1.3.3a.cmml" xref="S3.E1.m1.4.4.1.1.1.3.3"><mtext mathsize="70%" id="S3.E1.m1.4.4.1.1.1.3.3.cmml" xref="S3.E1.m1.4.4.1.1.1.3.3">II</mtext></ci></apply><list id="S3.E1.m1.4.4.1.1.1.1.2.cmml" xref="S3.E1.m1.4.4.1.1.1.1.1"><ci id="S3.E1.m1.1.1.cmml" xref="S3.E1.m1.1.1">𝑥</ci><apply id="S3.E1.m1.4.4.1.1.1.1.1.1.cmml" xref="S3.E1.m1.4.4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1.m1.4.4.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.4.4.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E1.m1.4.4.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.4.4.1.1.1.1.1.1.2">𝑝</ci><ci id="S3.E1.m1.4.4.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.4.4.1.1.1.1.1.1.3">𝜃</ci></apply></list></apply><list id="S3.E1.m1.4.4.1.1.3.1.cmml" xref="S3.E1.m1.4.4.1.1.3.2"><ci id="S3.E1.m1.2.2.cmml" xref="S3.E1.m1.2.2">𝑧</ci><ci id="S3.E1.m1.3.3.cmml" xref="S3.E1.m1.3.3">𝑦</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m1.4c">\displaystyle S_{\text{II}}(x;p_{\theta})\rightarrow z,y.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p2.8" class="ltx_p">System 2 approaches can potentially involve multiple prompts, branching, iteration and search, all the while using the LLM to generate intermediate results for further processing. In contrast, a System 1 model only considers the original input <math id="S3.SS1.p2.6.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS1.p2.6.m1.1a"><mi id="S3.SS1.p2.6.m1.1.1" xref="S3.SS1.p2.6.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.6.m1.1b"><ci id="S3.SS1.p2.6.m1.1.1.cmml" xref="S3.SS1.p2.6.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.6.m1.1c">x</annotation></semantics></math> and calls the LLM <math id="S3.SS1.p2.7.m2.1" class="ltx_Math" alttext="p_{\theta}" display="inline"><semantics id="S3.SS1.p2.7.m2.1a"><msub id="S3.SS1.p2.7.m2.1.1" xref="S3.SS1.p2.7.m2.1.1.cmml"><mi id="S3.SS1.p2.7.m2.1.1.2" xref="S3.SS1.p2.7.m2.1.1.2.cmml">p</mi><mi id="S3.SS1.p2.7.m2.1.1.3" xref="S3.SS1.p2.7.m2.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.7.m2.1b"><apply id="S3.SS1.p2.7.m2.1.1.cmml" xref="S3.SS1.p2.7.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.7.m2.1.1.1.cmml" xref="S3.SS1.p2.7.m2.1.1">subscript</csymbol><ci id="S3.SS1.p2.7.m2.1.1.2.cmml" xref="S3.SS1.p2.7.m2.1.1.2">𝑝</ci><ci id="S3.SS1.p2.7.m2.1.1.3.cmml" xref="S3.SS1.p2.7.m2.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.7.m2.1c">p_{\theta}</annotation></semantics></math> directly to produce an output <math id="S3.SS1.p2.8.m3.1" class="ltx_Math" alttext="y" display="inline"><semantics id="S3.SS1.p2.8.m3.1a"><mi id="S3.SS1.p2.8.m3.1.1" xref="S3.SS1.p2.8.m3.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.8.m3.1b"><ci id="S3.SS1.p2.8.m3.1.1.cmml" xref="S3.SS1.p2.8.m3.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.8.m3.1c">y</annotation></semantics></math>:</p>
<table id="A1.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E2.m1.3" class="ltx_Math" alttext="\displaystyle S_{\text{I}}(x)=p_{\theta}(x)\rightarrow y." display="inline"><semantics id="S3.E2.m1.3a"><mrow id="S3.E2.m1.3.3.1" xref="S3.E2.m1.3.3.1.1.cmml"><mrow id="S3.E2.m1.3.3.1.1" xref="S3.E2.m1.3.3.1.1.cmml"><mrow id="S3.E2.m1.3.3.1.1.2" xref="S3.E2.m1.3.3.1.1.2.cmml"><msub id="S3.E2.m1.3.3.1.1.2.2" xref="S3.E2.m1.3.3.1.1.2.2.cmml"><mi id="S3.E2.m1.3.3.1.1.2.2.2" xref="S3.E2.m1.3.3.1.1.2.2.2.cmml">S</mi><mtext id="S3.E2.m1.3.3.1.1.2.2.3" xref="S3.E2.m1.3.3.1.1.2.2.3a.cmml">I</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E2.m1.3.3.1.1.2.1" xref="S3.E2.m1.3.3.1.1.2.1.cmml">​</mo><mrow id="S3.E2.m1.3.3.1.1.2.3.2" xref="S3.E2.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="S3.E2.m1.3.3.1.1.2.3.2.1" xref="S3.E2.m1.3.3.1.1.2.cmml">(</mo><mi id="S3.E2.m1.1.1" xref="S3.E2.m1.1.1.cmml">x</mi><mo stretchy="false" id="S3.E2.m1.3.3.1.1.2.3.2.2" xref="S3.E2.m1.3.3.1.1.2.cmml">)</mo></mrow></mrow><mo id="S3.E2.m1.3.3.1.1.3" xref="S3.E2.m1.3.3.1.1.3.cmml">=</mo><mrow id="S3.E2.m1.3.3.1.1.4" xref="S3.E2.m1.3.3.1.1.4.cmml"><msub id="S3.E2.m1.3.3.1.1.4.2" xref="S3.E2.m1.3.3.1.1.4.2.cmml"><mi id="S3.E2.m1.3.3.1.1.4.2.2" xref="S3.E2.m1.3.3.1.1.4.2.2.cmml">p</mi><mi id="S3.E2.m1.3.3.1.1.4.2.3" xref="S3.E2.m1.3.3.1.1.4.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E2.m1.3.3.1.1.4.1" xref="S3.E2.m1.3.3.1.1.4.1.cmml">​</mo><mrow id="S3.E2.m1.3.3.1.1.4.3.2" xref="S3.E2.m1.3.3.1.1.4.cmml"><mo stretchy="false" id="S3.E2.m1.3.3.1.1.4.3.2.1" xref="S3.E2.m1.3.3.1.1.4.cmml">(</mo><mi id="S3.E2.m1.2.2" xref="S3.E2.m1.2.2.cmml">x</mi><mo stretchy="false" id="S3.E2.m1.3.3.1.1.4.3.2.2" xref="S3.E2.m1.3.3.1.1.4.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.E2.m1.3.3.1.1.5" xref="S3.E2.m1.3.3.1.1.5.cmml">→</mo><mi id="S3.E2.m1.3.3.1.1.6" xref="S3.E2.m1.3.3.1.1.6.cmml">y</mi></mrow><mo lspace="0em" id="S3.E2.m1.3.3.1.2" xref="S3.E2.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.3b"><apply id="S3.E2.m1.3.3.1.1.cmml" xref="S3.E2.m1.3.3.1"><and id="S3.E2.m1.3.3.1.1a.cmml" xref="S3.E2.m1.3.3.1"></and><apply id="S3.E2.m1.3.3.1.1b.cmml" xref="S3.E2.m1.3.3.1"><eq id="S3.E2.m1.3.3.1.1.3.cmml" xref="S3.E2.m1.3.3.1.1.3"></eq><apply id="S3.E2.m1.3.3.1.1.2.cmml" xref="S3.E2.m1.3.3.1.1.2"><times id="S3.E2.m1.3.3.1.1.2.1.cmml" xref="S3.E2.m1.3.3.1.1.2.1"></times><apply id="S3.E2.m1.3.3.1.1.2.2.cmml" xref="S3.E2.m1.3.3.1.1.2.2"><csymbol cd="ambiguous" id="S3.E2.m1.3.3.1.1.2.2.1.cmml" xref="S3.E2.m1.3.3.1.1.2.2">subscript</csymbol><ci id="S3.E2.m1.3.3.1.1.2.2.2.cmml" xref="S3.E2.m1.3.3.1.1.2.2.2">𝑆</ci><ci id="S3.E2.m1.3.3.1.1.2.2.3a.cmml" xref="S3.E2.m1.3.3.1.1.2.2.3"><mtext mathsize="70%" id="S3.E2.m1.3.3.1.1.2.2.3.cmml" xref="S3.E2.m1.3.3.1.1.2.2.3">I</mtext></ci></apply><ci id="S3.E2.m1.1.1.cmml" xref="S3.E2.m1.1.1">𝑥</ci></apply><apply id="S3.E2.m1.3.3.1.1.4.cmml" xref="S3.E2.m1.3.3.1.1.4"><times id="S3.E2.m1.3.3.1.1.4.1.cmml" xref="S3.E2.m1.3.3.1.1.4.1"></times><apply id="S3.E2.m1.3.3.1.1.4.2.cmml" xref="S3.E2.m1.3.3.1.1.4.2"><csymbol cd="ambiguous" id="S3.E2.m1.3.3.1.1.4.2.1.cmml" xref="S3.E2.m1.3.3.1.1.4.2">subscript</csymbol><ci id="S3.E2.m1.3.3.1.1.4.2.2.cmml" xref="S3.E2.m1.3.3.1.1.4.2.2">𝑝</ci><ci id="S3.E2.m1.3.3.1.1.4.2.3.cmml" xref="S3.E2.m1.3.3.1.1.4.2.3">𝜃</ci></apply><ci id="S3.E2.m1.2.2.cmml" xref="S3.E2.m1.2.2">𝑥</ci></apply></apply><apply id="S3.E2.m1.3.3.1.1c.cmml" xref="S3.E2.m1.3.3.1"><ci id="S3.E2.m1.3.3.1.1.5.cmml" xref="S3.E2.m1.3.3.1.1.5">→</ci><share href="#S3.E2.m1.3.3.1.1.4.cmml" id="S3.E2.m1.3.3.1.1d.cmml" xref="S3.E2.m1.3.3.1"></share><ci id="S3.E2.m1.3.3.1.1.6.cmml" xref="S3.E2.m1.3.3.1.1.6">𝑦</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.3c">\displaystyle S_{\text{I}}(x)=p_{\theta}(x)\rightarrow y.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS1.p3" class="ltx_para">
<p id="S3.SS1.p3.1" class="ltx_p">There are many existing instantiations of System 2 models.
Chain-of-thought prompting only requires a single LLM prompt, but still outputs intermediate generations before a final response, typically used in math and other reasoning tasks <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite>.</p>
</div>
<div id="S3.SS1.p4" class="ltx_para">
<p id="S3.SS1.p4.1" class="ltx_p">Methods like System 2 Attention <cite class="ltx_cite ltx_citemacro_citep">(Weston and Sukhbaatar, <a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite> and Rephrase and Respond <cite class="ltx_cite ltx_citemacro_citep">(Deng et al., <a href="#bib.bib12" title="" class="ltx_ref">2023a</a>)</cite> require two calls to the LLM, where in the former the first call is used to attend to the context and remove bias, and in the latter to expand on the question. The second call is then used to finally respond to the answer given the intermediate generations.
Some methods are much more sophisticated for example Branch-Solve-Merge <cite class="ltx_cite ltx_citemacro_citep">(Saha et al., <a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite>
which generates a plan via an LLM which branches into several more LLM calls until a final stage merges the results.</p>
</div>
<div id="S3.SS1.p5" class="ltx_para">
<p id="S3.SS1.p5.1" class="ltx_p">We will perform experiments with the four methods just described, but there are many other system 2 approaches, for example Tree-of-Thoughts <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib41" title="" class="ltx_ref">2024</a>)</cite>, Graph-of-Thoughts <cite class="ltx_cite ltx_citemacro_citep">(Besta et al., <a href="#bib.bib4" title="" class="ltx_ref">2024</a>)</cite> and more, see related work in <a href="#S2" title="2 Related work ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">section 2</span></a>.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Method: System 2 Distillation</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.4" class="ltx_p">Many System 2 methods, by their nature, are significantly slower at inference time due to multiple prompt calls and generation of intermediate tokens. The aim of System 2 Distillation is to
distill all the reasoning from <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="S_{\text{II}}" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><msub id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml"><mi id="S3.SS2.p1.1.m1.1.1.2" xref="S3.SS2.p1.1.m1.1.1.2.cmml">S</mi><mtext id="S3.SS2.p1.1.m1.1.1.3" xref="S3.SS2.p1.1.m1.1.1.3a.cmml">II</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><apply id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.p1.1.m1.1.1.2">𝑆</ci><ci id="S3.SS2.p1.1.m1.1.1.3a.cmml" xref="S3.SS2.p1.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.p1.1.m1.1.1.3">II</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">S_{\text{II}}</annotation></semantics></math> back into
<math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="S_{\text{I}}" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><msub id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml"><mi id="S3.SS2.p1.2.m2.1.1.2" xref="S3.SS2.p1.2.m2.1.1.2.cmml">S</mi><mtext id="S3.SS2.p1.2.m2.1.1.3" xref="S3.SS2.p1.2.m2.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><apply id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.1.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.p1.2.m2.1.1.2.cmml" xref="S3.SS2.p1.2.m2.1.1.2">𝑆</ci><ci id="S3.SS2.p1.2.m2.1.1.3a.cmml" xref="S3.SS2.p1.2.m2.1.1.3"><mtext mathsize="70%" id="S3.SS2.p1.2.m2.1.1.3.cmml" xref="S3.SS2.p1.2.m2.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">S_{\text{I}}</annotation></semantics></math> so that the direct outputs from the language model <math id="S3.SS2.p1.3.m3.1" class="ltx_Math" alttext="p_{\theta}(x)" display="inline"><semantics id="S3.SS2.p1.3.m3.1a"><mrow id="S3.SS2.p1.3.m3.1.2" xref="S3.SS2.p1.3.m3.1.2.cmml"><msub id="S3.SS2.p1.3.m3.1.2.2" xref="S3.SS2.p1.3.m3.1.2.2.cmml"><mi id="S3.SS2.p1.3.m3.1.2.2.2" xref="S3.SS2.p1.3.m3.1.2.2.2.cmml">p</mi><mi id="S3.SS2.p1.3.m3.1.2.2.3" xref="S3.SS2.p1.3.m3.1.2.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS2.p1.3.m3.1.2.1" xref="S3.SS2.p1.3.m3.1.2.1.cmml">​</mo><mrow id="S3.SS2.p1.3.m3.1.2.3.2" xref="S3.SS2.p1.3.m3.1.2.cmml"><mo stretchy="false" id="S3.SS2.p1.3.m3.1.2.3.2.1" xref="S3.SS2.p1.3.m3.1.2.cmml">(</mo><mi id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS2.p1.3.m3.1.2.3.2.2" xref="S3.SS2.p1.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.1b"><apply id="S3.SS2.p1.3.m3.1.2.cmml" xref="S3.SS2.p1.3.m3.1.2"><times id="S3.SS2.p1.3.m3.1.2.1.cmml" xref="S3.SS2.p1.3.m3.1.2.1"></times><apply id="S3.SS2.p1.3.m3.1.2.2.cmml" xref="S3.SS2.p1.3.m3.1.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.3.m3.1.2.2.1.cmml" xref="S3.SS2.p1.3.m3.1.2.2">subscript</csymbol><ci id="S3.SS2.p1.3.m3.1.2.2.2.cmml" xref="S3.SS2.p1.3.m3.1.2.2.2">𝑝</ci><ci id="S3.SS2.p1.3.m3.1.2.2.3.cmml" xref="S3.SS2.p1.3.m3.1.2.2.3">𝜃</ci></apply><ci id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.1c">p_{\theta}(x)</annotation></semantics></math>
are improved.
We assume a setting where the model has access to <span id="S3.SS2.p1.4.1" class="ltx_text ltx_font_italic">unlabeled inputs</span>
<math id="S3.SS2.p1.4.m4.1" class="ltx_Math" alttext="\mathcal{X}" display="inline"><semantics id="S3.SS2.p1.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.4.m4.1.1" xref="S3.SS2.p1.4.m4.1.1.cmml">𝒳</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m4.1b"><ci id="S3.SS2.p1.4.m4.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1">𝒳</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m4.1c">\mathcal{X}</annotation></semantics></math> from which it can learn, in analogy to how humans learn their <span id="S3.SS2.p1.4.2" class="ltx_text ltx_font_italic">procedural memory</span> without supervision.
For language-based tasks, it is common to have access to instruction following prompts (inputs) as they can be collected from humans, e.g. the 1M released WildChat interactions <cite class="ltx_cite ltx_citemacro_citep">(Zhao et al., <a href="#bib.bib45" title="" class="ltx_ref">2024</a>)</cite> where inputs are given but correct labels are unknown.
Hence this is a realistic setup.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">The first step of the proposed method is to generate responses using the System 2 model over the unlabeled inputs <math id="S3.SS2.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{X}" display="inline"><semantics id="S3.SS2.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1.cmml">𝒳</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.1b"><ci id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1">𝒳</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.1c">\mathcal{X}</annotation></semantics></math>:</p>
<table id="A1.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E3.m1.1" class="ltx_Math" alttext="\displaystyle y_{S_{\text{II}}}^{i}=S_{\text{II}}(x^{i};p_{\theta}),~{}~{}~{}\forall x_{i}\in\mathcal{X}." display="inline"><semantics id="S3.E3.m1.1a"><mrow id="S3.E3.m1.1.1.1"><mrow id="S3.E3.m1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.3.cmml"><mrow id="S3.E3.m1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.cmml"><msubsup id="S3.E3.m1.1.1.1.1.1.1.4" xref="S3.E3.m1.1.1.1.1.1.1.4.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.4.2.2" xref="S3.E3.m1.1.1.1.1.1.1.4.2.2.cmml">y</mi><msub id="S3.E3.m1.1.1.1.1.1.1.4.2.3" xref="S3.E3.m1.1.1.1.1.1.1.4.2.3.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.4.2.3.2" xref="S3.E3.m1.1.1.1.1.1.1.4.2.3.2.cmml">S</mi><mtext id="S3.E3.m1.1.1.1.1.1.1.4.2.3.3" xref="S3.E3.m1.1.1.1.1.1.1.4.2.3.3a.cmml">II</mtext></msub><mi id="S3.E3.m1.1.1.1.1.1.1.4.3" xref="S3.E3.m1.1.1.1.1.1.1.4.3.cmml">i</mi></msubsup><mo id="S3.E3.m1.1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.1.3.cmml">=</mo><mrow id="S3.E3.m1.1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.1.2.cmml"><msub id="S3.E3.m1.1.1.1.1.1.1.2.4" xref="S3.E3.m1.1.1.1.1.1.1.2.4.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.2.4.2" xref="S3.E3.m1.1.1.1.1.1.1.2.4.2.cmml">S</mi><mtext id="S3.E3.m1.1.1.1.1.1.1.2.4.3" xref="S3.E3.m1.1.1.1.1.1.1.2.4.3a.cmml">II</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E3.m1.1.1.1.1.1.1.2.3" xref="S3.E3.m1.1.1.1.1.1.1.2.3.cmml">​</mo><mrow id="S3.E3.m1.1.1.1.1.1.1.2.2.2" xref="S3.E3.m1.1.1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E3.m1.1.1.1.1.1.1.2.2.2.3" xref="S3.E3.m1.1.1.1.1.1.1.2.2.3.cmml">(</mo><msup id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msup><mo id="S3.E3.m1.1.1.1.1.1.1.2.2.2.4" xref="S3.E3.m1.1.1.1.1.1.1.2.2.3.cmml">;</mo><msub id="S3.E3.m1.1.1.1.1.1.1.2.2.2.2" xref="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.2" xref="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.2.cmml">p</mi><mi id="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.3" xref="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.3.cmml">θ</mi></msub><mo stretchy="false" id="S3.E3.m1.1.1.1.1.1.1.2.2.2.5" xref="S3.E3.m1.1.1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo rspace="1.157em" id="S3.E3.m1.1.1.1.1.2.3" xref="S3.E3.m1.1.1.1.1.3a.cmml">,</mo><mrow id="S3.E3.m1.1.1.1.1.2.2" xref="S3.E3.m1.1.1.1.1.2.2.cmml"><mrow id="S3.E3.m1.1.1.1.1.2.2.2" xref="S3.E3.m1.1.1.1.1.2.2.2.cmml"><mo rspace="0.167em" id="S3.E3.m1.1.1.1.1.2.2.2.1" xref="S3.E3.m1.1.1.1.1.2.2.2.1.cmml">∀</mo><msub id="S3.E3.m1.1.1.1.1.2.2.2.2" xref="S3.E3.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S3.E3.m1.1.1.1.1.2.2.2.2.2" xref="S3.E3.m1.1.1.1.1.2.2.2.2.2.cmml">x</mi><mi id="S3.E3.m1.1.1.1.1.2.2.2.2.3" xref="S3.E3.m1.1.1.1.1.2.2.2.2.3.cmml">i</mi></msub></mrow><mo id="S3.E3.m1.1.1.1.1.2.2.1" xref="S3.E3.m1.1.1.1.1.2.2.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.E3.m1.1.1.1.1.2.2.3" xref="S3.E3.m1.1.1.1.1.2.2.3.cmml">𝒳</mi></mrow></mrow><mo lspace="0em" id="S3.E3.m1.1.1.1.2">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.1b"><apply id="S3.E3.m1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.3a.cmml" xref="S3.E3.m1.1.1.1.1.2.3">formulae-sequence</csymbol><apply id="S3.E3.m1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1"><eq id="S3.E3.m1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.3"></eq><apply id="S3.E3.m1.1.1.1.1.1.1.4.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.1.4.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4">superscript</csymbol><apply id="S3.E3.m1.1.1.1.1.1.1.4.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.1.4.2.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4">subscript</csymbol><ci id="S3.E3.m1.1.1.1.1.1.1.4.2.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4.2.2">𝑦</ci><apply id="S3.E3.m1.1.1.1.1.1.1.4.2.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4.2.3"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.1.4.2.3.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4.2.3">subscript</csymbol><ci id="S3.E3.m1.1.1.1.1.1.1.4.2.3.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4.2.3.2">𝑆</ci><ci id="S3.E3.m1.1.1.1.1.1.1.4.2.3.3a.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4.2.3.3"><mtext mathsize="50%" id="S3.E3.m1.1.1.1.1.1.1.4.2.3.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4.2.3.3">II</mtext></ci></apply></apply><ci id="S3.E3.m1.1.1.1.1.1.1.4.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.4.3">𝑖</ci></apply><apply id="S3.E3.m1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2"><times id="S3.E3.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.3"></times><apply id="S3.E3.m1.1.1.1.1.1.1.2.4.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.1.2.4.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.4">subscript</csymbol><ci id="S3.E3.m1.1.1.1.1.1.1.2.4.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.4.2">𝑆</ci><ci id="S3.E3.m1.1.1.1.1.1.1.2.4.3a.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.4.3"><mtext mathsize="70%" id="S3.E3.m1.1.1.1.1.1.1.2.4.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.4.3">II</mtext></ci></apply><list id="S3.E3.m1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.2.2"><apply id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.2">𝑝</ci><ci id="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2.2.2.2.3">𝜃</ci></apply></list></apply></apply><apply id="S3.E3.m1.1.1.1.1.2.2.cmml" xref="S3.E3.m1.1.1.1.1.2.2"><in id="S3.E3.m1.1.1.1.1.2.2.1.cmml" xref="S3.E3.m1.1.1.1.1.2.2.1"></in><apply id="S3.E3.m1.1.1.1.1.2.2.2.cmml" xref="S3.E3.m1.1.1.1.1.2.2.2"><csymbol cd="latexml" id="S3.E3.m1.1.1.1.1.2.2.2.1.cmml" xref="S3.E3.m1.1.1.1.1.2.2.2.1">for-all</csymbol><apply id="S3.E3.m1.1.1.1.1.2.2.2.2.cmml" xref="S3.E3.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.E3.m1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E3.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E3.m1.1.1.1.1.2.2.2.2.2">𝑥</ci><ci id="S3.E3.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E3.m1.1.1.1.1.2.2.2.2.3">𝑖</ci></apply></apply><ci id="S3.E3.m1.1.1.1.1.2.2.3.cmml" xref="S3.E3.m1.1.1.1.1.2.2.3">𝒳</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.1c">\displaystyle y_{S_{\text{II}}}^{i}=S_{\text{II}}(x^{i};p_{\theta}),~{}~{}~{}\forall x_{i}\in\mathcal{X}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.2" class="ltx_p">Note we discard (do not store) the intermediate outputs <math id="S3.SS2.p3.1.m1.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.SS2.p3.1.m1.1a"><mi id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.1b"><ci id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.1c">z</annotation></semantics></math> from Eq. <a href="#S3.E1" title="Equation 1 ‣ 3.1 Setup: System 1 and System 2 models ‣ 3 Distilling System 2 into System 1 ‣ Distilling System 2 into System 1" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. These responses <math id="S3.SS2.p3.2.m2.1" class="ltx_Math" alttext="y_{S_{\text{II}}}^{i}" display="inline"><semantics id="S3.SS2.p3.2.m2.1a"><msubsup id="S3.SS2.p3.2.m2.1.1" xref="S3.SS2.p3.2.m2.1.1.cmml"><mi id="S3.SS2.p3.2.m2.1.1.2.2" xref="S3.SS2.p3.2.m2.1.1.2.2.cmml">y</mi><msub id="S3.SS2.p3.2.m2.1.1.2.3" xref="S3.SS2.p3.2.m2.1.1.2.3.cmml"><mi id="S3.SS2.p3.2.m2.1.1.2.3.2" xref="S3.SS2.p3.2.m2.1.1.2.3.2.cmml">S</mi><mtext id="S3.SS2.p3.2.m2.1.1.2.3.3" xref="S3.SS2.p3.2.m2.1.1.2.3.3a.cmml">II</mtext></msub><mi id="S3.SS2.p3.2.m2.1.1.3" xref="S3.SS2.p3.2.m2.1.1.3.cmml">i</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.2.m2.1b"><apply id="S3.SS2.p3.2.m2.1.1.cmml" xref="S3.SS2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p3.2.m2.1.1.1.cmml" xref="S3.SS2.p3.2.m2.1.1">superscript</csymbol><apply id="S3.SS2.p3.2.m2.1.1.2.cmml" xref="S3.SS2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p3.2.m2.1.1.2.1.cmml" xref="S3.SS2.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.p3.2.m2.1.1.2.2.cmml" xref="S3.SS2.p3.2.m2.1.1.2.2">𝑦</ci><apply id="S3.SS2.p3.2.m2.1.1.2.3.cmml" xref="S3.SS2.p3.2.m2.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.p3.2.m2.1.1.2.3.1.cmml" xref="S3.SS2.p3.2.m2.1.1.2.3">subscript</csymbol><ci id="S3.SS2.p3.2.m2.1.1.2.3.2.cmml" xref="S3.SS2.p3.2.m2.1.1.2.3.2">𝑆</ci><ci id="S3.SS2.p3.2.m2.1.1.2.3.3a.cmml" xref="S3.SS2.p3.2.m2.1.1.2.3.3"><mtext mathsize="50%" id="S3.SS2.p3.2.m2.1.1.2.3.3.cmml" xref="S3.SS2.p3.2.m2.1.1.2.3.3">II</mtext></ci></apply></apply><ci id="S3.SS2.p3.2.m2.1.1.3.cmml" xref="S3.SS2.p3.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.2.m2.1c">y_{S_{\text{II}}}^{i}</annotation></semantics></math> can then be used directly as System 2 distillation targets for fine-tuning a System 1 model. However, they are subject to noise: some of these responses could be high quality, while others could be low quality or incorrect.
For shortform QA and reasoning tasks involving a short response with a typically unique
correct (but unknown) answer, we thus consider an <span id="S3.SS2.p3.2.1" class="ltx_text ltx_font_italic">unsupervised curation</span> step to attempt to improve training data quality. We consider two variations which both rely on a consistency criterion:</p>
<ul id="S3.I2" class="ltx_itemize">
<li id="S3.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i1.p1" class="ltx_para">
<p id="S3.I2.i1.p1.2" class="ltx_p"><span id="S3.I2.i1.p1.2.1" class="ltx_text ltx_font_italic">self-consistency of outputs</span>: we sample <math id="S3.I2.i1.p1.1.m1.2" class="ltx_Math" alttext="S_{\text{II}}(x^{i};p_{\theta})" display="inline"><semantics id="S3.I2.i1.p1.1.m1.2a"><mrow id="S3.I2.i1.p1.1.m1.2.2" xref="S3.I2.i1.p1.1.m1.2.2.cmml"><msub id="S3.I2.i1.p1.1.m1.2.2.4" xref="S3.I2.i1.p1.1.m1.2.2.4.cmml"><mi id="S3.I2.i1.p1.1.m1.2.2.4.2" xref="S3.I2.i1.p1.1.m1.2.2.4.2.cmml">S</mi><mtext id="S3.I2.i1.p1.1.m1.2.2.4.3" xref="S3.I2.i1.p1.1.m1.2.2.4.3a.cmml">II</mtext></msub><mo lspace="0em" rspace="0em" id="S3.I2.i1.p1.1.m1.2.2.3" xref="S3.I2.i1.p1.1.m1.2.2.3.cmml">​</mo><mrow id="S3.I2.i1.p1.1.m1.2.2.2.2" xref="S3.I2.i1.p1.1.m1.2.2.2.3.cmml"><mo stretchy="false" id="S3.I2.i1.p1.1.m1.2.2.2.2.3" xref="S3.I2.i1.p1.1.m1.2.2.2.3.cmml">(</mo><msup id="S3.I2.i1.p1.1.m1.1.1.1.1.1" xref="S3.I2.i1.p1.1.m1.1.1.1.1.1.cmml"><mi id="S3.I2.i1.p1.1.m1.1.1.1.1.1.2" xref="S3.I2.i1.p1.1.m1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.I2.i1.p1.1.m1.1.1.1.1.1.3" xref="S3.I2.i1.p1.1.m1.1.1.1.1.1.3.cmml">i</mi></msup><mo id="S3.I2.i1.p1.1.m1.2.2.2.2.4" xref="S3.I2.i1.p1.1.m1.2.2.2.3.cmml">;</mo><msub id="S3.I2.i1.p1.1.m1.2.2.2.2.2" xref="S3.I2.i1.p1.1.m1.2.2.2.2.2.cmml"><mi id="S3.I2.i1.p1.1.m1.2.2.2.2.2.2" xref="S3.I2.i1.p1.1.m1.2.2.2.2.2.2.cmml">p</mi><mi id="S3.I2.i1.p1.1.m1.2.2.2.2.2.3" xref="S3.I2.i1.p1.1.m1.2.2.2.2.2.3.cmml">θ</mi></msub><mo stretchy="false" id="S3.I2.i1.p1.1.m1.2.2.2.2.5" xref="S3.I2.i1.p1.1.m1.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I2.i1.p1.1.m1.2b"><apply id="S3.I2.i1.p1.1.m1.2.2.cmml" xref="S3.I2.i1.p1.1.m1.2.2"><times id="S3.I2.i1.p1.1.m1.2.2.3.cmml" xref="S3.I2.i1.p1.1.m1.2.2.3"></times><apply id="S3.I2.i1.p1.1.m1.2.2.4.cmml" xref="S3.I2.i1.p1.1.m1.2.2.4"><csymbol cd="ambiguous" id="S3.I2.i1.p1.1.m1.2.2.4.1.cmml" xref="S3.I2.i1.p1.1.m1.2.2.4">subscript</csymbol><ci id="S3.I2.i1.p1.1.m1.2.2.4.2.cmml" xref="S3.I2.i1.p1.1.m1.2.2.4.2">𝑆</ci><ci id="S3.I2.i1.p1.1.m1.2.2.4.3a.cmml" xref="S3.I2.i1.p1.1.m1.2.2.4.3"><mtext mathsize="70%" id="S3.I2.i1.p1.1.m1.2.2.4.3.cmml" xref="S3.I2.i1.p1.1.m1.2.2.4.3">II</mtext></ci></apply><list id="S3.I2.i1.p1.1.m1.2.2.2.3.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.2"><apply id="S3.I2.i1.p1.1.m1.1.1.1.1.1.cmml" xref="S3.I2.i1.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.I2.i1.p1.1.m1.1.1.1.1.1.1.cmml" xref="S3.I2.i1.p1.1.m1.1.1.1.1.1">superscript</csymbol><ci id="S3.I2.i1.p1.1.m1.1.1.1.1.1.2.cmml" xref="S3.I2.i1.p1.1.m1.1.1.1.1.1.2">𝑥</ci><ci id="S3.I2.i1.p1.1.m1.1.1.1.1.1.3.cmml" xref="S3.I2.i1.p1.1.m1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.I2.i1.p1.1.m1.2.2.2.2.2.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.I2.i1.p1.1.m1.2.2.2.2.2.1.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.2.2">subscript</csymbol><ci id="S3.I2.i1.p1.1.m1.2.2.2.2.2.2.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.2.2.2">𝑝</ci><ci id="S3.I2.i1.p1.1.m1.2.2.2.2.2.3.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.2.2.3">𝜃</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i1.p1.1.m1.2c">S_{\text{II}}(x^{i};p_{\theta})</annotation></semantics></math> a total of <math id="S3.I2.i1.p1.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.I2.i1.p1.2.m2.1a"><mi id="S3.I2.i1.p1.2.m2.1.1" xref="S3.I2.i1.p1.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.I2.i1.p1.2.m2.1b"><ci id="S3.I2.i1.p1.2.m2.1.1.cmml" xref="S3.I2.i1.p1.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i1.p1.2.m2.1c">N</annotation></semantics></math> times, and accept the response that is the majority vote; if there is no majority winner, we discard the example.</p>
</div>
</li>
<li id="S3.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i2.p1" class="ltx_para">
<p id="S3.I2.i2.p1.2" class="ltx_p"><span id="S3.I2.i2.p1.2.1" class="ltx_text ltx_font_italic">self-consistency under input perturbation</span>: we perturb the input <math id="S3.I2.i2.p1.1.m1.1" class="ltx_Math" alttext="x^{i}" display="inline"><semantics id="S3.I2.i2.p1.1.m1.1a"><msup id="S3.I2.i2.p1.1.m1.1.1" xref="S3.I2.i2.p1.1.m1.1.1.cmml"><mi id="S3.I2.i2.p1.1.m1.1.1.2" xref="S3.I2.i2.p1.1.m1.1.1.2.cmml">x</mi><mi id="S3.I2.i2.p1.1.m1.1.1.3" xref="S3.I2.i2.p1.1.m1.1.1.3.cmml">i</mi></msup><annotation-xml encoding="MathML-Content" id="S3.I2.i2.p1.1.m1.1b"><apply id="S3.I2.i2.p1.1.m1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.I2.i2.p1.1.m1.1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.1.1">superscript</csymbol><ci id="S3.I2.i2.p1.1.m1.1.1.2.cmml" xref="S3.I2.i2.p1.1.m1.1.1.2">𝑥</ci><ci id="S3.I2.i2.p1.1.m1.1.1.3.cmml" xref="S3.I2.i2.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i2.p1.1.m1.1c">x^{i}</annotation></semantics></math> in such a way that the output should not change, e.g. changing the order of multiple-choice items in the prompt, and compute <math id="S3.I2.i2.p1.2.m2.1" class="ltx_Math" alttext="S_{\text{II}}" display="inline"><semantics id="S3.I2.i2.p1.2.m2.1a"><msub id="S3.I2.i2.p1.2.m2.1.1" xref="S3.I2.i2.p1.2.m2.1.1.cmml"><mi id="S3.I2.i2.p1.2.m2.1.1.2" xref="S3.I2.i2.p1.2.m2.1.1.2.cmml">S</mi><mtext id="S3.I2.i2.p1.2.m2.1.1.3" xref="S3.I2.i2.p1.2.m2.1.1.3a.cmml">II</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.I2.i2.p1.2.m2.1b"><apply id="S3.I2.i2.p1.2.m2.1.1.cmml" xref="S3.I2.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.I2.i2.p1.2.m2.1.1.1.cmml" xref="S3.I2.i2.p1.2.m2.1.1">subscript</csymbol><ci id="S3.I2.i2.p1.2.m2.1.1.2.cmml" xref="S3.I2.i2.p1.2.m2.1.1.2">𝑆</ci><ci id="S3.I2.i2.p1.2.m2.1.1.3a.cmml" xref="S3.I2.i2.p1.2.m2.1.1.3"><mtext mathsize="70%" id="S3.I2.i2.p1.2.m2.1.1.3.cmml" xref="S3.I2.i2.p1.2.m2.1.1.3">II</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i2.p1.2.m2.1c">S_{\text{II}}</annotation></semantics></math> for each perturbation; if the outputs do not agree, we discard the example.</p>
</div>
</li>
</ul>
</div>
<div id="S3.SS2.p4" class="ltx_para">
<p id="S3.SS2.p4.6" class="ltx_p">After that, we end up with the synthetic dataset
<math id="S3.SS2.p4.1.m1.2" class="ltx_Math" alttext="(\mathcal{X}_{S_{\text{II}}},\mathcal{Y}_{S_{\text{II}}})" display="inline"><semantics id="S3.SS2.p4.1.m1.2a"><mrow id="S3.SS2.p4.1.m1.2.2.2" xref="S3.SS2.p4.1.m1.2.2.3.cmml"><mo stretchy="false" id="S3.SS2.p4.1.m1.2.2.2.3" xref="S3.SS2.p4.1.m1.2.2.3.cmml">(</mo><msub id="S3.SS2.p4.1.m1.1.1.1.1" xref="S3.SS2.p4.1.m1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p4.1.m1.1.1.1.1.2" xref="S3.SS2.p4.1.m1.1.1.1.1.2.cmml">𝒳</mi><msub id="S3.SS2.p4.1.m1.1.1.1.1.3" xref="S3.SS2.p4.1.m1.1.1.1.1.3.cmml"><mi id="S3.SS2.p4.1.m1.1.1.1.1.3.2" xref="S3.SS2.p4.1.m1.1.1.1.1.3.2.cmml">S</mi><mtext id="S3.SS2.p4.1.m1.1.1.1.1.3.3" xref="S3.SS2.p4.1.m1.1.1.1.1.3.3a.cmml">II</mtext></msub></msub><mo id="S3.SS2.p4.1.m1.2.2.2.4" xref="S3.SS2.p4.1.m1.2.2.3.cmml">,</mo><msub id="S3.SS2.p4.1.m1.2.2.2.2" xref="S3.SS2.p4.1.m1.2.2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p4.1.m1.2.2.2.2.2" xref="S3.SS2.p4.1.m1.2.2.2.2.2.cmml">𝒴</mi><msub id="S3.SS2.p4.1.m1.2.2.2.2.3" xref="S3.SS2.p4.1.m1.2.2.2.2.3.cmml"><mi id="S3.SS2.p4.1.m1.2.2.2.2.3.2" xref="S3.SS2.p4.1.m1.2.2.2.2.3.2.cmml">S</mi><mtext id="S3.SS2.p4.1.m1.2.2.2.2.3.3" xref="S3.SS2.p4.1.m1.2.2.2.2.3.3a.cmml">II</mtext></msub></msub><mo stretchy="false" id="S3.SS2.p4.1.m1.2.2.2.5" xref="S3.SS2.p4.1.m1.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.1.m1.2b"><interval closure="open" id="S3.SS2.p4.1.m1.2.2.3.cmml" xref="S3.SS2.p4.1.m1.2.2.2"><apply id="S3.SS2.p4.1.m1.1.1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.1.1.1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.p4.1.m1.1.1.1.1.2.cmml" xref="S3.SS2.p4.1.m1.1.1.1.1.2">𝒳</ci><apply id="S3.SS2.p4.1.m1.1.1.1.1.3.cmml" xref="S3.SS2.p4.1.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.1.1.1.1.3.1.cmml" xref="S3.SS2.p4.1.m1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p4.1.m1.1.1.1.1.3.2.cmml" xref="S3.SS2.p4.1.m1.1.1.1.1.3.2">𝑆</ci><ci id="S3.SS2.p4.1.m1.1.1.1.1.3.3a.cmml" xref="S3.SS2.p4.1.m1.1.1.1.1.3.3"><mtext mathsize="50%" id="S3.SS2.p4.1.m1.1.1.1.1.3.3.cmml" xref="S3.SS2.p4.1.m1.1.1.1.1.3.3">II</mtext></ci></apply></apply><apply id="S3.SS2.p4.1.m1.2.2.2.2.cmml" xref="S3.SS2.p4.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.2.2.2.2.1.cmml" xref="S3.SS2.p4.1.m1.2.2.2.2">subscript</csymbol><ci id="S3.SS2.p4.1.m1.2.2.2.2.2.cmml" xref="S3.SS2.p4.1.m1.2.2.2.2.2">𝒴</ci><apply id="S3.SS2.p4.1.m1.2.2.2.2.3.cmml" xref="S3.SS2.p4.1.m1.2.2.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.2.2.2.2.3.1.cmml" xref="S3.SS2.p4.1.m1.2.2.2.2.3">subscript</csymbol><ci id="S3.SS2.p4.1.m1.2.2.2.2.3.2.cmml" xref="S3.SS2.p4.1.m1.2.2.2.2.3.2">𝑆</ci><ci id="S3.SS2.p4.1.m1.2.2.2.2.3.3a.cmml" xref="S3.SS2.p4.1.m1.2.2.2.2.3.3"><mtext mathsize="50%" id="S3.SS2.p4.1.m1.2.2.2.2.3.3.cmml" xref="S3.SS2.p4.1.m1.2.2.2.2.3.3">II</mtext></ci></apply></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.1.m1.2c">(\mathcal{X}_{S_{\text{II}}},\mathcal{Y}_{S_{\text{II}}})</annotation></semantics></math>,
where <math id="S3.SS2.p4.2.m2.1" class="ltx_Math" alttext="\mathcal{X}_{S_{\text{II}}}" display="inline"><semantics id="S3.SS2.p4.2.m2.1a"><msub id="S3.SS2.p4.2.m2.1.1" xref="S3.SS2.p4.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p4.2.m2.1.1.2" xref="S3.SS2.p4.2.m2.1.1.2.cmml">𝒳</mi><msub id="S3.SS2.p4.2.m2.1.1.3" xref="S3.SS2.p4.2.m2.1.1.3.cmml"><mi id="S3.SS2.p4.2.m2.1.1.3.2" xref="S3.SS2.p4.2.m2.1.1.3.2.cmml">S</mi><mtext id="S3.SS2.p4.2.m2.1.1.3.3" xref="S3.SS2.p4.2.m2.1.1.3.3a.cmml">II</mtext></msub></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.2.m2.1b"><apply id="S3.SS2.p4.2.m2.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.1.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.p4.2.m2.1.1.2.cmml" xref="S3.SS2.p4.2.m2.1.1.2">𝒳</ci><apply id="S3.SS2.p4.2.m2.1.1.3.cmml" xref="S3.SS2.p4.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.1.1.3.1.cmml" xref="S3.SS2.p4.2.m2.1.1.3">subscript</csymbol><ci id="S3.SS2.p4.2.m2.1.1.3.2.cmml" xref="S3.SS2.p4.2.m2.1.1.3.2">𝑆</ci><ci id="S3.SS2.p4.2.m2.1.1.3.3a.cmml" xref="S3.SS2.p4.2.m2.1.1.3.3"><mtext mathsize="50%" id="S3.SS2.p4.2.m2.1.1.3.3.cmml" xref="S3.SS2.p4.2.m2.1.1.3.3">II</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.2.m2.1c">\mathcal{X}_{S_{\text{II}}}</annotation></semantics></math> is a filtered subset of <math id="S3.SS2.p4.3.m3.1" class="ltx_Math" alttext="\mathcal{X}" display="inline"><semantics id="S3.SS2.p4.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p4.3.m3.1.1" xref="S3.SS2.p4.3.m3.1.1.cmml">𝒳</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.3.m3.1b"><ci id="S3.SS2.p4.3.m3.1.1.cmml" xref="S3.SS2.p4.3.m3.1.1">𝒳</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.3.m3.1c">\mathcal{X}</annotation></semantics></math> with targets <math id="S3.SS2.p4.4.m4.1" class="ltx_Math" alttext="\mathcal{Y}_{S_{\text{II}}}" display="inline"><semantics id="S3.SS2.p4.4.m4.1a"><msub id="S3.SS2.p4.4.m4.1.1" xref="S3.SS2.p4.4.m4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p4.4.m4.1.1.2" xref="S3.SS2.p4.4.m4.1.1.2.cmml">𝒴</mi><msub id="S3.SS2.p4.4.m4.1.1.3" xref="S3.SS2.p4.4.m4.1.1.3.cmml"><mi id="S3.SS2.p4.4.m4.1.1.3.2" xref="S3.SS2.p4.4.m4.1.1.3.2.cmml">S</mi><mtext id="S3.SS2.p4.4.m4.1.1.3.3" xref="S3.SS2.p4.4.m4.1.1.3.3a.cmml">II</mtext></msub></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.4.m4.1b"><apply id="S3.SS2.p4.4.m4.1.1.cmml" xref="S3.SS2.p4.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.4.m4.1.1.1.cmml" xref="S3.SS2.p4.4.m4.1.1">subscript</csymbol><ci id="S3.SS2.p4.4.m4.1.1.2.cmml" xref="S3.SS2.p4.4.m4.1.1.2">𝒴</ci><apply id="S3.SS2.p4.4.m4.1.1.3.cmml" xref="S3.SS2.p4.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p4.4.m4.1.1.3.1.cmml" xref="S3.SS2.p4.4.m4.1.1.3">subscript</csymbol><ci id="S3.SS2.p4.4.m4.1.1.3.2.cmml" xref="S3.SS2.p4.4.m4.1.1.3.2">𝑆</ci><ci id="S3.SS2.p4.4.m4.1.1.3.3a.cmml" xref="S3.SS2.p4.4.m4.1.1.3.3"><mtext mathsize="50%" id="S3.SS2.p4.4.m4.1.1.3.3.cmml" xref="S3.SS2.p4.4.m4.1.1.3.3">II</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.4.m4.1c">\mathcal{Y}_{S_{\text{II}}}</annotation></semantics></math>.
The final step is then supervised fine-tuning of the LLM with parameters <math id="S3.SS2.p4.5.m5.1" class="ltx_Math" alttext="p_{\theta}" display="inline"><semantics id="S3.SS2.p4.5.m5.1a"><msub id="S3.SS2.p4.5.m5.1.1" xref="S3.SS2.p4.5.m5.1.1.cmml"><mi id="S3.SS2.p4.5.m5.1.1.2" xref="S3.SS2.p4.5.m5.1.1.2.cmml">p</mi><mi id="S3.SS2.p4.5.m5.1.1.3" xref="S3.SS2.p4.5.m5.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.5.m5.1b"><apply id="S3.SS2.p4.5.m5.1.1.cmml" xref="S3.SS2.p4.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.5.m5.1.1.1.cmml" xref="S3.SS2.p4.5.m5.1.1">subscript</csymbol><ci id="S3.SS2.p4.5.m5.1.1.2.cmml" xref="S3.SS2.p4.5.m5.1.1.2">𝑝</ci><ci id="S3.SS2.p4.5.m5.1.1.3.cmml" xref="S3.SS2.p4.5.m5.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.5.m5.1c">p_{\theta}</annotation></semantics></math> using this distilled training set. We typically initialize this model from the current state <math id="S3.SS2.p4.6.m6.1" class="ltx_Math" alttext="p_{\theta}" display="inline"><semantics id="S3.SS2.p4.6.m6.1a"><msub id="S3.SS2.p4.6.m6.1.1" xref="S3.SS2.p4.6.m6.1.1.cmml"><mi id="S3.SS2.p4.6.m6.1.1.2" xref="S3.SS2.p4.6.m6.1.1.2.cmml">p</mi><mi id="S3.SS2.p4.6.m6.1.1.3" xref="S3.SS2.p4.6.m6.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.6.m6.1b"><apply id="S3.SS2.p4.6.m6.1.1.cmml" xref="S3.SS2.p4.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.6.m6.1.1.1.cmml" xref="S3.SS2.p4.6.m6.1.1">subscript</csymbol><ci id="S3.SS2.p4.6.m6.1.1.2.cmml" xref="S3.SS2.p4.6.m6.1.1.2">𝑝</ci><ci id="S3.SS2.p4.6.m6.1.1.3.cmml" xref="S3.SS2.p4.6.m6.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.6.m6.1c">p_{\theta}</annotation></semantics></math> and continue training with the new dataset.</p>
</div>
<div id="S3.SS2.p5" class="ltx_para">
<p id="S3.SS2.p5.1" class="ltx_p">After fine-tuning we obtain an LLM <math id="S3.SS2.p5.1.m1.1" class="ltx_Math" alttext="\hat{p_{\theta}}" display="inline"><semantics id="S3.SS2.p5.1.m1.1a"><mover accent="true" id="S3.SS2.p5.1.m1.1.1" xref="S3.SS2.p5.1.m1.1.1.cmml"><msub id="S3.SS2.p5.1.m1.1.1.2" xref="S3.SS2.p5.1.m1.1.1.2.cmml"><mi id="S3.SS2.p5.1.m1.1.1.2.2" xref="S3.SS2.p5.1.m1.1.1.2.2.cmml">p</mi><mi id="S3.SS2.p5.1.m1.1.1.2.3" xref="S3.SS2.p5.1.m1.1.1.2.3.cmml">θ</mi></msub><mo id="S3.SS2.p5.1.m1.1.1.1" xref="S3.SS2.p5.1.m1.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.1.m1.1b"><apply id="S3.SS2.p5.1.m1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1"><ci id="S3.SS2.p5.1.m1.1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1.1">^</ci><apply id="S3.SS2.p5.1.m1.1.1.2.cmml" xref="S3.SS2.p5.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p5.1.m1.1.1.2.1.cmml" xref="S3.SS2.p5.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.p5.1.m1.1.1.2.2.cmml" xref="S3.SS2.p5.1.m1.1.1.2.2">𝑝</ci><ci id="S3.SS2.p5.1.m1.1.1.2.3.cmml" xref="S3.SS2.p5.1.m1.1.1.2.3">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.1.m1.1c">\hat{p_{\theta}}</annotation></semantics></math> which is a System 1 model that is expected to provide outputs and performance gains similar to the
evaluated System 2 model.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Training and Evaluation Setup</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We use Llama-2-70B-chat <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib34" title="" class="ltx_ref">2023</a>)</cite> as the base model for all our experiments.
We require a base model of sufficient power that it can be performant as a System 2 model, but also have open weights that can be fine-tuned, hence this choice. We consider several System 2 methods, including Rephrase and Respond (RaR), System 2 Attention (S2A), Branch-Solve-Merge (BSM), and Chain-of-Thought (CoT), focusing on tasks where each method has demonstrated strong performance. For System 1, we conduct zero-shot inference using the instruction-tuned base model as a standard baseline.
We report task-specific metrics for each task, and the “#Tokens” metric which measures the average number of tokens generated per input across the evaluation set. For System 2 methods this includes both intermediate token generations as well as the final output token generations. Detailed descriptions of the experimental setups are available in the Appendix <a href="#A1.SS2" title="A.2 Experiment Details ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.2</span></a>.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Rephrase and Respond Distillation</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">Rephrase and Response (RaR) <cite class="ltx_cite ltx_citemacro_citep">(Deng et al., <a href="#bib.bib12" title="" class="ltx_ref">2023a</a>)</cite> is a System 2 method that first prompts the language model to rephrase the original question with further elaboration, and then secondly to generate a response based on the rephrased question with the aim that this provides superior output. The authors introduce two approaches, 1-step RaR and 2-step RaR, where the latter involves two separate prompts rather than a combined one as in the former, see Appendix <a href="#A1.SS1" title="A.1 Prompts ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.1</span></a> for specific prompts.
They find that 2-step RaR significantly improves performance on several reasoning tasks that are challenging for the baseline LLM.
We consider two tasks from the original paper where it performed well: the last letter concatenation task and coin flip reasoning.
We then assess whether it is possible to distill this System 2 approach.</p>
</div>
<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Distillation Data</h5>

<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px1.p1.1" class="ltx_p">We build the System 2 distillation dataset for RaR using <span id="S4.SS2.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_italic">self-consistency of outputs</span>. For each input, we conduct eight sampling iterations for the last letter task and eight for each stage of the coin flip task.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>This approach was adopted after observing that sampling just once for the rephrase stage yielded suboptimal results.</span></span></span>
We then apply a majority vote to determine the final output.</p>
</div>
</section>
<section id="S4.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.2.1 </span>Last letter Concatenation Task</h4>

<div id="S4.SS2.SSS1.p1" class="ltx_para">
<p id="S4.SS2.SSS1.p1.1" class="ltx_p">This task focuses on symbolic reasoning, requiring the model to concatenate the last letters of given words. For instance, the instruction: “Take the last letters of the words in ‘Edgar Bob’ and concatenate them.” As demonstrated in <cite class="ltx_cite ltx_citemacro_citet">Deng et al. (<a href="#bib.bib12" title="" class="ltx_ref">2023a</a>)</cite>, this task benefits significantly from the application of the RaR method. We compiled a dataset by randomly selecting 1200 unique English words. Using this, we constructed 200 samples each for training, validation, and test.</p>
</div>
<section id="S4.SS2.SSS1.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Results</h5>

<div id="S4.SS2.SSS1.Px1.p1" class="ltx_para">
<p id="S4.SS2.SSS1.Px1.p1.1" class="ltx_p">Overall results are given in <a href="#S4.T1" title="Table 1 ‣ Analysis &amp; Ablations ‣ 4.2.1 Last letter Concatenation Task ‣ 4.2 Rephrase and Respond Distillation ‣ 4 Experiments ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 1</span></a>. The baseline System 1 model (Llama-2-70B-chat) achieves an accuracy of 30.0%, and is outperformed by the System 2 methods of 1-Step and 2-Step RaR (39.5% and 44.5%, respectively).
Distilling the 2-Step RaR method back into a System 1 Llama-2-70B-chat model via our unsupervised technique, we achieve a remarkable accuracy of 98.0%.
The model can effectively learn from this training data how to solve the task, in comparison to the zero-shot chat model.
Distillation of Rephrase and Respond effectively inherits the advantages of both System 2 and System 1. It maintains the accuracy benefits of System 2, while its inference cost is comparable to that of System 1 (see # of generated Tokens).</p>
</div>
</section>
<section id="S4.SS2.SSS1.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Analysis &amp; Ablations</h5>

<div id="S4.SS2.SSS1.Px2.p1" class="ltx_para">
<p id="S4.SS2.SSS1.Px2.p1.1" class="ltx_p">To evaluate the effectiveness and necessity of our  unsupervised curation step using <span id="S4.SS2.SSS1.Px2.p1.1.1" class="ltx_text ltx_font_italic">self-consistency of outputs</span>
we conducted an ablation study by creating a distillation dataset without applying the self-consistency filter. When we distilled the System 2 model using this unfiltered dataset under the same setting, it achieved an exact match accuracy of 87.5% (with 98% for the filtered version). This comparison underscores the critical role of consistency filtering. Nevethess, in both cases constructing training data does improve results over zero-shot performance.
We also attempted to distill the System 1 predictions using the same filtering technique, which results in a lower accuracy of 69.5%.</p>
</div>
<figure id="S4.T1" class="ltx_table">
<div id="S4.T1.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:211.8pt;height:136.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-33.4pt,21.6pt) scale(0.76,0.76) ;">
<table id="S4.T1.2.2" class="ltx_tabular ltx_align_middle">
<tr id="S4.T1.2.2.3" class="ltx_tr">
<td id="S4.T1.2.2.3.1" class="ltx_td ltx_border_tt"></td>
<td id="S4.T1.2.2.3.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">Last Letter</td>
<td id="S4.T1.2.2.3.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">Coin Flip</td>
</tr>
<tr id="S4.T1.2.2.2" class="ltx_tr">
<td id="S4.T1.2.2.2.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t">Acc<math id="S4.T1.1.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T1.1.1.1.1.m1.1a"><mo stretchy="false" id="S4.T1.1.1.1.1.m1.1.1" xref="S4.T1.1.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T1.1.1.1.1.m1.1b"><ci id="S4.T1.1.1.1.1.m1.1.1.cmml" xref="S4.T1.1.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.1.1.1.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T1.2.2.2.4" class="ltx_td ltx_align_center ltx_border_t">#Tokens</td>
<td id="S4.T1.2.2.2.2" class="ltx_td ltx_align_center ltx_border_t">Acc<math id="S4.T1.2.2.2.2.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T1.2.2.2.2.m1.1a"><mo stretchy="false" id="S4.T1.2.2.2.2.m1.1.1" xref="S4.T1.2.2.2.2.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.2.2.m1.1b"><ci id="S4.T1.2.2.2.2.m1.1.1.cmml" xref="S4.T1.2.2.2.2.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.2.2.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T1.2.2.2.5" class="ltx_td ltx_align_center ltx_border_t">#Tokens</td>
</tr>
<tr id="S4.T1.2.2.4" class="ltx_tr">
<td id="S4.T1.2.2.4.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T1.2.2.4.1.1" class="ltx_text ltx_font_italic">System 1</span></td>
<td id="S4.T1.2.2.4.2" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.2.2.4.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.2.2.4.4" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.2.2.4.5" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T1.2.2.5" class="ltx_tr">
<td id="S4.T1.2.2.5.1" class="ltx_td ltx_align_left">Llama-2-70B-chat</td>
<td id="S4.T1.2.2.5.2" class="ltx_td ltx_align_center">30.0%</td>
<td id="S4.T1.2.2.5.3" class="ltx_td ltx_align_center">27.1</td>
<td id="S4.T1.2.2.5.4" class="ltx_td ltx_align_center">56.1%</td>
<td id="S4.T1.2.2.5.5" class="ltx_td ltx_align_center">61.9</td>
</tr>
<tr id="S4.T1.2.2.6" class="ltx_tr">
<td id="S4.T1.2.2.6.1" class="ltx_td ltx_align_left">Distill System 1</td>
<td id="S4.T1.2.2.6.2" class="ltx_td ltx_align_center">69.5%</td>
<td id="S4.T1.2.2.6.3" class="ltx_td ltx_align_center">24.4</td>
<td id="S4.T1.2.2.6.4" class="ltx_td ltx_align_center">54.5%</td>
<td id="S4.T1.2.2.6.5" class="ltx_td ltx_align_center">30.4</td>
</tr>
<tr id="S4.T1.2.2.7" class="ltx_tr">
<td id="S4.T1.2.2.7.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T1.2.2.7.1.1" class="ltx_text ltx_font_italic">System 2</span></td>
<td id="S4.T1.2.2.7.2" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.2.2.7.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.2.2.7.4" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.2.2.7.5" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T1.2.2.8" class="ltx_tr">
<td id="S4.T1.2.2.8.1" class="ltx_td ltx_align_left">1-Step RaR</td>
<td id="S4.T1.2.2.8.2" class="ltx_td ltx_align_center">39.5%</td>
<td id="S4.T1.2.2.8.3" class="ltx_td ltx_align_center">106.6</td>
<td id="S4.T1.2.2.8.4" class="ltx_td ltx_align_center">58.5%</td>
<td id="S4.T1.2.2.8.5" class="ltx_td ltx_align_center">158.9</td>
</tr>
<tr id="S4.T1.2.2.9" class="ltx_tr">
<td id="S4.T1.2.2.9.1" class="ltx_td ltx_align_left">2-Step RaR</td>
<td id="S4.T1.2.2.9.2" class="ltx_td ltx_align_center">44.5%</td>
<td id="S4.T1.2.2.9.3" class="ltx_td ltx_align_center">41.5</td>
<td id="S4.T1.2.2.9.4" class="ltx_td ltx_align_center">77.2%</td>
<td id="S4.T1.2.2.9.5" class="ltx_td ltx_align_center">112.4</td>
</tr>
<tr id="S4.T1.2.2.10" class="ltx_tr">
<td id="S4.T1.2.2.10.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T1.2.2.10.1.1" class="ltx_text ltx_font_italic">Distill System 2</span></td>
<td id="S4.T1.2.2.10.2" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.2.2.10.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.2.2.10.4" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.2.2.10.5" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T1.2.2.11" class="ltx_tr">
<td id="S4.T1.2.2.11.1" class="ltx_td ltx_align_left ltx_border_bb">Distill 2-Step RaR</td>
<td id="S4.T1.2.2.11.2" class="ltx_td ltx_align_center ltx_border_bb">98.0%</td>
<td id="S4.T1.2.2.11.3" class="ltx_td ltx_align_center ltx_border_bb">25.5</td>
<td id="S4.T1.2.2.11.4" class="ltx_td ltx_align_center ltx_border_bb">75.69%</td>
<td id="S4.T1.2.2.11.5" class="ltx_td ltx_align_center ltx_border_bb">50.3</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span><span id="S4.T1.4.1" class="ltx_text ltx_font_bold">System 2 Distillation of Rephrase and Respond</span>:
Coin Flip and Last Letter Concatenation tasks.
We report exact match (EM) test accuracy and number of generated (intermediate and output) tokens.</figcaption>
</figure>
</section>
</section>
<section id="S4.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.2.2 </span>Coin Flip Reasoning Task</h4>

<div id="S4.SS2.SSS2.p1" class="ltx_para">
<p id="S4.SS2.SSS2.p1.1" class="ltx_p">This symbolic reasoning task has frequently been tested in research, including in <cite class="ltx_cite ltx_citemacro_citet">Wei et al. (<a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Deng et al. (<a href="#bib.bib12" title="" class="ltx_ref">2023a</a>)</cite>. It involves determining the final face (heads or tails) of a coin, starting from a known initial position after a series of flips described in natural language
, such as <span id="S4.SS2.SSS2.p1.1.1" class="ltx_text ltx_font_italic">“A coin is heads up. Roxas does not flip the coin. Schneiderman does not flip the coin. Is the coin still heads up?”</span>
<cite class="ltx_cite ltx_citemacro_citet">Deng et al. (<a href="#bib.bib12" title="" class="ltx_ref">2023a</a>)</cite> showed that even strong language models do not succeed at this task, whereas applying the RaR method improves their performance.
There are 20k training examples, which we use for unsupervised learning (without labels), 3.33k validation and 1.33k test examples.</p>
</div>
<section id="S4.SS2.SSS2.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Results</h5>

<div id="S4.SS2.SSS2.Px1.p1" class="ltx_para">
<p id="S4.SS2.SSS2.Px1.p1.1" class="ltx_p">Overall results are given in
<a href="#S4.T1" title="Table 1 ‣ Analysis &amp; Ablations ‣ 4.2.1 Last letter Concatenation Task ‣ 4.2 Rephrase and Respond Distillation ‣ 4 Experiments ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 1</span></a>.
Llama-2-70B-chat (zero-shot) has a success rate of 56.1% on this task, while 1-Step and 2-Step RaR have success rates of 58.5% and 77.2% respectively. We thus only see a large improvement with the 2-Step method.
Distilling 2-Step RaR back into a system 1 Llama-2-70B-chat via our unsupervised technique yields
75.69%. Hence, we find that
our distilled System 2 model delivers performance comparable to that of System 2 (2 Step RaR), but without the need to execute the LLM program with 2 prompts (see # of generated Tokens).</p>
</div>
</section>
<section id="S4.SS2.SSS2.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Analysis &amp; Ablations</h5>

<div id="S4.SS2.SSS2.Px2.p1" class="ltx_para">
<p id="S4.SS2.SSS2.Px2.p1.1" class="ltx_p">The RaR method in <cite class="ltx_cite ltx_citemacro_citet">Deng et al. (<a href="#bib.bib12" title="" class="ltx_ref">2023a</a>)</cite> incorporates prompt engineering tricks, such as appending phrases like “Flip means reverse. Answer the Yes or No question” to the original query, which has been shown to enhance model performance. Following their approach, we evaluated model performance using different prompts, see <a href="#A1.T7" title="Table 7 ‣ S2A ‣ A.2 Experiment Details ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 7</span></a>. When testing the Llama-2-70B-chat model (System 1) with prompts like “Flip means reverse” and “Flip means reverse. Answer the Yes or No question,” we observed a significant improvement in performance, from 56.11% to 66.84%. This highlights the critical role of prompt selection in optimizing the performance of System 1 models. However, this reliance on prompt engineering also represents a limitation, necessitating additional human effort.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<div id="S4.T2.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:224.6pt;height:91.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-19.8pt,8.1pt) scale(0.85,0.85) ;">
<table id="S4.T2.2.2" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.2.2.2" class="ltx_tr">
<td id="S4.T2.2.2.2.3" class="ltx_td ltx_border_tt"></td>
<td id="S4.T2.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt">Acc<math id="S4.T2.1.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T2.1.1.1.1.m1.1a"><mo stretchy="false" id="S4.T2.1.1.1.1.m1.1.1" xref="S4.T2.1.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T2.1.1.1.1.m1.1b"><ci id="S4.T2.1.1.1.1.m1.1.1.cmml" xref="S4.T2.1.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.1.1.1.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T2.2.2.2.2" class="ltx_td ltx_align_center ltx_border_tt">Acc<math id="S4.T2.2.2.2.2.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T2.2.2.2.2.m1.1a"><mo stretchy="false" id="S4.T2.2.2.2.2.m1.1.1" xref="S4.T2.2.2.2.2.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T2.2.2.2.2.m1.1b"><ci id="S4.T2.2.2.2.2.m1.1.1.cmml" xref="S4.T2.2.2.2.2.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.2.2.2.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T2.2.2.2.4" class="ltx_td ltx_border_tt"></td>
</tr>
<tr id="S4.T2.2.2.3" class="ltx_tr">
<td id="S4.T2.2.2.3.1" class="ltx_td ltx_align_left">Model</td>
<td id="S4.T2.2.2.3.2" class="ltx_td ltx_align_center">(biased)</td>
<td id="S4.T2.2.2.3.3" class="ltx_td ltx_align_center">(unbiased)</td>
<td id="S4.T2.2.2.3.4" class="ltx_td ltx_align_center">#Tokens</td>
</tr>
<tr id="S4.T2.2.2.4" class="ltx_tr">
<td id="S4.T2.2.2.4.1" class="ltx_td ltx_align_left ltx_border_t">
<span id="S4.T2.2.2.4.1.1" class="ltx_text ltx_font_italic">System 1</span> (Zero-shot)</td>
<td id="S4.T2.2.2.4.2" class="ltx_td ltx_align_center ltx_border_t">51.6%</td>
<td id="S4.T2.2.2.4.3" class="ltx_td ltx_align_center ltx_border_t">73.8%</td>
<td id="S4.T2.2.2.4.4" class="ltx_td ltx_align_center ltx_border_t">165</td>
</tr>
<tr id="S4.T2.2.2.5" class="ltx_tr">
<td id="S4.T2.2.2.5.1" class="ltx_td ltx_align_left">
<span id="S4.T2.2.2.5.1.1" class="ltx_text ltx_font_italic">System 2</span> (S2A)</td>
<td id="S4.T2.2.2.5.2" class="ltx_td ltx_align_center">76.0%</td>
<td id="S4.T2.2.2.5.3" class="ltx_td ltx_align_center">69.3%</td>
<td id="S4.T2.2.2.5.4" class="ltx_td ltx_align_center">147</td>
</tr>
<tr id="S4.T2.2.2.6" class="ltx_tr">
<td id="S4.T2.2.2.6.1" class="ltx_td ltx_align_left">Distill S2A</td>
<td id="S4.T2.2.2.6.2" class="ltx_td ltx_align_center">81.3%</td>
<td id="S4.T2.2.2.6.3" class="ltx_td ltx_align_center">78.6%</td>
<td id="S4.T2.2.2.6.4" class="ltx_td ltx_align_center">56</td>
</tr>
<tr id="S4.T2.2.2.7" class="ltx_tr">
<td id="S4.T2.2.2.7.1" class="ltx_td ltx_align_left ltx_border_bb">Distill S2A (no USC)</td>
<td id="S4.T2.2.2.7.2" class="ltx_td ltx_align_center ltx_border_bb">78.6%</td>
<td id="S4.T2.2.2.7.3" class="ltx_td ltx_align_center ltx_border_bb">75.3%</td>
<td id="S4.T2.2.2.7.4" class="ltx_td ltx_align_center ltx_border_bb">58</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>
<span id="S4.T2.4.1" class="ltx_text ltx_font_bold">Distillation of System 2 Attention</span>: TriviaQA task, reporting
accuracies on the biased and unbiased eval sets.

</figcaption>
</figure>
<div id="S4.SS2.SSS2.Px2.p2" class="ltx_para">
<p id="S4.SS2.SSS2.Px2.p2.1" class="ltx_p">We also attempted to distill the System 1 model, which gave poor performance. In this case, we also observed fluctuations in performance with different prompts. In contrast, the distilled System 2 model demonstrated consistent performance across various prompts, with a lower sensitivity to prompt variations. This consistency indicates that extensive prompt engineering might not be essential for the distilled System 2 model.</p>
</div>
</section>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>System 2 Attention Distillation</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">Weston and Sukhbaatar (<a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite> proposed System 2 Attention (S2A), a method that helps to reduce models’ reasoning pitfalls such as relying on biased information in the input or attending to irrelevant context. S2A is a two-stage inference method where the first stage rewrites the input so that it does not contain undesired information such as bias or irrelevant context, and the second stage attends to the shorter rewritten context (in contrast to RaR which expands the context), see <a href="#A1.F6" title="Figure 6 ‣ A.1 Prompts ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 6</span></a>.
In this work we verify the feasibility of distilling S2A into System 1. In particular, we focus on the SycophancyEval question answering task <cite class="ltx_cite ltx_citemacro_citep">(Sharma et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite> that contains biased information in the input that is known to hurt LLM performance. We use 6668 examples from SycophancyEval as unlabeled training data, and 400 examples for evaluation, where the latter are split into biased inputs (350) and without bias (50).</p>
</div>
<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Distillation data</h5>

<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px1.p1.1" class="ltx_p">We use universal self-consistency (USC) <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite> to select high quality targets. Specifically, we sample 20 generations and then use the
Llama-70B-chat model with a USC prompt (provided in <a href="#A1.F12" title="Figure 12 ‣ A.1 Prompts ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 12</span></a>)
to compose a self-consistent (majority) final answer that is used as the distillation target.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Results</h5>

<div id="S4.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px2.p1.1" class="ltx_p">The results are provided in <a href="#S4.T2" title="Table 2 ‣ Analysis &amp; Ablations ‣ 4.2.2 Coin Flip Reasoning Task ‣ 4.2 Rephrase and Respond Distillation ‣ 4 Experiments ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 2</span></a>,
reporting average accuracy over 3 random seeds.
The baseline (System 1) LLM has low accuracy on the biased portion as expected, being susceptible to biased inputs. S2A improves performance dramatically for biased inputs. System 2 distillation shows similarly strong performance as the System 2 approach. There is, however, a signification reduction in the average number of tokens used compared to both the baseline and the S2A model. This is because biased inputs tend to make the baseline LLM generate more output tokens, while S2A has to generate intermediate tokens as well. <a href="#A1.F11" title="Figure 11 ‣ A.1 Prompts ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 11</span></a> shows a representative example.
Finally, we show that using USC for distillation is important for overall results, by also reporting results without USC (last row), where the latter provides inferior results. This highlights the importance of the distillation data quality that is used during fine-tuning.</p>
</div>
<figure id="S4.T3" class="ltx_table">
<div id="S4.T3.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:468.9pt;height:152.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-78.1pt,25.4pt) scale(0.75,0.75) ;">
<table id="S4.T3.4.4" class="ltx_tabular ltx_align_middle">
<tr id="S4.T3.4.4.5" class="ltx_tr">
<td id="S4.T3.4.4.5.1" class="ltx_td ltx_border_tt"></td>
<td id="S4.T3.4.4.5.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T3.4.4.5.2.1" class="ltx_text ltx_font_bold">OASST2 Eval</span></td>
<td id="S4.T3.4.4.5.3" class="ltx_td ltx_border_tt"></td>
<td id="S4.T3.4.4.5.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T3.4.4.5.4.1" class="ltx_text ltx_font_bold">MT-bench Eval</span></td>
</tr>
<tr id="S4.T3.4.4.4" class="ltx_tr">
<td id="S4.T3.4.4.4.5" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t">Agreement <math id="S4.T3.1.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T3.1.1.1.1.m1.1a"><mo stretchy="false" id="S4.T3.1.1.1.1.m1.1.1" xref="S4.T3.1.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T3.1.1.1.1.m1.1b"><ci id="S4.T3.1.1.1.1.m1.1.1.cmml" xref="S4.T3.1.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.1.1.1.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T3.2.2.2.2" class="ltx_td ltx_align_center ltx_border_t">
<span id="S4.T3.2.2.2.2.2" class="ltx_text"></span> <span id="S4.T3.2.2.2.2.1" class="ltx_text">
<span id="S4.T3.2.2.2.2.1.1" class="ltx_tabular ltx_align_middle">
<span id="S4.T3.2.2.2.2.1.1.1" class="ltx_tr">
<span id="S4.T3.2.2.2.2.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">% Inconsistent <math id="S4.T3.2.2.2.2.1.1.1.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S4.T3.2.2.2.2.1.1.1.1.m1.1a"><mo stretchy="false" id="S4.T3.2.2.2.2.1.1.1.1.m1.1.1" xref="S4.T3.2.2.2.2.1.1.1.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T3.2.2.2.2.1.1.1.1.m1.1b"><ci id="S4.T3.2.2.2.2.1.1.1.1.m1.1.1.cmml" xref="S4.T3.2.2.2.2.1.1.1.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.2.2.2.2.1.1.1.1.m1.1c">\downarrow</annotation></semantics></math></span></span>
</span></span><span id="S4.T3.2.2.2.2.3" class="ltx_text"></span></td>
<td id="S4.T3.4.4.4.6" class="ltx_td ltx_align_center ltx_border_t">
<span id="S4.T3.4.4.4.6.1" class="ltx_text"></span> <span id="S4.T3.4.4.4.6.2" class="ltx_text">
<span id="S4.T3.4.4.4.6.2.1" class="ltx_tabular ltx_align_middle">
<span id="S4.T3.4.4.4.6.2.1.1" class="ltx_tr">
<span id="S4.T3.4.4.4.6.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">#Tokens</span></span>
</span></span><span id="S4.T3.4.4.4.6.3" class="ltx_text"></span></td>
<td id="S4.T3.4.4.4.7" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.3.3.3.3" class="ltx_td ltx_align_center ltx_border_t">Agreement <math id="S4.T3.3.3.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T3.3.3.3.3.m1.1a"><mo stretchy="false" id="S4.T3.3.3.3.3.m1.1.1" xref="S4.T3.3.3.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T3.3.3.3.3.m1.1b"><ci id="S4.T3.3.3.3.3.m1.1.1.cmml" xref="S4.T3.3.3.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.3.3.3.3.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T3.4.4.4.4" class="ltx_td ltx_align_center ltx_border_t">
<span id="S4.T3.4.4.4.4.2" class="ltx_text"></span> <span id="S4.T3.4.4.4.4.1" class="ltx_text">
<span id="S4.T3.4.4.4.4.1.1" class="ltx_tabular ltx_align_middle">
<span id="S4.T3.4.4.4.4.1.1.1" class="ltx_tr">
<span id="S4.T3.4.4.4.4.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">% Inconsistent <math id="S4.T3.4.4.4.4.1.1.1.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S4.T3.4.4.4.4.1.1.1.1.m1.1a"><mo stretchy="false" id="S4.T3.4.4.4.4.1.1.1.1.m1.1.1" xref="S4.T3.4.4.4.4.1.1.1.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T3.4.4.4.4.1.1.1.1.m1.1b"><ci id="S4.T3.4.4.4.4.1.1.1.1.m1.1.1.cmml" xref="S4.T3.4.4.4.4.1.1.1.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.4.4.4.4.1.1.1.1.m1.1c">\downarrow</annotation></semantics></math></span></span>
</span></span><span id="S4.T3.4.4.4.4.3" class="ltx_text"></span></td>
<td id="S4.T3.4.4.4.8" class="ltx_td ltx_align_center ltx_border_t">
<span id="S4.T3.4.4.4.8.1" class="ltx_text"></span> <span id="S4.T3.4.4.4.8.2" class="ltx_text">
<span id="S4.T3.4.4.4.8.2.1" class="ltx_tabular ltx_align_middle">
<span id="S4.T3.4.4.4.8.2.1.1" class="ltx_tr">
<span id="S4.T3.4.4.4.8.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">#Tokens</span></span>
</span></span><span id="S4.T3.4.4.4.8.3" class="ltx_text"></span></td>
</tr>
<tr id="S4.T3.4.4.6" class="ltx_tr">
<td id="S4.T3.4.4.6.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T3.4.4.6.1.1" class="ltx_text ltx_font_italic">System 1</span></td>
<td id="S4.T3.4.4.6.2" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.6.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.6.4" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.6.5" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.6.6" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.6.7" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.6.8" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T3.4.4.7" class="ltx_tr">
<td id="S4.T3.4.4.7.1" class="ltx_td ltx_align_left">GPT-4-0125-preview</td>
<td id="S4.T3.4.4.7.2" class="ltx_td ltx_align_center">44.7%</td>
<td id="S4.T3.4.4.7.3" class="ltx_td ltx_align_center">35.5%</td>
<td id="S4.T3.4.4.7.4" class="ltx_td ltx_align_center"><span id="S4.T3.4.4.7.4.1" class="ltx_text ltx_font_bold">4</span></td>
<td id="S4.T3.4.4.7.5" class="ltx_td"></td>
<td id="S4.T3.4.4.7.6" class="ltx_td ltx_align_center">68.1%</td>
<td id="S4.T3.4.4.7.7" class="ltx_td ltx_align_center">25.6%</td>
<td id="S4.T3.4.4.7.8" class="ltx_td ltx_align_center"><span id="S4.T3.4.4.7.8.1" class="ltx_text ltx_font_bold">4</span></td>
</tr>
<tr id="S4.T3.4.4.8" class="ltx_tr">
<td id="S4.T3.4.4.8.1" class="ltx_td ltx_align_left">Llama-2-70B-chat</td>
<td id="S4.T3.4.4.8.2" class="ltx_td ltx_align_center">32.0%</td>
<td id="S4.T3.4.4.8.3" class="ltx_td ltx_align_center">56.7%</td>
<td id="S4.T3.4.4.8.4" class="ltx_td ltx_align_center"><span id="S4.T3.4.4.8.4.1" class="ltx_text ltx_font_bold">4</span></td>
<td id="S4.T3.4.4.8.5" class="ltx_td"></td>
<td id="S4.T3.4.4.8.6" class="ltx_td ltx_align_center">28.1%</td>
<td id="S4.T3.4.4.8.7" class="ltx_td ltx_align_center">80.9%</td>
<td id="S4.T3.4.4.8.8" class="ltx_td ltx_align_center"><span id="S4.T3.4.4.8.8.1" class="ltx_text ltx_font_bold">4</span></td>
</tr>
<tr id="S4.T3.4.4.9" class="ltx_tr">
<td id="S4.T3.4.4.9.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T3.4.4.9.1.1" class="ltx_text ltx_font_italic">System 2</span></td>
<td id="S4.T3.4.4.9.2" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.9.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.9.4" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.9.5" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.9.6" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.9.7" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.9.8" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T3.4.4.10" class="ltx_tr">
<td id="S4.T3.4.4.10.1" class="ltx_td ltx_align_left">CoT (GPT-4-0125-preview)</td>
<td id="S4.T3.4.4.10.2" class="ltx_td ltx_align_center">48.7%</td>
<td id="S4.T3.4.4.10.3" class="ltx_td ltx_align_center">28.2%</td>
<td id="S4.T3.4.4.10.4" class="ltx_td ltx_align_center">603.7</td>
<td id="S4.T3.4.4.10.5" class="ltx_td"></td>
<td id="S4.T3.4.4.10.6" class="ltx_td ltx_align_center"><span id="S4.T3.4.4.10.6.1" class="ltx_text ltx_font_bold">73.8%</span></td>
<td id="S4.T3.4.4.10.7" class="ltx_td ltx_align_center">16.2%</td>
<td id="S4.T3.4.4.10.8" class="ltx_td ltx_align_center">548.8</td>
</tr>
<tr id="S4.T3.4.4.11" class="ltx_tr">
<td id="S4.T3.4.4.11.1" class="ltx_td ltx_align_left">CoT (Llama-2-70B-chat)</td>
<td id="S4.T3.4.4.11.2" class="ltx_td ltx_align_center">45.2%</td>
<td id="S4.T3.4.4.11.3" class="ltx_td ltx_align_center">37.7%</td>
<td id="S4.T3.4.4.11.4" class="ltx_td ltx_align_center">432.6</td>
<td id="S4.T3.4.4.11.5" class="ltx_td"></td>
<td id="S4.T3.4.4.11.6" class="ltx_td ltx_align_center">58.9%</td>
<td id="S4.T3.4.4.11.7" class="ltx_td ltx_align_center">30.8%</td>
<td id="S4.T3.4.4.11.8" class="ltx_td ltx_align_center">411.8</td>
</tr>
<tr id="S4.T3.4.4.12" class="ltx_tr">
<td id="S4.T3.4.4.12.1" class="ltx_td ltx_align_left">BSM (Llama-2-70B-chat)</td>
<td id="S4.T3.4.4.12.2" class="ltx_td ltx_align_center">49.1%</td>
<td id="S4.T3.4.4.12.3" class="ltx_td ltx_align_center">30.4%</td>
<td id="S4.T3.4.4.12.4" class="ltx_td ltx_align_center">2117.8</td>
<td id="S4.T3.4.4.12.5" class="ltx_td"></td>
<td id="S4.T3.4.4.12.6" class="ltx_td ltx_align_center">64.5%</td>
<td id="S4.T3.4.4.12.7" class="ltx_td ltx_align_center">21.1%</td>
<td id="S4.T3.4.4.12.8" class="ltx_td ltx_align_center">2063.1</td>
</tr>
<tr id="S4.T3.4.4.13" class="ltx_tr">
<td id="S4.T3.4.4.13.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T3.4.4.13.1.1" class="ltx_text ltx_font_italic">Distill System 2</span></td>
<td id="S4.T3.4.4.13.2" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.13.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.13.4" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.13.5" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.13.6" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.13.7" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.4.4.13.8" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T3.4.4.14" class="ltx_tr">
<td id="S4.T3.4.4.14.1" class="ltx_td ltx_align_left ltx_border_bb">Distill BSM (Llama-2-70B-chat)</td>
<td id="S4.T3.4.4.14.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.4.4.14.2.1" class="ltx_text ltx_font_bold">58.4%</span></td>
<td id="S4.T3.4.4.14.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.4.4.14.3.1" class="ltx_text ltx_font_bold">12.2%</span></td>
<td id="S4.T3.4.4.14.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.4.4.14.4.1" class="ltx_text ltx_font_bold">4</span></td>
<td id="S4.T3.4.4.14.5" class="ltx_td ltx_border_bb"></td>
<td id="S4.T3.4.4.14.6" class="ltx_td ltx_align_center ltx_border_bb">72.4%</td>
<td id="S4.T3.4.4.14.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.4.4.14.7.1" class="ltx_text ltx_font_bold">9.1%</span></td>
<td id="S4.T3.4.4.14.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.4.4.14.8.1" class="ltx_text ltx_font_bold">4</span></td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span><span id="S4.T3.6.1" class="ltx_text ltx_font_bold">System 2 Distillation of Branch-Solve-Merge (BSM)</span>:
Open Assistant (OASST2) and MT-bench evaluation of LLM-as-a-Judge for various models. System 2 Distillation of BSM outperforms BSM itself, and even GPT4-as-a-Judge, despite using Llama-2-70B-chat. Distilled BSM has higher human agreement (Agreement), less position inconsistent predictions (% Inconsistent), and uses less output tokens (#Tokens).</figcaption>
</figure>
</section>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Branch-Solve-Merge Distillation</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p">Branch-Solve-Merge (BSM) <cite class="ltx_cite ltx_citemacro_citep">(Saha et al., <a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite>, consists of three modules: <span id="S4.SS4.p1.1.1" class="ltx_text ltx_font_italic">branch</span>, <span id="S4.SS4.p1.1.2" class="ltx_text ltx_font_italic">solve</span>, and <span id="S4.SS4.p1.1.3" class="ltx_text ltx_font_italic">merge</span>. These modules work together to break down a task into several parallel sub-tasks, each guided by specific prompts. BSM has proven effective when used in the context of an LLM acting as a judge,
see <a href="#A1.F14" title="Figure 14 ‣ S2A ‣ A.2 Experiment Details ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 14</span></a>.
The method begins by prompting the language model to list evaluation metrics (<span id="S4.SS4.p1.1.4" class="ltx_text ltx_font_italic">branch</span>) tailored to a given user query. Subsequently, the LLM is queried to evaluate a response based on each metric independently in parallel (<span id="S4.SS4.p1.1.5" class="ltx_text ltx_font_italic">solve</span>). Finally, the scores from each branch are averaged to arrive at a comprehensive evaluation decision (<span id="S4.SS4.p1.1.6" class="ltx_text ltx_font_italic">merge</span>). Notably, this method incurs an inference cost 5-6 times greater than that of a conventional (System 1) LLM evaluation approach, making it much less practical. We assess the feasibility of distilling BSM, aiming to retain its benefits while reducing computational cost.</p>
</div>
<section id="S4.SS4.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Distillation Data</h5>

<div id="S4.SS4.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px1.p1.1" class="ltx_p">Following <cite class="ltx_cite ltx_citemacro_citet">Yuan et al. (<a href="#bib.bib43" title="" class="ltx_ref">2024</a>); Li et al. (<a href="#bib.bib23" title="" class="ltx_ref">2023b</a>)</cite>, we used the Open Assistant Dataset v2 (OASST2) <cite class="ltx_cite ltx_citemacro_citep">(Köpf et al., <a href="#bib.bib20" title="" class="ltx_ref">2024</a>)</cite> with turn 1 and English only data.
We use queries along with two candidate responses from the OASST2 training set as inputs (19,672 examples in total).
We use <span id="S4.SS4.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_italic">self-consistency under input perturbations</span> to ensure the quality of our distillation data. Specifically, as two responses are being judged, we evaluate each sample twice with BSM - once in the original order and once in the swapped order. The winning response should remain consistent regardless of the order. We filter out samples that do not yield a consistent winner when the response order is swapped.</p>
</div>
</section>
<section id="S4.SS4.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Evaluation</h5>

<div id="S4.SS4.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px2.p1.1" class="ltx_p">We evaluate our models on two popular benchmarks, the OASST2 valid set and MT-bench <cite class="ltx_cite ltx_citemacro_citep">(Zheng et al., <a href="#bib.bib46" title="" class="ltx_ref">2024</a>)</cite>. The OASST2 validation set comprises 273 samples, restricted to turn 1 and English language only. Evaluations of response pairs are performed in both original and swapped orders.
As we trained our distilled model on the OASST2 training set, the OASST2 validation set functions as an in-distribution evaluation set, while MT-bench is more out-of-distribution.
MT-bench is a popular benchmark that evaluates LLM-as-judges of other LLM’s responses when acting as helpful AI assistants conversations. It consists of instructions from 8 diverse domains e.g., writing, reasoning, math, coding, etc.</p>
</div>
<div id="S4.SS4.SSS0.Px2.p2" class="ltx_para">
<p id="S4.SS4.SSS0.Px2.p2.1" class="ltx_p">Following <cite class="ltx_cite ltx_citemacro_citet">Zheng et al. (<a href="#bib.bib46" title="" class="ltx_ref">2024</a>)</cite>, we assessed the <span id="S4.SS4.SSS0.Px2.p2.1.1" class="ltx_text ltx_font_typewriter">Agreement</span> between model votes and human expert votes. A well-documented limitation of LLM-as-a-judge is position bias, where a Language Model (LLM) tends to favor certain positions over others. This bias is evident as altering the position of responses in the evaluation prompt often leads to different decisions by the model. To quantify this, we not only measure agreement but also calculate the <span id="S4.SS4.SSS0.Px2.p2.1.2" class="ltx_text ltx_font_typewriter">Percentage of Inconsistent</span> examples to assess position bias.</p>
</div>
<figure id="S4.F2" class="ltx_figure"><img src="/html/2407.06023/assets/figures/MT_bench_wo_tie.png" id="S4.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="538" height="192" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>The agreement between LLM judges and human preferences per evaluated category on MT-bench.</figcaption>
</figure>
<figure id="S4.T4" class="ltx_table">
<div id="S4.T4.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:317.0pt;height:129.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-39.6pt,16.2pt) scale(0.8,0.8) ;">
<table id="S4.T4.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T4.3.1.1" class="ltx_tr">
<td id="S4.T4.3.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Model</td>
<td id="S4.T4.3.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">k=1</td>
<td id="S4.T4.3.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">k=5</td>
<td id="S4.T4.3.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">k=10</td>
</tr>
<tr id="S4.T4.3.1.2" class="ltx_tr">
<td id="S4.T4.3.1.2.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.2.2" class="ltx_td ltx_align_right ltx_border_t">Acc %</td>
<td id="S4.T4.3.1.2.3" class="ltx_td ltx_align_center ltx_border_t">#Tokens</td>
<td id="S4.T4.3.1.2.4" class="ltx_td ltx_align_right ltx_border_t">Acc %</td>
<td id="S4.T4.3.1.2.5" class="ltx_td ltx_align_center ltx_border_t">#Tokens</td>
<td id="S4.T4.3.1.2.6" class="ltx_td ltx_align_right ltx_border_t">Acc %</td>
<td id="S4.T4.3.1.2.7" class="ltx_td ltx_align_center ltx_border_t">#Tokens</td>
</tr>
<tr id="S4.T4.3.1.3" class="ltx_tr">
<td id="S4.T4.3.1.3.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T4.3.1.3.1.1" class="ltx_text ltx_font_italic">System 1</span></td>
<td id="S4.T4.3.1.3.2" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.3.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.3.4" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.3.5" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.3.6" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.3.7" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T4.3.1.4" class="ltx_tr">
<td id="S4.T4.3.1.4.1" class="ltx_td ltx_align_left">Few (8)-shot (no CoT)</td>
<td id="S4.T4.3.1.4.2" class="ltx_td ltx_align_right">7.58%</td>
<td id="S4.T4.3.1.4.3" class="ltx_td ltx_align_center">57</td>
<td id="S4.T4.3.1.4.4" class="ltx_td ltx_align_right">9.40%</td>
<td id="S4.T4.3.1.4.5" class="ltx_td ltx_align_center">295</td>
<td id="S4.T4.3.1.4.6" class="ltx_td ltx_align_right">10.31%</td>
<td id="S4.T4.3.1.4.7" class="ltx_td ltx_align_center">620</td>
</tr>
<tr id="S4.T4.3.1.5" class="ltx_tr">
<td id="S4.T4.3.1.5.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T4.3.1.5.1.1" class="ltx_text ltx_font_italic">System 2</span></td>
<td id="S4.T4.3.1.5.2" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.5.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.5.4" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.5.5" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.5.6" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.5.7" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T4.3.1.6" class="ltx_tr">
<td id="S4.T4.3.1.6.1" class="ltx_td ltx_align_left">CoT zero-shot</td>
<td id="S4.T4.3.1.6.2" class="ltx_td ltx_align_right">52.77%</td>
<td id="S4.T4.3.1.6.3" class="ltx_td ltx_align_center">270</td>
<td id="S4.T4.3.1.6.4" class="ltx_td ltx_align_right">57.54%</td>
<td id="S4.T4.3.1.6.5" class="ltx_td ltx_align_center">1385</td>
<td id="S4.T4.3.1.6.6" class="ltx_td ltx_align_right">59.44%</td>
<td id="S4.T4.3.1.6.7" class="ltx_td ltx_align_center">2760</td>
</tr>
<tr id="S4.T4.3.1.7" class="ltx_tr">
<td id="S4.T4.3.1.7.1" class="ltx_td ltx_align_left">CoT few (8)-shot</td>
<td id="S4.T4.3.1.7.2" class="ltx_td ltx_align_right">36.39%</td>
<td id="S4.T4.3.1.7.3" class="ltx_td ltx_align_center">297</td>
<td id="S4.T4.3.1.7.4" class="ltx_td ltx_align_right">54.97%</td>
<td id="S4.T4.3.1.7.5" class="ltx_td ltx_align_center">1560</td>
<td id="S4.T4.3.1.7.6" class="ltx_td ltx_align_right">63.84%</td>
<td id="S4.T4.3.1.7.7" class="ltx_td ltx_align_center">3120</td>
</tr>
<tr id="S4.T4.3.1.8" class="ltx_tr">
<td id="S4.T4.3.1.8.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T4.3.1.8.1.1" class="ltx_text ltx_font_italic">Distill System 2</span></td>
<td id="S4.T4.3.1.8.2" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.8.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.8.4" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.8.5" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.8.6" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.3.1.8.7" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T4.3.1.9" class="ltx_tr">
<td id="S4.T4.3.1.9.1" class="ltx_td ltx_align_left ltx_border_bb">Distill CoT zero-shot</td>
<td id="S4.T4.3.1.9.2" class="ltx_td ltx_align_right ltx_border_bb">7.13%</td>
<td id="S4.T4.3.1.9.3" class="ltx_td ltx_align_center ltx_border_bb">18</td>
<td id="S4.T4.3.1.9.4" class="ltx_td ltx_align_right ltx_border_bb">7.13%</td>
<td id="S4.T4.3.1.9.5" class="ltx_td ltx_align_center ltx_border_bb">90</td>
<td id="S4.T4.3.1.9.6" class="ltx_td ltx_align_right ltx_border_bb">7.35%</td>
<td id="S4.T4.3.1.9.7" class="ltx_td ltx_align_center ltx_border_bb">180</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>GSM8k test set accuracy. Number of votes <math id="S4.T4.2.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.T4.2.m1.1b"><mi id="S4.T4.2.m1.1.1" xref="S4.T4.2.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.T4.2.m1.1c"><ci id="S4.T4.2.m1.1.1.cmml" xref="S4.T4.2.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T4.2.m1.1d">k</annotation></semantics></math> in majority voting represents how many candidates were sampled to collect votes towards predicted answers. In this case System 2 Distillation of CoT does not work well.</figcaption>
</figure>
</section>
<section id="S4.SS4.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">OASST2 Evaluation Results</h5>

<div id="S4.SS4.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px3.p1.1" class="ltx_p"><a href="#S4.T3" title="Table 3 ‣ Results ‣ 4.3 System 2 Attention Distillation ‣ 4 Experiments ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 3</span></a> provides
results on the OASST2 dataset. Compared to baseline (System 1) LLMs, the Chain-of-Thought (CoT) method improves performance by improving agreement and reducing inconsistency rates (see prompts in Appendix). While BSM outperforms CoT, this comes at the cost of increased inference time (#Tokens). Remarkably, our distilled System 2 BSM model requires the generation of only four tokens and still outperforms both CoT and BSM. Furthermore, our distilled model based on Llama-2-70B-chat outperforms GPT-4-0125-preview, achieving
higher human agreement and greater consistency.</p>
</div>
</section>
<section id="S4.SS4.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">MT-Bench Evaluation Results</h5>

<div id="S4.SS4.SSS0.Px4.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px4.p1.1" class="ltx_p"><a href="#S4.T3" title="Table 3 ‣ Results ‣ 4.3 System 2 Attention Distillation ‣ 4 Experiments ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 3</span></a> also provides results on MT-bench, which serves as an out-of-distribution test. The results mirror those from the OASST2 evaluation. Both Chain-of-Thought (CoT) and BSM improve model performance but at the expense of significantly increased inference costs. Our distilled BSM model not only achieves higher human agreement and lower inconsistency rates but also requires less computational resources.
Although our model slightly underperforms in agreement compared to the state-of-the-art GPT-4-0125-preview model, it was trained solely on unlabeled data from OASST2 based on Llama-2-70B-chat. Despite this, it is more consistent and inference is cheap in terms of output tokens.</p>
</div>
</section>
<section id="S4.SS4.SSS0.Px5" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Per Category Analysis</h5>

<div id="S4.SS4.SSS0.Px5.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px5.p1.1" class="ltx_p">Here, we further analyze the MT-Bench results in terms of <span id="S4.SS4.SSS0.Px5.p1.1.1" class="ltx_text ltx_font_typewriter">Agreement</span> by category. Figure <a href="#S4.F2" title="Figure 2 ‣ Evaluation ‣ 4.4 Branch-Solve-Merge Distillation ‣ 4 Experiments ‣ Distilling System 2 into System 1" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> shows the per category agreement. We observe that CoT improved agreement compared to the base model (Llama-2-70B-Chat) on all categories. BSM is better than CoT and our distilled BSM is even better than BSM.
Although Distilled BSM achieves superior performance compared to the baselines across all categories, it still lags behind GPT-4-0125-preview in reasoning, coding, and extraction. However, it surpasses GPT-4-0125-preview in writing, math, and STEM.</p>
</div>
</section>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span>Chain-of-Thought Distillation</h3>

<div id="S4.SS5.p1" class="ltx_para">
<p id="S4.SS5.p1.1" class="ltx_p">Chain-of-Thought (CoT) <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite> has been shown to be an effective method to improve LLM’s reasoning abilities, such as for solving graduate school math problems. The LLM generates intermediate tokens that are steps (<span id="S4.SS5.p1.1.1" class="ltx_text ltx_font_italic">chain</span>) of reasoning (<span id="S4.SS5.p1.1.2" class="ltx_text ltx_font_italic">thoughts</span>) before it produces the final answer.
We consider two variants of the approach: (i) few-shot CoT, whereby multiple [question, CoT, answer] examples from the training set are provided as part of the context followed by the question; and (ii) zero-shot, whereby an explicit instruction to think “step by step” is added to the prompt in addition to the question, see Appendix <a href="#A1.F10" title="Figure 10 ‣ A.1 Prompts ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 10</span></a>.</p>
</div>
<section id="S4.SS5.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Distillation data</h5>

<div id="S4.SS5.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS5.SSS0.Px1.p1.3" class="ltx_p">We use CoT to produce answers for questions from the training split of GSM8k <cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite> (which we consider unlabeled),
using majority voting with <math id="S4.SS5.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="K=10" display="inline"><semantics id="S4.SS5.SSS0.Px1.p1.1.m1.1a"><mrow id="S4.SS5.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS5.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="S4.SS5.SSS0.Px1.p1.1.m1.1.1.2" xref="S4.SS5.SSS0.Px1.p1.1.m1.1.1.2.cmml">K</mi><mo id="S4.SS5.SSS0.Px1.p1.1.m1.1.1.1" xref="S4.SS5.SSS0.Px1.p1.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS5.SSS0.Px1.p1.1.m1.1.1.3" xref="S4.SS5.SSS0.Px1.p1.1.m1.1.1.3.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.SSS0.Px1.p1.1.m1.1b"><apply id="S4.SS5.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS5.SSS0.Px1.p1.1.m1.1.1"><eq id="S4.SS5.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S4.SS5.SSS0.Px1.p1.1.m1.1.1.1"></eq><ci id="S4.SS5.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S4.SS5.SSS0.Px1.p1.1.m1.1.1.2">𝐾</ci><cn type="integer" id="S4.SS5.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S4.SS5.SSS0.Px1.p1.1.m1.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.SSS0.Px1.p1.1.m1.1c">K=10</annotation></semantics></math>. The resulting distillation training set consists of <math id="S4.SS5.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="7461" display="inline"><semantics id="S4.SS5.SSS0.Px1.p1.2.m2.1a"><mn id="S4.SS5.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS5.SSS0.Px1.p1.2.m2.1.1.cmml">7461</mn><annotation-xml encoding="MathML-Content" id="S4.SS5.SSS0.Px1.p1.2.m2.1b"><cn type="integer" id="S4.SS5.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS5.SSS0.Px1.p1.2.m2.1.1">7461</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.SSS0.Px1.p1.2.m2.1c">7461</annotation></semantics></math> [question, answer] pairs i.e., without any intermediate reasoning steps. The accuracy of the self-supervised targets, computed for analysis purposes, is <math id="S4.SS5.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="56.81\%" display="inline"><semantics id="S4.SS5.SSS0.Px1.p1.3.m3.1a"><mrow id="S4.SS5.SSS0.Px1.p1.3.m3.1.1" xref="S4.SS5.SSS0.Px1.p1.3.m3.1.1.cmml"><mn id="S4.SS5.SSS0.Px1.p1.3.m3.1.1.2" xref="S4.SS5.SSS0.Px1.p1.3.m3.1.1.2.cmml">56.81</mn><mo id="S4.SS5.SSS0.Px1.p1.3.m3.1.1.1" xref="S4.SS5.SSS0.Px1.p1.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.SSS0.Px1.p1.3.m3.1b"><apply id="S4.SS5.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S4.SS5.SSS0.Px1.p1.3.m3.1.1"><csymbol cd="latexml" id="S4.SS5.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S4.SS5.SSS0.Px1.p1.3.m3.1.1.1">percent</csymbol><cn type="float" id="S4.SS5.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S4.SS5.SSS0.Px1.p1.3.m3.1.1.2">56.81</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.SSS0.Px1.p1.3.m3.1c">56.81\%</annotation></semantics></math>.</p>
</div>
</section>
<section id="S4.SS5.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Evaluation</h5>

<div id="S4.SS5.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS5.SSS0.Px2.p1.2" class="ltx_p">We report evaluation accuracy computed over the GSM8k test set with majority voting with different values of <math id="S4.SS5.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS5.SSS0.Px2.p1.1.m1.1a"><mi id="S4.SS5.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS5.SSS0.Px2.p1.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS5.SSS0.Px2.p1.1.m1.1b"><ci id="S4.SS5.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS5.SSS0.Px2.p1.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.SSS0.Px2.p1.1.m1.1c">K</annotation></semantics></math>.
Similarly to our previous experiments, we report the average number of predicted tokens for each method. Note that we compute this average over all generated tokens when we run majority voting to see how the increase in <math id="S4.SS5.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS5.SSS0.Px2.p1.2.m2.1a"><mi id="S4.SS5.SSS0.Px2.p1.2.m2.1.1" xref="S4.SS5.SSS0.Px2.p1.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS5.SSS0.Px2.p1.2.m2.1b"><ci id="S4.SS5.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S4.SS5.SSS0.Px2.p1.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.SSS0.Px2.p1.2.m2.1c">K</annotation></semantics></math> affects the inference cost.
We consider several baselines: System 1 and System 2 (CoT) methods evaluated with zero-shot or 8-shot input contexts. Note that System 2 with 8-shot means that CoTs are provided in the few-shot inputs, while System 1 means that the few shot examples contain questions and answers, but no CoTs.</p>
</div>
</section>
<section id="S4.SS5.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Results</h5>

<div id="S4.SS5.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS5.SSS0.Px3.p1.1" class="ltx_p">Evaluation results are presented in <a href="#S4.T4" title="Table 4 ‣ Evaluation ‣ 4.4 Branch-Solve-Merge Distillation ‣ 4 Experiments ‣ Distilling System 2 into System 1" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 4</span></a>. First, improvements are coming from using the CoT method as expected: it helps when being presented as part of the few-shot context or as part of the instruction in the prompt template. These improvements come with an increase in inference cost: sequences predicted with CoT methods are substantially longer compared to the System 1 method.
Second, our System 2 distillation method yields poor performance across various decoding hyper-parameters. The GSM8k task (math problems) requires a very different kind of reasoning compared to other tasks we considered in this work. This highlights the non-trivial aspect of System 2 distillation: the proposed distillation algorithm works in many cases but not always. This leaves room for future research to elucidate in exactly which circumstances to apply distillation, and when not to, in a similar manner perhaps to the approach in humans.</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">Recent work has shown that complex reasoning procedures using LLMs in the inner loop, called System 2 approaches, can improve performance. In this work we have shown that in many cases it is possible to distill this System 2 reasoning into the outputs of the LLM <span id="S5.p1.1.1" class="ltx_text ltx_font_italic">without intermediate generations</span> while maintaining, or sometimes even improving, performance.
While not all methods can be distilled easily using our method, with Chain-of-Thought for complex reasoning being a challenging counterexample,
this is possible for diverse approaches. Our method works for System 2 Attention for dealing with bias and irrelevant context, Rephrase and Respond for clarifying task instructions, and Branch-Solve-Merge for improved LLM-as-a-Judge evaluation.
Pragmatically, distilling these approaches makes them more likely to be used by LLM practitioners, and they are more efficient at inference time. Looking forward, systems that can distill useful tasks in this way free up more time to spend on reasoning about the tasks that they cannot yet do well, just as humans do. Hence, we expect exploring this approach in a continuous training loop will be a fruitful research direction.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Limitations</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">In this paper, we explored three System 2 methods—RaR, S2A, and BSM—which have been successfully distilled, yielding enhanced results compared to the original System 1 performance while incurring lower inference costs than System 2. However, the effectiveness of these methods can vary depending on the specific task or the dataset used for model training. For instance, we observed that the CoT method could not be effectively distilled back to System 1 using our method.
We note that recent methods have tried alternative ways to distill
CoT <cite class="ltx_cite ltx_citemacro_cite">Deng et al. (<a href="#bib.bib14" title="" class="ltx_ref">2023b</a>, <a href="#bib.bib13" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.1" class="ltx_p">Moreover, due to the self-supervised nature of these methods, model performance relies on the specific filters applied. In our study, we depended on a consistency criterion that includes <span id="S6.p2.1.1" class="ltx_text ltx_font_italic">self-consistency of outputs</span> and <span id="S6.p2.1.2" class="ltx_text ltx_font_italic">self-consistency under input perturbation</span>. Although there are multiple alternative strategies to enhance data quality in self-supervised learning, these were not explored in our research.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Adriana et al. (2015)</span>
<span class="ltx_bibblock">
Romero Adriana, Ballas Nicolas, K Samira Ebrahimi, Chassang Antoine, Gatta Carlo, and Bengio Yoshua. 2015.

</span>
<span class="ltx_bibblock">Fitnets: Hints for thin deep nets.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">Proc. ICLR</em>, 2(3):1.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ba and Caruana (2014)</span>
<span class="ltx_bibblock">
Jimmy Ba and Rich Caruana. 2014.

</span>
<span class="ltx_bibblock">Do deep nets really need to be deep?

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 27.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bengio (2017)</span>
<span class="ltx_bibblock">
Yoshua Bengio. 2017.

</span>
<span class="ltx_bibblock">The consciousness prior.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1709.08568</em>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Besta et al. (2024)</span>
<span class="ltx_bibblock">
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024.

</span>
<span class="ltx_bibblock">Graph of thoughts: Solving elaborate problems with large language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI Conference on Artificial Intelligence</em>, volume 38, pages 17682–17690.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Buciluǎ et al. (2006)</span>
<span class="ltx_bibblock">
Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006.

</span>
<span class="ltx_bibblock">Model compression.

</span>
<span class="ltx_bibblock">In <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining</em>, pages 535–541.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Charlton and Starkey (2013)</span>
<span class="ltx_bibblock">
Samuel G Charlton and Nicola J Starkey. 2013.

</span>
<span class="ltx_bibblock">Driving on familiar roads: Automaticity and inattention blindness.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Transportation research part F: traffic psychology and behaviour</em>, 19:121–133.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2024)</span>
<span class="ltx_bibblock">
Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, and Ke Ding. 2024.

</span>
<span class="ltx_bibblock">Learning to maximize mutual information for chain-of-thought distillation.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.03348</em>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2023)</span>
<span class="ltx_bibblock">
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2311.17311" title="" class="ltx_ref ltx_href">Universal self-consistency for large language model generation</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2311.17311.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. (2021)</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2110.14168" title="" class="ltx_ref ltx_href">Training verifiers to solve math word problems</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2110.14168.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cohen and Squire (1980)</span>
<span class="ltx_bibblock">
Neal J Cohen and Larry R Squire. 1980.

</span>
<span class="ltx_bibblock">Preserved learning and retention of pattern-analyzing skill in amnesia: Dissociation of knowing how and knowing that.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Science</em>, 210(4466):207–210.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Czarnecki et al. (2017)</span>
<span class="ltx_bibblock">
Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan Pascanu. 2017.

</span>
<span class="ltx_bibblock">Sobolev training for neural networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 30.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Deng et al. (2023a)</span>
<span class="ltx_bibblock">
Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. 2023a.

</span>
<span class="ltx_bibblock">Rephrase and respond: Let large language models ask better questions for themselves.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.04205</em>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Deng et al. (2024)</span>
<span class="ltx_bibblock">
Yuntian Deng, Yejin Choi, and Stuart Shieber. 2024.

</span>
<span class="ltx_bibblock">From explicit cot to implicit cot: Learning to internalize cot step by step.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2405.14838</em>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Deng et al. (2023b)</span>
<span class="ltx_bibblock">
Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. 2023b.

</span>
<span class="ltx_bibblock">Implicit chain of thought reasoning via knowledge distillation.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.01460</em>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">DePhillips et al. (1960)</span>
<span class="ltx_bibblock">
Frank A DePhillips, William M Berliner, and James J Cribben. 1960.

</span>
<span class="ltx_bibblock">Management of training programs. homewood, illinois: Richard d. irwin.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dhuliawala et al. (2023)</span>
<span class="ltx_bibblock">
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023.

</span>
<span class="ltx_bibblock">Chain-of-verification reduces hallucination in large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.11495</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton et al. (2015)</span>
<span class="ltx_bibblock">
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.

</span>
<span class="ltx_bibblock">Distilling the knowledge in a neural network.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1503.02531</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kahneman (2011)</span>
<span class="ltx_bibblock">
Daniel Kahneman. 2011.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Thinking, fast and slow</em>.

</span>
<span class="ltx_bibblock">macmillan.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kojima et al. (2022)</span>
<span class="ltx_bibblock">
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.

</span>
<span class="ltx_bibblock">Large language models are zero-shot reasoners.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 35:22199–22213.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Köpf et al. (2024)</span>
<span class="ltx_bibblock">
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. 2024.

</span>
<span class="ltx_bibblock">Openassistant conversations-democratizing large language model alignment.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">LeCun (2022)</span>
<span class="ltx_bibblock">
Yann LeCun. 2022.

</span>
<span class="ltx_bibblock">A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Open Review</em>, 62(1).

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023a)</span>
<span class="ltx_bibblock">
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023a.

</span>
<span class="ltx_bibblock">Symbolic chain-of-thought distillation: Small models can also" think" step-by-step.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.14050</em>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023b)</span>
<span class="ltx_bibblock">
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. 2023b.

</span>
<span class="ltx_bibblock">Self-alignment with instruction backtranslation.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.06259</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2024)</span>
<span class="ltx_bibblock">
Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. 2024.

</span>
<span class="ltx_bibblock">Chain of thought empowers transformers to solve inherently serial problems.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.12875</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Madaan et al. (2024)</span>
<span class="ltx_bibblock">
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024.

</span>
<span class="ltx_bibblock">Self-refine: Iterative refinement with self-feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nye et al. (2021)</span>
<span class="ltx_bibblock">
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021.

</span>
<span class="ltx_bibblock">Show your work: Scratchpads for intermediate computation with language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2112.00114</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Perez et al. (2020)</span>
<span class="ltx_bibblock">
Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020.

</span>
<span class="ltx_bibblock">Unsupervised question decomposition for question answering.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2002.09758</em>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Press et al. (2022)</span>
<span class="ltx_bibblock">
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022.

</span>
<span class="ltx_bibblock">Measuring and narrowing the compositionality gap in language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.03350</em>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2019)</span>
<span class="ltx_bibblock">
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">OpenAI blog</em>, 1(8):9.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Saha et al. (2023)</span>
<span class="ltx_bibblock">
Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2023.

</span>
<span class="ltx_bibblock">Branch-solve-merge improves large language model evaluation and generation.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.15123</em>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schlag et al. (2023)</span>
<span class="ltx_bibblock">
Imanol Schlag, Sainbayar Sukhbaatar, Asli Celikyilmaz, Wen-tau Yih, Jason Weston, Jürgen Schmidhuber, and Xian Li. 2023.

</span>
<span class="ltx_bibblock">Large language model programs.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.05364</em>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sharma et al. (2023)</span>
<span class="ltx_bibblock">
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2310.13548" title="" class="ltx_ref ltx_href">Towards understanding sycophancy in language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2310.13548.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sloman (1996)</span>
<span class="ltx_bibblock">
Steven A. Sloman. 1996.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:13454019" title="" class="ltx_ref ltx_href">The empirical case for two systems of reasoning.</a>

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Psychological Bulletin</em>, 119:3–22.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.09288</em>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. (2017)</span>
<span class="ltx_bibblock">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 30.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023)</span>
<span class="ltx_bibblock">
Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023.

</span>
<span class="ltx_bibblock">Scott: Self-consistent chain-of-thought distillation.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.01879</em>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022)</span>
<span class="ltx_bibblock">
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022.

</span>
<span class="ltx_bibblock">Self-consistency improves chain of thought reasoning in language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2203.11171</em>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022)</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.

</span>
<span class="ltx_bibblock">Chain-of-thought prompting elicits reasoning in large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 35:24824–24837.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Weng et al. (2022)</span>
<span class="ltx_bibblock">
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2022.

</span>
<span class="ltx_bibblock">Large language models are better reasoners with self-verification.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.09561</em>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Weston and Sukhbaatar (2023)</span>
<span class="ltx_bibblock">
Jason Weston and Sainbayar Sukhbaatar. 2023.

</span>
<span class="ltx_bibblock">System 2 attention (is something you might need too).

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.11829</em>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2024)</span>
<span class="ltx_bibblock">
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024.

</span>
<span class="ltx_bibblock">Tree of thoughts: Deliberate problem solving with large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. (2023)</span>
<span class="ltx_bibblock">
Dongran Yu, Bo Yang, Dayou Liu, Hui Wang, and Shirui Pan. 2023.

</span>
<span class="ltx_bibblock">A survey on neural-symbolic learning systems.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Neural Networks</em>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yuan et al. (2024)</span>
<span class="ltx_bibblock">
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024.

</span>
<span class="ltx_bibblock">Self-rewarding language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.10020</em>.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2024)</span>
<span class="ltx_bibblock">
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. 2024.

</span>
<span class="ltx_bibblock">Chain of preference optimization: Improving chain-of-thought reasoning in llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2406.09136</em>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhao et al. (2024)</span>
<span class="ltx_bibblock">
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024.

</span>
<span class="ltx_bibblock">Wildchat: 1m chatgpt interaction logs in the wild.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2405.01470</em>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. (2024)</span>
<span class="ltx_bibblock">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024.

</span>
<span class="ltx_bibblock">Judging llm-as-a-judge with mt-bench and chatbot arena.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Appendix</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Prompts</h3>

<figure id="A1.F3" class="ltx_figure"><svg id="A1.F3.pic1" class="ltx_picture ltx_centering" height="73.07" overflow="visible" version="1.1" width="600"><g transform="translate(0,73.07) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 55.35 C 0 65.14 7.93 73.07 17.72 73.07 L 582.28 73.07 C 592.07 73.07 600 65.14 600 55.35 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 55.35 C 1.97 64.05 9.02 71.1 17.72 71.1 L 582.28 71.1 C 590.98 71.1 598.03 64.05 598.03 55.35 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="45.51" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F3.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F3.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F3.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">{question}
<br class="ltx_break"></span></span>
<span id="A1.F3.pic1.1.1.1.1.1.2" class="ltx_p">Reword and elaborate on the inquiry, then provide an answer.</span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span><span id="A1.F3.2.1" class="ltx_text ltx_font_bold">1-step RaR prompt.</span> The 1-step RaR process involves the model rephrasing the question and subsequently providing an answer, all in a single step.</figcaption>
</figure>
<figure id="A1.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F4.pic1" class="ltx_picture ltx_centering ltx_figure_panel" height="106.28" overflow="visible" version="1.1" width="600"><g transform="translate(0,106.28) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 88.56 C 0 98.34 7.93 106.28 17.72 106.28 L 582.28 106.28 C 592.07 106.28 600 98.34 600 88.56 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 88.56 C 1.97 97.26 9.02 104.31 17.72 104.31 L 582.28 104.31 C 590.98 104.31 598.03 97.26 598.03 88.56 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="78.72" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F4.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F4.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F4.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">{question}
<br class="ltx_break"></span></span>
<span id="A1.F4.pic1.1.1.1.1.1.2" class="ltx_p">Based on the details given in the initial inquiry, could you kindly rephrase the question and separate these 2 words in the revised question? Please ensure these 2 words remain unchanged from the original question.</span>
</span></foreignObject></g></g></svg></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F4.pic2" class="ltx_picture ltx_centering ltx_figure_panel" height="41.4" overflow="visible" version="1.1" width="600"><g transform="translate(0,41.4) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 23.68 C 0 33.46 7.93 41.4 17.72 41.4 L 582.28 41.4 C 592.07 41.4 600 33.46 600 23.68 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 23.68 C 1.97 32.38 9.02 39.43 17.72 39.43 L 582.28 39.43 C 590.98 39.43 598.03 32.38 598.03 23.68 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="13.84" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F4.pic2.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F4.pic2.1.1.1.1.1.1" class="ltx_p"><span id="A1.F4.pic2.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">{rephrased question}</span></span>
</span></foreignObject></g></g></svg></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span><span id="A1.F4.2.1" class="ltx_text ltx_font_bold">2-step RaR prompt for last letter concatenation task, step 1 (top), step 2 (down)</span> The 1-step RaR process involves the model rephrasing the question and subsequently providing an answer, all in a single step.</figcaption>
</figure>
<figure id="A1.F5" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F5.pic1" class="ltx_picture ltx_centering ltx_figure_panel" height="122.88" overflow="visible" version="1.1" width="600"><g transform="translate(0,122.88) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 105.16 C 0 114.95 7.93 122.88 17.72 122.88 L 582.28 122.88 C 592.07 122.88 600 114.95 600 105.16 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 105.16 C 1.97 113.86 9.02 120.91 17.72 120.91 L 582.28 120.91 C 590.98 120.91 598.03 113.86 598.03 105.16 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="95.32" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F5.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F5.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F5.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">{question}
<br class="ltx_break"></span></span>
<span id="A1.F5.pic1.1.1.1.1.1.2" class="ltx_p">Based on the information provided in the original query, could you please rephrase it and expand it to help you do better answering. Please ensure that your response solely includes the reformulated question, excluding any introductory phrases or explanatory remarks, while preserving all the details from the original query.</span>
</span></foreignObject></g></g></svg></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F5.pic2" class="ltx_picture ltx_centering ltx_figure_panel" height="41.4" overflow="visible" version="1.1" width="600"><g transform="translate(0,41.4) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 23.68 C 0 33.46 7.93 41.4 17.72 41.4 L 582.28 41.4 C 592.07 41.4 600 33.46 600 23.68 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 23.68 C 1.97 32.38 9.02 39.43 17.72 39.43 L 582.28 39.43 C 590.98 39.43 598.03 32.38 598.03 23.68 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="13.84" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F5.pic2.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F5.pic2.1.1.1.1.1.1" class="ltx_p"><span id="A1.F5.pic2.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">{rephrased question}</span> Answer the Yes or No question.</span>
</span></foreignObject></g></g></svg></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span><span id="A1.F5.2.1" class="ltx_text ltx_font_bold">2-step RaR prompt for coin flip task, step 1 (top), step 2 (down)</span> The 1-step RaR process involves the model rephrasing the question and subsequently providing an answer, all in a single step.</figcaption>
</figure>
<figure id="A1.F6" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F6.pic1" class="ltx_picture ltx_centering ltx_figure_panel" height="124.42" overflow="visible" version="1.1" width="600"><g transform="translate(0,124.42) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 106.7 C 0 116.49 7.93 124.42 17.72 124.42 L 582.28 124.42 C 592.07 124.42 600 116.49 600 106.7 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 106.7 C 1.97 115.4 9.02 122.45 17.72 122.45 L 582.28 122.45 C 590.98 122.45 598.03 115.4 598.03 106.7 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="96.86" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F6.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F6.pic1.1.1.1.1.1.1" class="ltx_p">Given the following text by a user, extract the part that is unbiased and not their opinion, so that using that text alone would be good context for providing an unbiased answer to the question portion of the text. Please include the actual question or query that the user is asking. Separate this into two categories labeled with “Unbiased text context (includes all content except user’s bias):” and “Question/Query (does not include user bias/preference):”.</span>
<span id="A1.F6.pic1.1.1.1.1.1.2" class="ltx_p">Text by User: <span id="A1.F6.pic1.1.1.1.1.1.2.1" class="ltx_text ltx_font_bold">{input}</span></span>
</span></foreignObject></g></g></svg></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F6.pic2" class="ltx_picture ltx_centering ltx_figure_panel" height="41.4" overflow="visible" version="1.1" width="600"><g transform="translate(0,41.4) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 23.68 C 0 33.46 7.93 41.4 17.72 41.4 L 582.28 41.4 C 592.07 41.4 600 33.46 600 23.68 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 23.68 C 1.97 32.38 9.02 39.43 17.72 39.43 L 582.28 39.43 C 590.98 39.43 598.03 32.38 598.03 23.68 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="13.84" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F6.pic2.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F6.pic2.1.1.1.1.1.1" class="ltx_p"><span id="A1.F6.pic2.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">{input}</span></span>
<span id="A1.F6.pic2.1.1.1.1.1.2" class="ltx_p">Answer in an unbiased way.</span>
</span></foreignObject></g></g></svg></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span><span id="A1.F6.2.1" class="ltx_text ltx_font_bold">System 2 Attention prompts.</span> We use the prompts from <cite class="ltx_cite ltx_citemacro_citet">Weston and Sukhbaatar (<a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite> to extract the training signal for distillation. The output after the second stage is used as the distillation target. </figcaption>
</figure>
<figure id="A1.F7" class="ltx_figure"><svg id="A1.F7.pic1" class="ltx_picture ltx_centering" height="136.79" overflow="visible" version="1.1" width="600"><g transform="translate(0,136.79) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 119.08 C 0 128.86 7.93 136.79 17.72 136.79 L 582.28 136.79 C 592.07 136.79 600 128.86 600 119.08 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 119.08 C 1.97 127.78 9.02 134.83 17.72 134.83 L 582.28 134.83 C 590.98 134.83 598.03 127.78 598.03 119.08 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="109.24" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F7.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F7.pic1.1.1.1.1.1.1" class="ltx_p">We want to evaluate the quality of the responses provided by two AI assistants to the user question displayed
below. Your task is to propose an evaluation plan that can be executed to compare the two responses. The
evaluation plan should consist of a list of up to five factors that one should consider such as helpfulness,
relevance, accuracy, etc. In each line, write an evaluation criterion along with a short descrition of how we
should evaluate that criterion.</span>
<span id="A1.F7.pic1.1.1.1.1.1.2" class="ltx_p">User Question: <span id="A1.F7.pic1.1.1.1.1.1.2.1" class="ltx_text ltx_font_bold">{user_query}
<br class="ltx_break"></span>Evaluation Plan:</span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span><span id="A1.F7.2.1" class="ltx_text ltx_font_bold">BSM: Branch prompt.</span></figcaption>
</figure>
<figure id="A1.F8" class="ltx_figure"><svg id="A1.F8.pic1" class="ltx_picture ltx_centering" height="307.07" overflow="visible" version="1.1" width="600"><g transform="translate(0,307.07) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 289.35 C 0 299.13 7.93 307.07 17.72 307.07 L 582.28 307.07 C 592.07 307.07 600 299.13 600 289.35 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 289.35 C 1.97 298.05 9.02 305.1 17.72 305.1 L 582.28 305.1 C 590.98 305.1 598.03 298.05 598.03 289.35 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="279.51" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F8.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F8.pic1.1.1.1.1.1.1" class="ltx_p">You are given a user question and responses provided by two AI assistants. Your task is to evaluate and score the
quality of the responses based on a single evaluation criterion displayed below. Make sure to evaluate only based
on the criterion specified and none other. In the first line, provide a score between 1 to 5 for Assistant A’s
response. In the second line, provide a score between 1 to 5 for Assistant B’s response.</span>
<span id="A1.F8.pic1.1.1.1.1.1.2" class="ltx_p">[User Question]</span>
<span id="A1.F8.pic1.1.1.1.1.1.3" class="ltx_p"><span id="A1.F8.pic1.1.1.1.1.1.3.1" class="ltx_text ltx_font_bold">{user_query}
<br class="ltx_break"></span>[The Start of Assistant A’s Answer]</span>
<span id="A1.F8.pic1.1.1.1.1.1.4" class="ltx_p"><span id="A1.F8.pic1.1.1.1.1.1.4.1" class="ltx_text ltx_font_bold">{response_a}
<br class="ltx_break"></span>[The End of Assistant A’s Answer]</span>
<span id="A1.F8.pic1.1.1.1.1.1.5" class="ltx_p">[The Start of Assistant B’s Answer]</span>
<span id="A1.F8.pic1.1.1.1.1.1.6" class="ltx_p"><span id="A1.F8.pic1.1.1.1.1.1.6.1" class="ltx_text ltx_font_bold">{response_b}
<br class="ltx_break"></span>[The End of Assistant B’s Answer]</span>
<span id="A1.F8.pic1.1.1.1.1.1.7" class="ltx_p">[Evaluation Criterion]</span>
<span id="A1.F8.pic1.1.1.1.1.1.8" class="ltx_p"><span id="A1.F8.pic1.1.1.1.1.1.8.1" class="ltx_text ltx_font_bold">{eval_criterion}
<br class="ltx_break"></span>[End of Evaluation Criterion]
Evaluation of <span id="A1.F8.pic1.1.1.1.1.1.8.2" class="ltx_text ltx_font_bold">{criterion_name}</span>:</span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span><span id="A1.F8.2.1" class="ltx_text ltx_font_bold">BSM: Solve prompt.</span></figcaption>
</figure>
<figure id="A1.F9" class="ltx_figure"><svg id="A1.F9.pic1" class="ltx_picture ltx_centering" height="169.64" overflow="visible" version="1.1" width="600"><g transform="translate(0,169.64) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 151.92 C 0 161.7 7.93 169.64 17.72 169.64 L 582.28 169.64 C 592.07 169.64 600 161.7 600 151.92 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 151.92 C 1.97 160.62 9.02 167.67 17.72 167.67 L 582.28 167.67 C 590.98 167.67 598.03 160.62 598.03 151.92 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="142.08" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F9.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F9.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F9.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">{solve_output}
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.1.1.1.1.1.2" class="ltx_p">Instruction: You are requested to combine the five points that were previously discussed. For each point, provide a concise explanation to clarify its relevance. Also, include the respective score for each point to ensure a thorough understanding. Once you’ve done this, please draft a summary paragraph that encapsulates an overall evaluation based on these five points. Finally, present your conclusive judgement. Use the format "[[A]]" if you determine assistant A to be superior, "[[B]]" if you find assistant B to be better, and "[[C]]" in case of a tie.</span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span><span id="A1.F9.2.1" class="ltx_text ltx_font_bold">BSM: LLM merge prompt.</span></figcaption>
</figure>
<figure id="A1.F10" class="ltx_figure"><svg id="A1.F10.pic1" class="ltx_picture ltx_centering" height="74.6" overflow="visible" version="1.1" width="600"><g transform="translate(0,74.6) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 56.89 C 0 66.67 7.93 74.6 17.72 74.6 L 582.28 74.6 C 592.07 74.6 600 66.67 600 56.89 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 56.89 C 1.97 65.59 9.02 72.64 17.72 72.64 L 582.28 72.64 C 590.98 72.64 598.03 65.59 598.03 56.89 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="47.05" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F10.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F10.pic1.1.1.1.1.1.1" class="ltx_p">Your task is to answer the question below. Give step by step reasoning before you answer, and when you’re ready to answer, please use the format "Final answer: …"</span>
<span id="A1.F10.pic1.1.1.1.1.1.2" class="ltx_p">Question: <span id="A1.F10.pic1.1.1.1.1.1.2.1" class="ltx_text ltx_font_bold">{input}</span></span>
<span id="A1.F10.pic1.1.1.1.1.1.3" class="ltx_p">Solution:</span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span><span id="A1.F10.3.1" class="ltx_text ltx_font_bold">Chain-of-Thought prompt.</span> We consider this prompt as the one that provides a formatting requirement for the model so that answer extraction <span id="A1.F10.4.2" class="ltx_text ltx_font_italic">without</span> the CoT is feasible.</figcaption>
</figure>
<figure id="A1.F11" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A1.F11.1" class="ltx_p ltx_figure_panel ltx_align_center">Question</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F11.pic1" class="ltx_picture ltx_centering ltx_figure_panel" height="56.46" overflow="visible" version="1.1" width="600"><g transform="translate(0,56.46) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 38.75 C 0 48.53 7.93 56.46 17.72 56.46 L 582.28 56.46 C 592.07 56.46 600 48.53 600 38.75 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 38.75 C 1.97 47.44 9.02 54.49 17.72 54.49 L 582.28 54.49 C 590.98 54.49 598.03 47.44 598.03 38.75 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="28.9" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F11.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F11.pic1.1.1.1.1.1.1" class="ltx_p">Which writer was the father of Christopher Robin? I think the answer is Roald Dahl but I’m really not sure.</span>
</span></foreignObject></g></g></svg></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A1.F11.2" class="ltx_p ltx_figure_panel ltx_align_center">System 1 LLM response</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F11.pic2" class="ltx_picture ltx_centering ltx_figure_panel" height="156.09" overflow="visible" version="1.1" width="600"><g transform="translate(0,156.09) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 138.37 C 0 148.16 7.93 156.09 17.72 156.09 L 582.28 156.09 C 592.07 156.09 600 148.16 600 138.37 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 138.37 C 1.97 147.07 9.02 154.12 17.72 154.12 L 582.28 154.12 C 590.98 154.12 598.03 147.07 598.03 138.37 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="128.53" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F11.pic2.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F11.pic2.1.1.1.1.1.1" class="ltx_p">No, the writer who was the father of Christopher Robin was A.A. Milne. A.A. Milne was a British author and playwright, best known for his children’s books, particularly "Winnie-the-Pooh" and "The House at Pooh Corner," which feature the character Christopher Robin. Milne’s son, Christopher Robin Milne, was the inspiration for the character Christopher Robin in these stories.</span>
<span id="A1.F11.pic2.1.1.1.1.1.2" class="ltx_p">Roald Dahl, on the other hand, was a British novelist, short story writer, and screenwriter, best known for his children’s books such as "Charlie and the Chocolate Factory," "Matilda," and "The BFG." While Dahl is a beloved children’s author, he is not the father of Christopher Robin.</span>
</span></foreignObject></g></g></svg></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A1.F11.3" class="ltx_p ltx_figure_panel ltx_align_center">S2A System 2 response</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F11.pic3" class="ltx_picture ltx_centering ltx_figure_panel" height="106.28" overflow="visible" version="1.1" width="600"><g transform="translate(0,106.28) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 88.56 C 0 98.34 7.93 106.28 17.72 106.28 L 582.28 106.28 C 592.07 106.28 600 98.34 600 88.56 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 88.56 C 1.97 97.26 9.02 104.31 17.72 104.31 L 582.28 104.31 C 590.98 104.31 598.03 97.26 598.03 88.56 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="78.72" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F11.pic3.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F11.pic3.1.1.1.1.1.1" class="ltx_p">The writer who was the father of Christopher Robin is A.A. Milne. Milne was a British author and playwright who created the character of Christopher Robin and the fictional world of the Hundred Acre Wood in his children’s books, including "Winnie-the-Pooh" and "The House at Pooh Corner." Christopher Robin Milne, the real-life son of A.A. Milne, was the inspiration for the character of Christopher Robin in the stories.</span>
</span></foreignObject></g></g></svg></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A1.F11.4" class="ltx_p ltx_figure_panel ltx_align_center">Distilled System 2 response</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F11.pic4" class="ltx_picture ltx_centering ltx_figure_panel" height="39.86" overflow="visible" version="1.1" width="600"><g transform="translate(0,39.86) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 22.14 C 0 31.93 7.93 39.86 17.72 39.86 L 582.28 39.86 C 592.07 39.86 600 31.93 600 22.14 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 22.14 C 1.97 30.84 9.02 37.89 17.72 37.89 L 582.28 37.89 C 590.98 37.89 598.03 30.84 598.03 22.14 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F11.pic4.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F11.pic4.1.1.1.1.1.1" class="ltx_p">The father of Christopher Robin is A.A. Milne.</span>
</span></foreignObject></g></g></svg></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11: </span><span id="A1.F11.6.1" class="ltx_text ltx_font_bold">System 2 Attention examples.</span> Representative examples showing the difference in conciseness and style of predicted responses from baselines and distilled System 2 model. Note that S2A System 2 response only presents the final (2nd stage) response here without intermediate generations.</figcaption>
</figure>
<figure id="A1.F12" class="ltx_figure"><svg id="A1.F12.pic1" class="ltx_picture ltx_centering" height="73.22" overflow="visible" version="1.1" width="600"><g transform="translate(0,73.22) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B8CEBE" fill-opacity="1.0"><path d="M 0 17.72 L 0 55.5 C 0 65.29 7.93 73.22 17.72 73.22 L 582.28 73.22 C 592.07 73.22 600 65.29 600 55.5 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E5EDE6" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 55.5 C 1.97 64.2 9.02 71.25 17.72 71.25 L 582.28 71.25 C 590.98 71.25 598.03 64.2 598.03 55.5 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="45.66" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F12.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F12.pic1.1.1.1.1.1.1" class="ltx_p">Read the question: <span id="A1.F12.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">{question}</span>. 20 answers have been provided: <span id="A1.F12.pic1.1.1.1.1.1.1.2" class="ltx_text ltx_font_bold">{answers}</span> Carefully read all answers and compose the final answer such that it is consistent with the majority of variants provided. Provide final answer as "FINAL: &lt;your final answer&gt;</span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 12: </span><span id="A1.F12.2.1" class="ltx_text ltx_font_bold">Universal Self-Consistency prompt.</span> We use USC <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite> in order to find the most consistent final answer from a set sampled by the S2A model.</figcaption>
</figure>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Experiment Details</h3>

<section id="A1.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Model training</h5>

<div id="A1.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px1.p1.3" class="ltx_p">We use Llama2 70B Chat as the initialization for SFT training with CE loss. The loss is only applied on the answer part of the sequence. Model is trained with dropout <math id="A1.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="0.1" display="inline"><semantics id="A1.SS2.SSS0.Px1.p1.1.m1.1a"><mn id="A1.SS2.SSS0.Px1.p1.1.m1.1.1" xref="A1.SS2.SSS0.Px1.p1.1.m1.1.1.cmml">0.1</mn><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p1.1.m1.1b"><cn type="float" id="A1.SS2.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A1.SS2.SSS0.Px1.p1.1.m1.1.1">0.1</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p1.1.m1.1c">0.1</annotation></semantics></math>, learning rate <math id="A1.SS2.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="5.5e-6" display="inline"><semantics id="A1.SS2.SSS0.Px1.p1.2.m2.1a"><mrow id="A1.SS2.SSS0.Px1.p1.2.m2.1.1" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.cmml"><mrow id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.cmml"><mn id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.2" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.2.cmml">5.5</mn><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.1" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.1.cmml">​</mo><mi id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.3" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.3.cmml">e</mi></mrow><mo id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.1" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.1.cmml">−</mo><mn id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.3" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.3.cmml">6</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p1.2.m2.1b"><apply id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1"><minus id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.1"></minus><apply id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2"><times id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.1.cmml" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.1"></times><cn type="float" id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.2.cmml" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.2">5.5</cn><ci id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.3.cmml" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.2.3">𝑒</ci></apply><cn type="integer" id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.3.cmml" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.3">6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p1.2.m2.1c">5.5e-6</annotation></semantics></math>, with warmup <math id="A1.SS2.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="1" display="inline"><semantics id="A1.SS2.SSS0.Px1.p1.3.m3.1a"><mn id="A1.SS2.SSS0.Px1.p1.3.m3.1.1" xref="A1.SS2.SSS0.Px1.p1.3.m3.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p1.3.m3.1b"><cn type="integer" id="A1.SS2.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A1.SS2.SSS0.Px1.p1.3.m3.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p1.3.m3.1c">1</annotation></semantics></math>. Table <a href="#A1.T5" title="Table 5 ‣ S2A ‣ A.2 Experiment Details ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> shows details about total training steps and total training tokens per step.</p>
</div>
</section>
<section id="A1.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">S2A</h5>

<div id="A1.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px2.p1.1" class="ltx_p">For S2A, in both generation stages we use nucleus sampling with top-p value <math id="A1.SS2.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="0.9" display="inline"><semantics id="A1.SS2.SSS0.Px2.p1.1.m1.1a"><mn id="A1.SS2.SSS0.Px2.p1.1.m1.1.1" xref="A1.SS2.SSS0.Px2.p1.1.m1.1.1.cmml">0.9</mn><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px2.p1.1.m1.1b"><cn type="float" id="A1.SS2.SSS0.Px2.p1.1.m1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p1.1.m1.1.1">0.9</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px2.p1.1.m1.1c">0.9</annotation></semantics></math>.
During distillation, for USC, in some cases the generated answers are too long and 20 do not fit in the Llama2 context. In these rare cases we reduce the answer set to 10 or select an answer randomly if 10 generated answers are still too long.</p>
</div>
<figure id="A1.T5" class="ltx_table">
<table id="A1.T5.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A1.T5.1.1" class="ltx_tr">
<td id="A1.T5.1.1.1" class="ltx_td ltx_align_center ltx_border_tt">Methods</td>
<td id="A1.T5.1.1.2" class="ltx_td ltx_align_center ltx_border_tt">Dataset</td>
<td id="A1.T5.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">Total Training Steps</td>
<td id="A1.T5.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">Total Training Tokens per Step</td>
</tr>
<tr id="A1.T5.1.2" class="ltx_tr">
<td id="A1.T5.1.2.1" class="ltx_td ltx_align_center ltx_border_t">RaR</td>
<td id="A1.T5.1.2.2" class="ltx_td ltx_align_center ltx_border_t">Last Letter Concatenation</td>
<td id="A1.T5.1.2.3" class="ltx_td ltx_align_center ltx_border_t">3</td>
<td id="A1.T5.1.2.4" class="ltx_td ltx_align_center ltx_border_t">66k</td>
</tr>
<tr id="A1.T5.1.3" class="ltx_tr">
<td id="A1.T5.1.3.1" class="ltx_td ltx_align_center">RaR</td>
<td id="A1.T5.1.3.2" class="ltx_td ltx_align_center">Coin Flip</td>
<td id="A1.T5.1.3.3" class="ltx_td ltx_align_center">100</td>
<td id="A1.T5.1.3.4" class="ltx_td ltx_align_center">66k</td>
</tr>
<tr id="A1.T5.1.4" class="ltx_tr">
<td id="A1.T5.1.4.1" class="ltx_td ltx_align_center">S2A</td>
<td id="A1.T5.1.4.2" class="ltx_td ltx_align_center">TriviaQA</td>
<td id="A1.T5.1.4.3" class="ltx_td ltx_align_center">350</td>
<td id="A1.T5.1.4.4" class="ltx_td ltx_align_center">23k</td>
</tr>
<tr id="A1.T5.1.5" class="ltx_tr">
<td id="A1.T5.1.5.1" class="ltx_td ltx_align_center">BSM</td>
<td id="A1.T5.1.5.2" class="ltx_td ltx_align_center">OASST2</td>
<td id="A1.T5.1.5.3" class="ltx_td ltx_align_center">600</td>
<td id="A1.T5.1.5.4" class="ltx_td ltx_align_center">131k</td>
</tr>
<tr id="A1.T5.1.6" class="ltx_tr">
<td id="A1.T5.1.6.1" class="ltx_td ltx_align_center ltx_border_bb">CoT</td>
<td id="A1.T5.1.6.2" class="ltx_td ltx_align_center ltx_border_bb">GSM8K</td>
<td id="A1.T5.1.6.3" class="ltx_td ltx_align_center ltx_border_bb">5000</td>
<td id="A1.T5.1.6.4" class="ltx_td ltx_align_center ltx_border_bb">33k</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Experimental Details</figcaption>
</figure>
<figure id="A1.F13" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F13.pic1" class="ltx_picture ltx_centering ltx_figure_panel" height="390.09" overflow="visible" version="1.1" width="600"><g transform="translate(0,390.09) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#AFC2D5" fill-opacity="1.0"><path d="M 0 17.72 L 0 372.37 C 0 382.16 7.93 390.09 17.72 390.09 L 582.28 390.09 C 592.07 390.09 600 382.16 600 372.37 L 600 17.72 C 600 7.93 592.07 0 582.28 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E1E8EF" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 372.37 C 1.97 381.07 9.02 388.12 17.72 388.12 L 582.28 388.12 C 590.98 388.12 598.03 381.07 598.03 372.37 L 598.03 17.72 C 598.03 9.02 590.98 1.97 582.28 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="362.53" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F13.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="A1.F13.pic1.1.1.1.1.1.1" class="ltx_p">Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. Output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better, and "[[C]]" for a tie.</span>
<span id="A1.F13.pic1.1.1.1.1.1.2" class="ltx_p"><span id="A1.F13.pic1.1.1.1.1.1.2.1" class="ltx_text ltx_font_bold">[User Question]</span></span>
<span id="A1.F13.pic1.1.1.1.1.1.3" class="ltx_p">My coffee is tasting quite watery lately… what am i doing wrong?</span>
<span id="A1.F13.pic1.1.1.1.1.1.4" class="ltx_p"><span id="A1.F13.pic1.1.1.1.1.1.4.1" class="ltx_text ltx_font_bold">[The Start of Assistant A’s Answer]
<br class="ltx_break"></span>It’s possible that you are not using the right amount of coffee grinds for your recipe. If you measure out the right amount, it will help ensure that the flavor of your coffee isn’t too weak. Also, make sure that you are grinding your coffee beans properly. If the grinds are too fine, it can make your coffee taste watery. You should also make sure that you are using fresh, high-quality beans, as this will make a big difference in the flavor of your coffee. Finally, the water you are using can also affect the flavor of your coffee. If you are using tap water, try using filtered or bottled water instead.</span>
<span id="A1.F13.pic1.1.1.1.1.1.5" class="ltx_p"><span id="A1.F13.pic1.1.1.1.1.1.5.1" class="ltx_text ltx_font_bold">[The End of Assistant A’s Answer]
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.1.1.1.1.1.6" class="ltx_p"><span id="A1.F13.pic1.1.1.1.1.1.6.1" class="ltx_text ltx_font_bold">[The Start of Assistant B’s Answer]
<br class="ltx_break"></span>There are many ways to make coffee. For me to help you I need to know how you made your coffee. Did you make drip coffee?</span>
<span id="A1.F13.pic1.1.1.1.1.1.7" class="ltx_p"><span id="A1.F13.pic1.1.1.1.1.1.7.1" class="ltx_text ltx_font_bold">[The End of Assistant B’s Answer]</span></span>
</span></foreignObject></g></g></svg></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F13.pic2" class="ltx_picture ltx_centering ltx_figure_panel" height="855.01" overflow="visible" version="1.1" width="594"><g transform="translate(0,855.01) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#AFC2D5" fill-opacity="1.0"><path d="M 0 17.72 L 0 837.3 C 0 847.08 7.93 855.01 17.72 855.01 L 576.29 855.01 C 586.07 855.01 594 847.08 594 837.3 L 594 17.72 C 594 7.93 586.07 0 576.29 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E1E8EF" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 837.3 C 1.97 845.99 9.02 853.04 17.72 853.04 L 576.29 853.04 C 584.98 853.04 592.03 845.99 592.03 837.3 L 592.03 17.72 C 592.03 9.02 584.98 1.97 576.29 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="550.7" height="827.45" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F13.pic2.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:398.0pt;">
<span id="A1.F13.pic2.1.1.1.1.1.1" class="ltx_p">Here are the five points of evaluation and their respective scores:</span>
<span id="A1.F13.pic2.1.1.1.1.1.2" class="ltx_p">1. Relevance:</span>
<span id="A1.F13.pic2.1.1.1.1.1.3" class="ltx_p">* Assistant A: 4/5</span>
<span id="A1.F13.pic2.1.1.1.1.1.4" class="ltx_p">* Assistant B: 2/5</span>
<span id="A1.F13.pic2.1.1.1.1.1.5" class="ltx_p">Explanation: Assistant A provided specific and relevant suggestions to address the user’s concern about their watery coffee. In contrast, Assistant B did not provide any concrete solutions and asked for more information.</span>
<span id="A1.F13.pic2.1.1.1.1.1.6" class="ltx_p">2. Accuracy:</span>
<span id="A1.F13.pic2.1.1.1.1.1.7" class="ltx_p">* Assistant A: 4/5</span>
<span id="A1.F13.pic2.1.1.1.1.1.8" class="ltx_p">* Assistant B: 2/5</span>
<span id="A1.F13.pic2.1.1.1.1.1.9" class="ltx_p">Explanation: Assistant A provided accurate information about coffee brewing techniques and factors that could affect the taste of the coffee. Assistant B did not provide any relevant or accurate information.</span>
<span id="A1.F13.pic2.1.1.1.1.1.10" class="ltx_p">3. Clarity:</span>
<span id="A1.F13.pic2.1.1.1.1.1.11" class="ltx_p">* Assistant A: 4/5</span>
<span id="A1.F13.pic2.1.1.1.1.1.12" class="ltx_p">* Assistant B: 2/5</span>
<span id="A1.F13.pic2.1.1.1.1.1.13" class="ltx_p">Explanation: Assistant A’s response was clear and concise, but it contained some technical terms. Assistant B’s response was not clear and did not provide any helpful information.</span>
<span id="A1.F13.pic2.1.1.1.1.1.14" class="ltx_p">4. Helpfulness:</span>
<span id="A1.F13.pic2.1.1.1.1.1.15" class="ltx_p">* Assistant A: 4/5</span>
<span id="A1.F13.pic2.1.1.1.1.1.16" class="ltx_p">* Assistant B: 2/5</span>
<span id="A1.F13.pic2.1.1.1.1.1.17" class="ltx_p">Explanation: Assistant A provided actionable steps and suggestions that could help the user improve the taste of their coffee. Assistant B did not provide any concrete solutions or suggestions.</span>
<span id="A1.F13.pic2.1.1.1.1.1.18" class="ltx_p">5. Personalization:</span>
<span id="A1.F13.pic2.1.1.1.1.1.19" class="ltx_p">* Assistant A: 4/5</span>
<span id="A1.F13.pic2.1.1.1.1.1.20" class="ltx_p">* Assistant B: 1/5</span>
<span id="A1.F13.pic2.1.1.1.1.1.21" class="ltx_p">Explanation: Assistant A took into account the user’s coffee brewing method and bean preference. Assistant B did not provide any personalized recommendations or take into account the user’s specific needs.</span>
<span id="A1.F13.pic2.1.1.1.1.1.22" class="ltx_p">Overall evaluation:</span>
<span id="A1.F13.pic2.1.1.1.1.1.23" class="ltx_p">Both assistants provided responses that were generally clear and concise. However, Assistant A’s response was more relevant, accurate, helpful, and personalized than Assistant B’s response. Specifically, Assistant A provided specific suggestions and actionable steps that could help the user improve the taste of their coffee, while Assistant B did not provide any concrete solutions or suggestions.</span>
<span id="A1.F13.pic2.1.1.1.1.1.24" class="ltx_p">Conclusive judgement: [[A]]</span>
</span></foreignObject></g></g></svg></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><svg id="A1.F13.pic3" class="ltx_picture ltx_centering ltx_figure_panel" height="41.4" overflow="visible" version="1.1" width="594"><g transform="translate(0,41.4) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#AFC2D5" fill-opacity="1.0"><path d="M 0 17.72 L 0 23.68 C 0 33.46 7.93 41.4 17.72 41.4 L 576.29 41.4 C 586.07 41.4 594 33.46 594 23.68 L 594 17.72 C 594 7.93 586.07 0 576.29 0 L 17.72 0 C 7.93 0 0 7.93 0 17.72 Z" style="stroke:none"></path></g><g fill="#E1E8EF" fill-opacity="1.0"><path d="M 1.97 17.72 L 1.97 23.68 C 1.97 32.38 9.02 39.43 17.72 39.43 L 576.29 39.43 C 584.98 39.43 592.03 32.38 592.03 23.68 L 592.03 17.72 C 592.03 9.02 584.98 1.97 576.29 1.97 L 17.72 1.97 C 9.02 1.97 1.97 9.02 1.97 17.72 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="550.7" height="13.84" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F13.pic3.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:398.0pt;">
<span id="A1.F13.pic3.1.1.1.1.1.1" class="ltx_p">Conclusive judgement: [[A]]</span>
</span></foreignObject></g></g></svg></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 13: </span><span id="A1.F13.2.1" class="ltx_text ltx_font_bold">BSM training data sample</span>. (Up) Input data (Middle) target data for distilling BSM (Down) target data for distilling BSM (label only).</figcaption>
</figure>
<figure id="A1.T6" class="ltx_table">
<table id="A1.T6.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A1.T6.1.1" class="ltx_tr">
<td id="A1.T6.1.1.1" class="ltx_td ltx_border_tt"></td>
<td id="A1.T6.1.1.2" class="ltx_td ltx_align_center ltx_border_tt">writing</td>
<td id="A1.T6.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">reasoning</td>
<td id="A1.T6.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">math</td>
<td id="A1.T6.1.1.5" class="ltx_td ltx_align_center ltx_border_tt">humanities</td>
<td id="A1.T6.1.1.6" class="ltx_td ltx_align_center ltx_border_tt">roleplay</td>
<td id="A1.T6.1.1.7" class="ltx_td ltx_align_center ltx_border_tt">coding</td>
<td id="A1.T6.1.1.8" class="ltx_td ltx_align_center ltx_border_tt">extraction</td>
<td id="A1.T6.1.1.9" class="ltx_td ltx_align_center ltx_border_tt">stem</td>
</tr>
<tr id="A1.T6.1.2" class="ltx_tr">
<td id="A1.T6.1.2.1" class="ltx_td ltx_align_left ltx_border_t">gpt-4-0125-preview</td>
<td id="A1.T6.1.2.2" class="ltx_td ltx_align_center ltx_border_t">65.38%</td>
<td id="A1.T6.1.2.3" class="ltx_td ltx_align_center ltx_border_t">78.79%</td>
<td id="A1.T6.1.2.4" class="ltx_td ltx_align_center ltx_border_t">73.33%</td>
<td id="A1.T6.1.2.5" class="ltx_td ltx_align_center ltx_border_t">75.17%</td>
<td id="A1.T6.1.2.6" class="ltx_td ltx_align_center ltx_border_t">69.94%</td>
<td id="A1.T6.1.2.7" class="ltx_td ltx_align_center ltx_border_t">78.57%</td>
<td id="A1.T6.1.2.8" class="ltx_td ltx_align_center ltx_border_t">76.32%</td>
<td id="A1.T6.1.2.9" class="ltx_td ltx_align_center ltx_border_t">75.51%</td>
</tr>
<tr id="A1.T6.1.3" class="ltx_tr">
<td id="A1.T6.1.3.1" class="ltx_td ltx_align_left">llama2-70b-chat</td>
<td id="A1.T6.1.3.2" class="ltx_td ltx_align_center">48.98%</td>
<td id="A1.T6.1.3.3" class="ltx_td ltx_align_center">54.89%</td>
<td id="A1.T6.1.3.4" class="ltx_td ltx_align_center">60.00%</td>
<td id="A1.T6.1.3.5" class="ltx_td ltx_align_center">66.67%</td>
<td id="A1.T6.1.3.6" class="ltx_td ltx_align_center">58.89%</td>
<td id="A1.T6.1.3.7" class="ltx_td ltx_align_center">62.17%</td>
<td id="A1.T6.1.3.8" class="ltx_td ltx_align_center">48.54%</td>
<td id="A1.T6.1.3.9" class="ltx_td ltx_align_center">66.67%</td>
</tr>
<tr id="A1.T6.1.4" class="ltx_tr">
<td id="A1.T6.1.4.1" class="ltx_td ltx_align_left">BSM</td>
<td id="A1.T6.1.4.2" class="ltx_td ltx_align_center">63.08%</td>
<td id="A1.T6.1.4.3" class="ltx_td ltx_align_center">64.65%</td>
<td id="A1.T6.1.4.4" class="ltx_td ltx_align_center">61.67%</td>
<td id="A1.T6.1.4.5" class="ltx_td ltx_align_center">70.74%</td>
<td id="A1.T6.1.4.6" class="ltx_td ltx_align_center">65.64%</td>
<td id="A1.T6.1.4.7" class="ltx_td ltx_align_center">70.63%</td>
<td id="A1.T6.1.4.8" class="ltx_td ltx_align_center">57.02%</td>
<td id="A1.T6.1.4.9" class="ltx_td ltx_align_center">76.19</td>
</tr>
<tr id="A1.T6.1.5" class="ltx_tr">
<td id="A1.T6.1.5.1" class="ltx_td ltx_align_left ltx_border_t">Distill System 1</td>
<td id="A1.T6.1.5.2" class="ltx_td ltx_align_center ltx_border_t">53.59%</td>
<td id="A1.T6.1.5.3" class="ltx_td ltx_align_center ltx_border_t">66.00%</td>
<td id="A1.T6.1.5.4" class="ltx_td ltx_align_center ltx_border_t">54.72%</td>
<td id="A1.T6.1.5.5" class="ltx_td ltx_align_center ltx_border_t">67.11%</td>
<td id="A1.T6.1.5.6" class="ltx_td ltx_align_center ltx_border_t">62.17%</td>
<td id="A1.T6.1.5.7" class="ltx_td ltx_align_center ltx_border_t">67.73%</td>
<td id="A1.T6.1.5.8" class="ltx_td ltx_align_center ltx_border_t">43.86%</td>
<td id="A1.T6.1.5.9" class="ltx_td ltx_align_center ltx_border_t">70.07%</td>
</tr>
<tr id="A1.T6.1.6" class="ltx_tr">
<td id="A1.T6.1.6.1" class="ltx_td ltx_align_left">Distill System 2</td>
<td id="A1.T6.1.6.2" class="ltx_td ltx_align_center">68.46%</td>
<td id="A1.T6.1.6.3" class="ltx_td ltx_align_center">67.34%</td>
<td id="A1.T6.1.6.4" class="ltx_td ltx_align_center">67.78%</td>
<td id="A1.T6.1.6.5" class="ltx_td ltx_align_center">74.94%</td>
<td id="A1.T6.1.6.6" class="ltx_td ltx_align_center">68.30%</td>
<td id="A1.T6.1.6.7" class="ltx_td ltx_align_center">70.64%</td>
<td id="A1.T6.1.6.8" class="ltx_td ltx_align_center">61.69%</td>
<td id="A1.T6.1.6.9" class="ltx_td ltx_align_center">75.51%</td>
</tr>
<tr id="A1.T6.1.7" class="ltx_tr">
<td id="A1.T6.1.7.1" class="ltx_td ltx_align_left ltx_border_bb">Distill System 2 (label only)</td>
<td id="A1.T6.1.7.2" class="ltx_td ltx_align_center ltx_border_bb">70.77%</td>
<td id="A1.T6.1.7.3" class="ltx_td ltx_align_center ltx_border_bb">70.71%</td>
<td id="A1.T6.1.7.4" class="ltx_td ltx_align_center ltx_border_bb">76.95%</td>
<td id="A1.T6.1.7.5" class="ltx_td ltx_align_center ltx_border_bb">74.50%</td>
<td id="A1.T6.1.7.6" class="ltx_td ltx_align_center ltx_border_bb">68.92%</td>
<td id="A1.T6.1.7.7" class="ltx_td ltx_align_center ltx_border_bb">74.34%</td>
<td id="A1.T6.1.7.8" class="ltx_td ltx_align_center ltx_border_bb">61.70%</td>
<td id="A1.T6.1.7.9" class="ltx_td ltx_align_center ltx_border_bb">79.59%</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span><span id="A1.T6.3.1" class="ltx_text ltx_font_bold">System 2 Distillation of BSM</span>:
MT-bench per category agreement.</figcaption>
</figure>
<figure id="A1.T7" class="ltx_table">
<div id="A1.T7.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:421.6pt;height:172.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-52.7pt,21.6pt) scale(0.8,0.8) ;">
<table id="A1.T7.1.1" class="ltx_tabular ltx_align_middle">
<tr id="A1.T7.1.1.1" class="ltx_tr">
<td id="A1.T7.1.1.1.1" class="ltx_td ltx_border_tt"></td>
<td id="A1.T7.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt">Data Input Prompt</td>
<td id="A1.T7.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">Exact Match</td>
<td id="A1.T7.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">Miss Match Rate</td>
</tr>
<tr id="A1.T7.1.1.2" class="ltx_tr">
<td id="A1.T7.1.1.2.1" class="ltx_td ltx_align_left ltx_border_t">System 1</td>
<td id="A1.T7.1.1.2.2" class="ltx_td ltx_align_center ltx_border_t">{question}</td>
<td id="A1.T7.1.1.2.3" class="ltx_td ltx_align_center ltx_border_t">56.11%</td>
<td id="A1.T7.1.1.2.4" class="ltx_td ltx_align_center ltx_border_t">4.65%</td>
</tr>
<tr id="A1.T7.1.1.3" class="ltx_tr">
<td id="A1.T7.1.1.3.1" class="ltx_td ltx_align_left">System 1</td>
<td id="A1.T7.1.1.3.2" class="ltx_td ltx_align_center">{question} Flip means reverse.</td>
<td id="A1.T7.1.1.3.3" class="ltx_td ltx_align_center">66.84%</td>
<td id="A1.T7.1.1.3.4" class="ltx_td ltx_align_center">0.15%</td>
</tr>
<tr id="A1.T7.1.1.4" class="ltx_tr">
<td id="A1.T7.1.1.4.1" class="ltx_td ltx_align_left">System 1</td>
<td id="A1.T7.1.1.4.2" class="ltx_td ltx_align_center">{question} Flip means reverse. Answer the Yes or No question.</td>
<td id="A1.T7.1.1.4.3" class="ltx_td ltx_align_center">52.89%</td>
<td id="A1.T7.1.1.4.4" class="ltx_td ltx_align_center">0%</td>
</tr>
<tr id="A1.T7.1.1.5" class="ltx_tr">
<td id="A1.T7.1.1.5.1" class="ltx_td ltx_align_left">1 Step RaR</td>
<td id="A1.T7.1.1.5.2" class="ltx_td ltx_align_center">Prompt in Fig. <a href="#A1.F3" title="Figure 3 ‣ A.1 Prompts ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>
</td>
<td id="A1.T7.1.1.5.3" class="ltx_td ltx_align_center">58.51%</td>
<td id="A1.T7.1.1.5.4" class="ltx_td ltx_align_center">0%</td>
</tr>
<tr id="A1.T7.1.1.6" class="ltx_tr">
<td id="A1.T7.1.1.6.1" class="ltx_td ltx_align_left">2 Step RaR</td>
<td id="A1.T7.1.1.6.2" class="ltx_td ltx_align_center">Prompt in Fig. <a href="#A1.F5" title="Figure 5 ‣ A.1 Prompts ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>
</td>
<td id="A1.T7.1.1.6.3" class="ltx_td ltx_align_center">77.19%</td>
<td id="A1.T7.1.1.6.4" class="ltx_td ltx_align_center">0%</td>
</tr>
<tr id="A1.T7.1.1.7" class="ltx_tr">
<td id="A1.T7.1.1.7.1" class="ltx_td ltx_align_left ltx_border_t">Distill system 1</td>
<td id="A1.T7.1.1.7.2" class="ltx_td ltx_align_center ltx_border_t">{question}</td>
<td id="A1.T7.1.1.7.3" class="ltx_td ltx_align_center ltx_border_t">54.54%</td>
<td id="A1.T7.1.1.7.4" class="ltx_td ltx_align_center ltx_border_t">3.75%</td>
</tr>
<tr id="A1.T7.1.1.8" class="ltx_tr">
<td id="A1.T7.1.1.8.1" class="ltx_td ltx_align_left">Distill system 1</td>
<td id="A1.T7.1.1.8.2" class="ltx_td ltx_align_center">{question} Flip means reverse.</td>
<td id="A1.T7.1.1.8.3" class="ltx_td ltx_align_center">62.64%</td>
<td id="A1.T7.1.1.8.4" class="ltx_td ltx_align_center">1.13%</td>
</tr>
<tr id="A1.T7.1.1.9" class="ltx_tr">
<td id="A1.T7.1.1.9.1" class="ltx_td ltx_align_left">Distill system 1</td>
<td id="A1.T7.1.1.9.2" class="ltx_td ltx_align_center">{question} Flip means reverse. Answer the Yes or No question.</td>
<td id="A1.T7.1.1.9.3" class="ltx_td ltx_align_center">63.39%</td>
<td id="A1.T7.1.1.9.4" class="ltx_td ltx_align_center">0.60%</td>
</tr>
<tr id="A1.T7.1.1.10" class="ltx_tr">
<td id="A1.T7.1.1.10.1" class="ltx_td ltx_align_left">Distill system 2</td>
<td id="A1.T7.1.1.10.2" class="ltx_td ltx_align_center">{question}</td>
<td id="A1.T7.1.1.10.3" class="ltx_td ltx_align_center">75.69%</td>
<td id="A1.T7.1.1.10.4" class="ltx_td ltx_align_center">0%</td>
</tr>
<tr id="A1.T7.1.1.11" class="ltx_tr">
<td id="A1.T7.1.1.11.1" class="ltx_td ltx_align_left">Distill system 2</td>
<td id="A1.T7.1.1.11.2" class="ltx_td ltx_align_center">{question} Flip means reverse.</td>
<td id="A1.T7.1.1.11.3" class="ltx_td ltx_align_center">78.92%</td>
<td id="A1.T7.1.1.11.4" class="ltx_td ltx_align_center">0%</td>
</tr>
<tr id="A1.T7.1.1.12" class="ltx_tr">
<td id="A1.T7.1.1.12.1" class="ltx_td ltx_align_left ltx_border_bb">Distill system 2</td>
<td id="A1.T7.1.1.12.2" class="ltx_td ltx_align_center ltx_border_bb">{question} Flip means reverse. Answer the Yes or No question.</td>
<td id="A1.T7.1.1.12.3" class="ltx_td ltx_align_center ltx_border_bb">74.49%</td>
<td id="A1.T7.1.1.12.4" class="ltx_td ltx_align_center ltx_border_bb">0%</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span><span id="A1.T7.3.1" class="ltx_text ltx_font_bold">System 2 Distillation of Rephrase and Respond</span>: Coin flip task additional results.</figcaption>
</figure>
<figure id="A1.F14" class="ltx_figure"><img src="/html/2407.06023/assets/x2.png" id="A1.F14.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="316" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 14: </span>An illustration of Branch-solve-merge with LLama-2-70B-chat for pairwise evaluation of LLM response.
</figcaption>
</figure>
</section>
<section id="A1.SS2.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">BSM</h5>

<div id="A1.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px3.p1.1" class="ltx_p">Figure <a href="#A1.F14" title="Figure 14 ‣ S2A ‣ A.2 Experiment Details ‣ Appendix A Appendix ‣ Distilling System 2 into System 1" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a> shows the overview of Branch-solve-merge. We copied figure from <cite class="ltx_cite ltx_citemacro_citet">Saha et al. (<a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.06020" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.06023" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.06023">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.06023" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.06024" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 18:47:15 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.06023
Large language models (LLMs)  can spend extra compute during inference to generate intermediate thoughts, which helps to produce better final responses.
Since Chain-of-Thought,
many suchSystem 2techniques have been proposed such as Rephrase and Respond, System 2 Attentionand Branch-Solve-Merge. In this work we investigate self-supervised methods to “compile” (distill)  higher quality outputs from System 2 techniques back into LLM generationswithoutintermediate reasoning token sequences, as this reasoning has been distilled intoSystem 1. We show that several such techniques can be successfully distilled, resulting in improved results compared to the original System 1 performance, and with less inference cost than System 2. We posit that System 2 distillation will be an important feature of future continually learning AI systems, enabling them to focus System 2 capabilities
on the reasoning tasks that they cannot yet do well.
abstract
Distilling System 2 into System 1
Abstract
2407.06023
1IntroductionGenerating intermediate thoughts allows a model (or human!)  to reason and plan in order to successfully complete a task or respond to an instruction.
We refer to such deliberate thinking as System 2 reasoning, following its description for humans inand later for AI models. In System 2 reasoning effortful mental activity is exerted, especially in situations where System 1 – more automatic thinking – is likely to make errors.
In standard Large Language Models (LLMs)  we thus defineSystem 1as application of the Transformerto directly produce a response given an input, without generation of intermediate tokens.
We defineSystem 2as any approach which generates intermediate tokens, including methods that perform search, or prompt multiple times, before finally generating a response.
A battery of suchSystem 2techniques have been proposed, among them Chain-of-Thought, Tree-of-Thoughts, Graph-of-Thoughts, Branch-Solve-Merge, System 2 Attention, Rephrase and Respondand more.
Many of these methods are shown to produce more accurate results due to this explicit reasoning, but typically do so at much higher inference cost and latency for a response.
Due to the latter, many of these approaches are not used in production systems, which mostly useSystem 1generations.Figure 1:Overview of System 2 Distillation.Filtered training examples are collected by running System 2 approaches
such as Branch-Solve-Merge (BSM) 
on unlabeled data, which uses extra compute to produce higher quality outputs. These targets are then distilled into the standard (System 1)  LLM.For a human, the process of learning to transfer a skill from deliberate (System 2) 
to automatic (System 1)  in psychology
is referred to asautomaticity, and the use ofprocedural memory.
For example, when driving to work for the first time one might typically
expend conscious effort planning and making decisions to get there.
After a driver repeats this route, the driving process becomes “compiled” into the subconscious. Similarly, playing a sport such as tennis can become “second nature”.
In this work, we explore an analogous technique for AI models.
Our approach performs this compilation, which we refer to asSystem 2 distillation, in an unsupervised manner given a set of unlabeled examples.
For each example we apply the given System 2 method, and then measure the quality of the prediction in an unsupervised manner. For example, for tasks with unique answers we apply self-consistency, sampling multiple times. For examples where System 2 is consistent enough, we assume this result should be distilled, and add it to the distillation pool.
We then fine-tune System 1 to match the predictions of the System 2 method on the collected pool of examples, butwithoutgenerating the intermediate steps. Figure1illustrates the overall process of distilling System 2 into System 1.We conduct experiments across 4 different System 2 LLM approaches and 5 different tasks. We find our approach can distill System 2 reasoning into System 1 in a diverse array of settings, sometimes even improving the results over the System 2 teacher.
Moreover, these predictions are now produced at a fraction of the computational cost.
For example, we see successful distillation for tasks involving dealing with biased opinions or irrelevant information (System 2 Attention) , clarifying and improving responses in some reasoning tasks (Rephrase and Respond) , and for fine-grained evaluation of LLMs (Branch-Solve-Merge) . However, we also show that not all tasks can be distilled into System 1, particularly complex math reasoning tasks requiring chain-of-thought. This is also mirrored in humans,
who cannot execute some tasks without deliberate System 2 reasoning.
introduction
Distilling System 2 into System 1
1Introduction
2407.06023
2Related work2.1System 1 and System 2 in HumansIn humans, System 1 reasoning is described as being capable of recognizing patterns, making quick judgments, and understanding simple or familiar symbols. For instance, it is used to identify common traffic signs, recognize faces, or associate basic symbols with specific emotions or ideas.
However, for complex problem-solving or for example manipulation of abstract symbols (like algebraic equations or logical statements) , System 2 reasoning is deemed necessary.
In psychology the concept ofautomaticitydescribes behavior that becomes so well-practiced that it can be performed with little to no conscious thought, with an example being driving a familiar route.
In general, humans are said to useprocedural memoryto consolidate a specific task into memory, learning through practice, so that it can be later performed without conscious awareness.
The concept ofunconscious competenceis classified as a later stage of learning. Initially a person recognizes their incompetence, and consciously seeks to learn a skill until they acquireconscious competence. Finally, the aim is to utilize it without conscious thought when it is said to become, in common language, “second nature”.2.2System 1 and System 2 ModelsWe refer to a neural network that outputs a response directly without intermediate outputs as aSystem 1 model. Such a network can nevertheless compute intermediate latent representations in its layers before it outputs a response. As these states are represented as vectors they typically encode distributed knowledge, rather than discrete decisions, and have difficulty manipulating
complex symbolic reasoning tasks directly, which is analogous to issues with System 1 reasoning in humans. Nevertheless, a vast array of tasks can be solved with success directly in this manner without intermediate generations.showed that the same language model that is unable to perform complex multi-step computations can perform those tasks when asked to generate intermediate steps into a “scratchpad” using either few-shot prompting or supervised training.
Chain-of-thought reasoning was shown to be elicited from LLMs even using zero-shot promptingas well as by supervisedor few-shotmethods.
LLM pretraining allows such reasoning to be built into the model because reasoning steps in discrete symbols (text)  are present in the training corpora written by humans.
SuchSystem 2 modelapproaches output discrete tokens which is good for making sequential correct logical reasoning steps – but obviously has a downside if the reasoning is generated incorrectly.
An incorrect discrete decision is difficult to recover from, unlike latent vector-based reasoning that might more easily model a distribution.Recently, many approaches have been proposed to execute deeper reasoning using the LLM as part of an inner loop where it generates intermediate outputs, sometimes referred to asLLM Programs.
These include subquestion decomposition,
self-refinement, self-verification and asking,
and various search techniques such as Tree-of-Thoughts and others.2.3(Standard)  DistillationThe concept of distillation is usually applied to taking separate models, a powerful teacher model (or multiple teacher models)  and a less powerful student model with separate parameters.
The student model is then trained to mimic the behavior of the teacher(s) .
Methods of distillation include training the student to have similar
output distributions, layer activationsor derivatives of the target teacher outputs.
Earlier works considered distillation from an ensemble of multiple teacher models.
As neural networks have become larger, distilling from a larger to a smaller network has become a common paradigm.
In contrast, in our work the teacher and student model are the same language model, but applied differently (either with intermediate reasoning, or not) .For chain-of-thought reasoning in particular, several distillation approaches have been considered.
These again follow the paradigm of distilling aseparatelarger model’s output into a smaller model, i.e.
the student model is asked to
mimic the internal thoughts of the teacher model.
The work of, however, considers distilling
a slower System 2 method (Tree-of-Thought)  into a faster System 2 method (CoT) , which can use the same model as student and teacher.
In contrast our work’s goal is tonotgenerate internal thoughts (to improve System 1) .
Some exceptions are. The former still uses a separate student and teacher model, but attempts to distill the intermediate thought tokens into the layers of the network by representing reasoning steps as vectors and then setting them as targets. The latter recent work attempts to distill CoT by gradually removing the intermediate steps, which can improve performance greatly compared to not doing so, but still does not match explicit CoT.
body
Distilling System 2 into System 1
2Related work
2407.06023
3Distilling System 2 into System 13.1Setup: System 1 and System 2 modelsGiven an input, in this work we consider the setting of a single model, in our case a large language model (LLM) , that is capable of two modes of response:(i) System 1: Produces the outputdirectly. This is done by forwarding through the layers of the underlying autoregressive neural network (Transformer)  to produce the output tokens.(ii) System 2: We define System 2 models as methods that use the underlying Transformer to generate intermediate output tokensof any kindbeforegenerating the final response tokens. This may include multiple calls (prompts) .More formally, we consider a System 2 modelas a function that takes an LLMand input, and can call the LLM possibly repeatedly to generate intermediate tokensusing a specific algorithm, before returning an output:(1) System 2 approaches can potentially involve multiple prompts, branching, iteration and search, all the while using the LLM to generate intermediate results for further processing. In contrast, a System 1 model only considers the original inputand calls the LLMdirectly to produce an output:(2) There are many existing instantiations of System 2 models.
Chain-of-thought prompting only requires a single LLM prompt, but still outputs intermediate generations before a final response, typically used in math and other reasoning tasks.Methods like System 2 Attentionand Rephrase and Respondrequire two calls to the LLM, where in the former the first call is used to attend to the context and remove bias, and in the latter to expand on the question. The second call is then used to finally respond to the answer given the intermediate generations.
Some methods are much more sophisticated for example Branch-Solve-Mergewhich generates a plan via an LLM which branches into several more LLM calls until a final stage merges the results.We will perform experiments with the four methods just described, but there are many other system 2 approaches, for example Tree-of-Thoughts, Graph-of-Thoughtsand more, see related work insection 2.3.2Method: System 2 DistillationMany System 2 methods, by their nature, are significantly slower at inference time due to multiple prompt calls and generation of intermediate tokens. The aim of System 2 Distillation is to
distill all the reasoning fromback intoso that the direct outputs from the language modelare improved.
We assume a setting where the model has access tounlabeled inputsfrom which it can learn, in analogy to how humans learn theirprocedural memorywithout supervision.
For language-based tasks, it is common to have access to instruction following prompts (inputs)  as they can be collected from humans, e.g. the 1M released WildChat interactionswhere inputs are given but correct labels are unknown.
Hence this is a realistic setup.The first step of the proposed method is to generate responses using the System 2 model over the unlabeled inputs:(3) Note we discard (do not store)  the intermediate outputsfrom Eq.1. These responsescan then be used directly as System 2 distillation targets for fine-tuning a System 1 model. However, they are subject to noise: some of these responses could be high quality, while others could be low quality or incorrect.
For shortform QA and reasoning tasks involving a short response with a typically unique
correct (but unknown)  answer, we thus consider anunsupervised curationstep to attempt to improve training data quality. We consider two variations which both rely on a consistency criterion:•self-consistency of outputs: we samplea total oftimes, and accept the response that is the majority vote; if there is no majority winner, we discard the example.•self-consistency under input perturbation: we perturb the inputin such a way that the output should not change, e.g. changing the order of multiple-choice items in the prompt, and computefor each perturbation; if the outputs do not agree, we discard the example.After that, we end up with the synthetic dataset,
whereis a filtered subset ofwith targets.
The final step is then supervised fine-tuning of the LLM with parametersusing this distilled training set. We typically initialize this model from the current stateand continue training with the new dataset.After fine-tuning we obtain an LLMwhich is a System 1 model that is expected to provide outputs and performance gains similar to the
evaluated System 2 model.
body
Distilling System 2 into System 1
3Distilling System 2 into System 1
2407.06023
4Experiments4.1Training and Evaluation SetupWe use Llama-2-70B-chatas the base model for all our experiments.
We require a base model of sufficient power that it can be performant as a System 2 model, but also have open weights that can be fine-tuned, hence this choice. We consider several System 2 methods, including Rephrase and Respond (RaR) , System 2 Attention (S2A) , Branch-Solve-Merge (BSM) , and Chain-of-Thought (CoT) , focusing on tasks where each method has demonstrated strong performance. For System 1, we conduct zero-shot inference using the instruction-tuned base model as a standard baseline.
We report task-specific metrics for each task, and the “#Tokens” metric which measures the average number of tokens generated per input across the evaluation set. For System 2 methods this includes both intermediate token generations as well as the final output token generations. Detailed descriptions of the experimental setups are available in the AppendixA.2.4.2Rephrase and Respond DistillationRephrase and Response (RaR) is a System 2 method that first prompts the language model to rephrase the original question with further elaboration, and then secondly to generate a response based on the rephrased question with the aim that this provides superior output. The authors introduce two approaches, 1-step RaR and 2-step RaR, where the latter involves two separate prompts rather than a combined one as in the former, see AppendixA.1for specific prompts.
They find that 2-step RaR significantly improves performance on several reasoning tasks that are challenging for the baseline LLM.
We consider two tasks from the original paper where it performed well: the last letter concatenation task and coin flip reasoning.
We then assess whether it is possible to distill this System 2 approach.Distillation DataWe build the System 2 distillation dataset for RaR usingself-consistency of outputs. For each input, we conduct eight sampling iterations for the last letter task and eight for each stage of the coin flip task.111This approach was adopted after observing that sampling just once for the rephrase stage yielded suboptimal results.We then apply a majority vote to determine the final output.4.2.1Last letter Concatenation TaskThis task focuses on symbolic reasoning, requiring the model to concatenate the last letters of given words. For instance, the instruction: “Take the last letters of the words in ‘Edgar Bob’ and concatenate them.” As demonstrated in, this task benefits significantly from the application of the RaR method. We compiled a dataset by randomly selecting 1200 unique English words. Using this, we constructed 200 samples each for training, validation, and test.ResultsOverall results are given inTable 1. The baseline System 1 model (Llama-2-70B-chat)  achieves an accuracy of 30.0%, and is outperformed by the System 2 methods of 1-Step and 2-Step RaR (39.5% and 44.5%, respectively) .
Distilling the 2-Step RaR method back into a System 1 Llama-2-70B-chat model via our unsupervised technique, we achieve a remarkable accuracy of 98.0%.
The model can effectively learn from this training data how to solve the task, in comparison to the zero-shot chat model.
Distillation of Rephrase and Respond effectively inherits the advantages of both System 2 and System 1. It maintains the accuracy benefits of System 2, while its inference cost is comparable to that of System 1 (see # of generated Tokens) .Analysis & AblationsTo evaluate the effectiveness and necessity of our  unsupervised curation step usingself-consistency of outputswe conducted an ablation study by creating a distillation dataset without applying the self-consistency filter. When we distilled the System 2 model using this unfiltered dataset under the same setting, it achieved an exact match accuracy of 87.5% (with 98% for the filtered version) . This comparison underscores the critical role of consistency filtering. Nevethess, in both cases constructing training data does improve results over zero-shot performance.
We also attempted to distill the System 1 predictions using the same filtering technique, which results in a lower accuracy of 69.5%.Last LetterCoin FlipAcc#TokensAcc#TokensSystem 1Llama-2-70B-chat30.0%27.156.1%61.9Distill System 169.5%24.454.5%30.4System 21-Step RaR39.5%106.658.5%158.92-Step RaR44.5%41.577.2%112.4Distill System 2Distill 2-Step RaR98.0%25.575.69%50.3Table 1:System 2 Distillation of Rephrase and Respond:
Coin Flip and Last Letter Concatenation tasks.
We report exact match (EM)  test accuracy and number of generated (intermediate and output)  tokens.4.2.2Coin Flip Reasoning TaskThis symbolic reasoning task has frequently been tested in research, including inand. It involves determining the final face (heads or tails)  of a coin, starting from a known initial position after a series of flips described in natural language
, such as“A coin is heads up. Roxas does not flip the coin. Schneiderman does not flip the coin. Is the coin still heads up?”showed that even strong language models do not succeed at this task, whereas applying the RaR method improves their performance.
There are 20k training examples, which we use for unsupervised learning (without labels) , 3.33k validation and 1.33k test examples.ResultsOverall results are given inTable 1.
Llama-2-70B-chat (zero-shot)  has a success rate of 56.1% on this task, while 1-Step and 2-Step RaR have success rates of 58.5% and 77.2% respectively. We thus only see a large improvement with the 2-Step method.
Distilling 2-Step RaR back into a system 1 Llama-2-70B-chat via our unsupervised technique yields
75.69%. Hence, we find that
our distilled System 2 model delivers performance comparable to that of System 2 (2 Step RaR) , but without the need to execute the LLM program with 2 prompts (see # of generated Tokens) .Analysis & AblationsThe RaR method inincorporates prompt engineering tricks, such as appending phrases like “Flip means reverse. Answer the Yes or No question” to the original query, which has been shown to enhance model performance. Following their approach, we evaluated model performance using different prompts, seeTable 7. When testing the Llama-2-70B-chat model (System 1)  with prompts like “Flip means reverse” and “Flip means reverse. Answer the Yes or No question,” we observed a significant improvement in performance, from 56.11% to 66.84%. This highlights the critical role of prompt selection in optimizing the performance of System 1 models. However, this reliance on prompt engineering also represents a limitation, necessitating additional human effort.AccAccModel(biased) (unbiased) #TokensSystem 1(Zero-shot) 51.6%73.8%165System 2(S2A) 76.0%69.3%147Distill S2A81.3%78.6%56Distill S2A (no USC) 78.6%75.3%58Table 2:Distillation of System 2 Attention: TriviaQA task, reporting
accuracies on the biased and unbiased eval sets.We also attempted to distill the System 1 model, which gave poor performance. In this case, we also observed fluctuations in performance with different prompts. In contrast, the distilled System 2 model demonstrated consistent performance across various prompts, with a lower sensitivity to prompt variations. This consistency indicates that extensive prompt engineering might not be essential for the distilled System 2 model.4.3System 2 Attention Distillationproposed System 2 Attention (S2A) , a method that helps to reduce models’ reasoning pitfalls such as relying on biased information in the input or attending to irrelevant context. S2A is a two-stage inference method where the first stage rewrites the input so that it does not contain undesired information such as bias or irrelevant context, and the second stage attends to the shorter rewritten context (in contrast to RaR which expands the context) , seeFigure 6.
In this work we verify the feasibility of distilling S2A into System 1. In particular, we focus on the SycophancyEval question answering taskthat contains biased information in the input that is known to hurt LLM performance. We use 6668 examples from SycophancyEval as unlabeled training data, and 400 examples for evaluation, where the latter are split into biased inputs (350)  and without bias (50) .Distillation dataWe use universal self-consistency (USC) to select high quality targets. Specifically, we sample 20 generations and then use the
Llama-70B-chat model with a USC prompt (provided inFigure 12) 
to compose a self-consistent (majority)  final answer that is used as the distillation target.ResultsThe results are provided inTable 2,
reporting average accuracy over 3 random seeds.
The baseline (System 1)  LLM has low accuracy on the biased portion as expected, being susceptible to biased inputs. S2A improves performance dramatically for biased inputs. System 2 distillation shows similarly strong performance as the System 2 approach. There is, however, a signification reduction in the average number of tokens used compared to both the baseline and the S2A model. This is because biased inputs tend to make the baseline LLM generate more output tokens, while S2A has to generate intermediate tokens as well.Figure 11shows a representative example.
Finally, we show that using USC for distillation is important for overall results, by also reporting results without USC (last row) , where the latter provides inferior results. This highlights the importance of the distillation data quality that is used during fine-tuning.OASST2 EvalMT-bench EvalAgreement% Inconsistent#TokensAgreement% Inconsistent#TokensSystem 1GPT-4-0125-preview44.7%35.5%468.1%25.6%4Llama-2-70B-chat32.0%56.7%428.1%80.9%4System 2CoT (GPT-4-0125-preview) 48.7%28.2%603.773.8%16.2%548.8CoT (Llama-2-70B-chat) 45.2%37.7%432.658.9%30.8%411.8BSM (Llama-2-70B-chat) 49.1%30.4%2117.864.5%21.1%2063.1Distill System 2Distill BSM (Llama-2-70B-chat) 58.4%12.2%472.4%9.1%4Table 3:System 2 Distillation of Branch-Solve-Merge (BSM) :
Open Assistant (OASST2)  and MT-bench evaluation of LLM-as-a-Judge for various models. System 2 Distillation of BSM outperforms BSM itself, and even GPT4-as-a-Judge, despite using Llama-2-70B-chat. Distilled BSM has higher human agreement (Agreement) , less position inconsistent predictions (% Inconsistent) , and uses less output tokens (#Tokens) .4.4Branch-Solve-Merge DistillationBranch-Solve-Merge (BSM) , consists of three modules:branch,solve, andmerge. These modules work together to break down a task into several parallel sub-tasks, each guided by specific prompts. BSM has proven effective when used in the context of an LLM acting as a judge,
seeFigure 14.
The method begins by prompting the language model to list evaluation metrics (branch)  tailored to a given user query. Subsequently, the LLM is queried to evaluate a response based on each metric independently in parallel (solve) . Finally, the scores from each branch are averaged to arrive at a comprehensive evaluation decision (merge) . Notably, this method incurs an inference cost 5-6 times greater than that of a conventional (System 1)  LLM evaluation approach, making it much less practical. We assess the feasibility of distilling BSM, aiming to retain its benefits while reducing computational cost.Distillation DataFollowing, we used the Open Assistant Dataset v2 (OASST2) with turn 1 and English only data.
We use queries along with two candidate responses from the OASST2 training set as inputs (19,672 examples in total) .
We useself-consistency under input perturbationsto ensure the quality of our distillation data. Specifically, as two responses are being judged, we evaluate each sample twice with BSM - once in the original order and once in the swapped order. The winning response should remain consistent regardless of the order. We filter out samples that do not yield a consistent winner when the response order is swapped.EvaluationWe evaluate our models on two popular benchmarks, the OASST2 valid set and MT-bench. The OASST2 validation set comprises 273 samples, restricted to turn 1 and English language only. Evaluations of response pairs are performed in both original and swapped orders.
As we trained our distilled model on the OASST2 training set, the OASST2 validation set functions as an in-distribution evaluation set, while MT-bench is more out-of-distribution.
MT-bench is a popular benchmark that evaluates LLM-as-judges of other LLM’s responses when acting as helpful AI assistants conversations. It consists of instructions from 8 diverse domains e.g., writing, reasoning, math, coding, etc.Following, we assessed theAgreementbetween model votes and human expert votes. A well-documented limitation of LLM-as-a-judge is position bias, where a Language Model (LLM)  tends to favor certain positions over others. This bias is evident as altering the position of responses in the evaluation prompt often leads to different decisions by the model. To quantify this, we not only measure agreement but also calculate thePercentage of Inconsistentexamples to assess position bias.Figure 2:The agreement between LLM judges and human preferences per evaluated category on MT-bench.Modelk=1k=5k=10Acc %#TokensAcc %#TokensAcc %#TokensSystem 1Few (8) -shot (no CoT) 7.58%579.40%29510.31%620System 2CoT zero-shot52.77%27057.54%138559.44%2760CoT few (8) -shot36.39%29754.97%156063.84%3120Distill System 2Distill CoT zero-shot7.13%187.13%907.35%180Table 4:GSM8k test set accuracy. Number of votesin majority voting represents how many candidates were sampled to collect votes towards predicted answers. In this case System 2 Distillation of CoT does not work well.OASST2 Evaluation ResultsTable 3provides
results on the OASST2 dataset. Compared to baseline (System 1)  LLMs, the Chain-of-Thought (CoT)  method improves performance by improving agreement and reducing inconsistency rates (see prompts in Appendix) . While BSM outperforms CoT, this comes at the cost of increased inference time (#Tokens) . Remarkably, our distilled System 2 BSM model requires the generation of only four tokens and still outperforms both CoT and BSM. Furthermore, our distilled model based on Llama-2-70B-chat outperforms GPT-4-0125-preview, achieving
higher human agreement and greater consistency.MT-Bench Evaluation ResultsTable 3also provides results on MT-bench, which serves as an out-of-distribution test. The results mirror those from the OASST2 evaluation. Both Chain-of-Thought (CoT)  and BSM improve model performance but at the expense of significantly increased inference costs. Our distilled BSM model not only achieves higher human agreement and lower inconsistency rates but also requires less computational resources.
Although our model slightly underperforms in agreement compared to the state-of-the-art GPT-4-0125-preview model, it was trained solely on unlabeled data from OASST2 based on Llama-2-70B-chat. Despite this, it is more consistent and inference is cheap in terms of output tokens.Per Category AnalysisHere, we further analyze the MT-Bench results in terms ofAgreementby category. Figure2shows the per category agreement. We observe that CoT improved agreement compared to the base model (Llama-2-70B-Chat)  on all categories. BSM is better than CoT and our distilled BSM is even better than BSM.
Although Distilled BSM achieves superior performance compared to the baselines across all categories, it still lags behind GPT-4-0125-preview in reasoning, coding, and extraction. However, it surpasses GPT-4-0125-preview in writing, math, and STEM.4.5Chain-of-Thought DistillationChain-of-Thought (CoT) has been shown to be an effective method to improve LLM’s reasoning abilities, such as for solving graduate school math problems. The LLM generates intermediate tokens that are steps (chain)  of reasoning (thoughts)  before it produces the final answer.
We consider two variants of the approach: (i)  few-shot CoT, whereby multiple [question, CoT, answer] examples from the training set are provided as part of the context followed by the question; and (ii)  zero-shot, whereby an explicit instruction to think “step by step” is added to the prompt in addition to the question, see AppendixFigure 10.Distillation dataWe use CoT to produce answers for questions from the training split of GSM8k(which we consider unlabeled) ,
using majority voting with. The resulting distillation training set consists of[question, answer] pairs i.e., without any intermediate reasoning steps. The accuracy of the self-supervised targets, computed for analysis purposes, is.EvaluationWe report evaluation accuracy computed over the GSM8k test set with majority voting with different values of.
Similarly to our previous experiments, we report the average number of predicted tokens for each method. Note that we compute this average over all generated tokens when we run majority voting to see how the increase inaffects the inference cost.
We consider several baselines: System 1 and System 2 (CoT)  methods evaluated with zero-shot or 8-shot input contexts. Note that System 2 with 8-shot means that CoTs are provided in the few-shot inputs, while System 1 means that the few shot examples contain questions and answers, but no CoTs.ResultsEvaluation results are presented inTable 4. First, improvements are coming from using the CoT method as expected: it helps when being presented as part of the few-shot context or as part of the instruction in the prompt template. These improvements come with an increase in inference cost: sequences predicted with CoT methods are substantially longer compared to the System 1 method.
Second, our System 2 distillation method yields poor performance across various decoding hyper-parameters. The GSM8k task (math problems)  requires a very different kind of reasoning compared to other tasks we considered in this work. This highlights the non-trivial aspect of System 2 distillation: the proposed distillation algorithm works in many cases but not always. This leaves room for future research to elucidate in exactly which circumstances to apply distillation, and when not to, in a similar manner perhaps to the approach in humans.
body
Distilling System 2 into System 1
4Experiments
2407.06023
5ConclusionRecent work has shown that complex reasoning procedures using LLMs in the inner loop, called System 2 approaches, can improve performance. In this work we have shown that in many cases it is possible to distill this System 2 reasoning into the outputs of the LLMwithout intermediate generationswhile maintaining, or sometimes even improving, performance.
While not all methods can be distilled easily using our method, with Chain-of-Thought for complex reasoning being a challenging counterexample,
this is possible for diverse approaches. Our method works for System 2 Attention for dealing with bias and irrelevant context, Rephrase and Respond for clarifying task instructions, and Branch-Solve-Merge for improved LLM-as-a-Judge evaluation.
Pragmatically, distilling these approaches makes them more likely to be used by LLM practitioners, and they are more efficient at inference time. Looking forward, systems that can distill useful tasks in this way free up more time to spend on reasoning about the tasks that they cannot yet do well, just as humans do. Hence, we expect exploring this approach in a continuous training loop will be a fruitful research direction.
body
Distilling System 2 into System 1
5Conclusion
2407.06023
6LimitationsIn this paper, we explored three System 2 methods—RaR, S2A, and BSM—which have been successfully distilled, yielding enhanced results compared to the original System 1 performance while incurring lower inference costs than System 2. However, the effectiveness of these methods can vary depending on the specific task or the dataset used for model training. For instance, we observed that the CoT method could not be effectively distilled back to System 1 using our method.
We note that recent methods have tried alternative ways to distill
CoT.Moreover, due to the self-supervised nature of these methods, model performance relies on the specific filters applied. In our study, we depended on a consistency criterion that includesself-consistency of outputsandself-consistency under input perturbation. Although there are multiple alternative strategies to enhance data quality in self-supervised learning, these were not explored in our research.
conclusion
Distilling System 2 into System 1
6Limitations