<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2311.04205] Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves</title><meta property="og:description" content="Misunderstandings arise not only in interpersonal communication but also between humans and Large Language Models (LLMs). Such discrepancies can make LLMs interpret seemingly unambiguous questions in unexpected ways, y…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2311.04205">

<!--Generated on Tue Feb 27 21:06:46 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document" style="font-size:207%;">Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
Yihe Deng
  and  
Weitong Zhang
  and  
Zixiang Chen
  and  
Quanquan Gu

</span><span class="ltx_author_notes">Department of Computer Science, University of California, Los Angeles, CA 90095, USA; e-mail: <span id="id1.1.id1" class="ltx_text ltx_font_typewriter">yihedeng@cs.ucla.edu</span>Department of Computer Science, University of California, Los Angeles, CA 90095, USA; e-mail: <span id="id2.2.id1" class="ltx_text ltx_font_typewriter">weightzero@ucla.edu</span>Department of Computer Science, University of California, Los Angeles, CA 90095, USA; e-mail: <span id="id3.3.id1" class="ltx_text ltx_font_typewriter">chenzx19@cs.ucla.edu</span>Department of Computer Science, University of California, Los Angeles, CA 90095, USA; e-mail: <span id="id4.4.id1" class="ltx_text ltx_font_typewriter">qgu@cs.ucla.edu</span></span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id5.id1" class="ltx_p">Misunderstandings arise not only in interpersonal communication but also between humans and Large Language Models (LLMs). Such discrepancies can make LLMs interpret seemingly unambiguous questions in unexpected ways, yielding incorrect responses.
While it is widely acknowledged that the quality of a prompt, such as a question, significantly impacts the quality of the response provided by LLMs, a systematic method for crafting questions that LLMs can better comprehend is still underdeveloped.
In this paper, we present a method named ‘Rephrase and Respond’ (RaR), which allows LLMs to rephrase and expand questions posed by humans and provide responses in a single prompt. This approach serves as a simple yet effective prompting method for improving performance.
We also introduce a two-step variant of RaR, where a rephrasing LLM first rephrases the question and then passes the original and rephrased questions together to a different responding LLM. This facilitates the effective utilization of rephrased questions generated by one LLM with another. Our experiments demonstrate that our methods significantly improve the performance of different models across a wide range to tasks.
We further provide a comprehensive comparison between RaR and the popular Chain-of-Thought (CoT) methods, both theoretically and empirically. We show that RaR is complementary to CoT and can be combined with CoT to achieve even better performance. Our work not only contributes to enhancing LLM performance efficiently and effectively but also sheds light on a fair evaluation of LLM capabilities. Data and codes are available at <a target="_blank" href="https://github.com/uclaml/Rephrase-and-Respond" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/uclaml/Rephrase-and-Respond</a>.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Misunderstandings in interpersonal communications often arise when individuals, shaped by distinct subjective experiences, interpret the same message differently.
In social science, such phenomena can be attributed to cognitive biases in frames in thought <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Druckman</span>, <a href="#bib.bib7" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2001</span></a>)</cite>.
A frame represents an individual’s scheme of interpretation, enabling their understanding and response to an input <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Erving</span>, <a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">1974</span></a>)</cite>.
A single message, framed in different ways, can lead individuals to different conclusions.
People habitually project their frames onto their received information, and only shift these frames when incongruence arises.
Recently, Large Language Models (LLMs), such as the GPT series <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Radford et al.</span>, <a href="#bib.bib28" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2019</span></a>; <span class="ltx_text" style="font-size:70%;">Brown et al.</span>, <a href="#bib.bib5" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2020</span></a>; <span class="ltx_text" style="font-size:70%;">OpenAI</span>, <a href="#bib.bib22" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>, have witnessed a surge in popularity due to their profound impact on various real-world applications, including question answering <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Lu et al.</span>, <a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>, code generation <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Poesia et al.</span>, <a href="#bib.bib25" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>)</cite>, and conversational agents <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Bozkurt</span>, <a href="#bib.bib4" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>.
The wide applicability and efficacy of these models have led to rapidly growing research on understanding and improving the use of LLMs.
In this work, we posit that LLMs also exhibit their own frames in thought, and it is not uncommon to observe a disparity between the frames used by humans and LLMs.
It is widely acknowledged that the quality of the prompt generated by human critically influences the response quality of the LLMs, emphasizing the importance of effective queries that prioritize specificity, detail, and precision <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">OpenAI</span>, <a href="#bib.bib21" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>)</cite>.
However, because of an individual’s unique frame of thought, it can be challenging for humans to assess the clarity of their questions and to align their frames with those of LLMs.
To illustrate this, we first present a motivating example by investigating a recent work <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Allen-Zhu and Li</span>, <a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite> in detail.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2311.04205/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="174" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Motivating examples of our study. We can observe several ambiguities in the question for LLMs. An LLM can interpret “even month” as the month with even number of days, which diverges from human intention. We note that at each independent query, GPT-4 generates different outputs. Please refer to our experiments for more comprehensive results.</figcaption>
</figure>
<section id="S1.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Motivating Example.</h5>

<div id="S1.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S1.SS0.SSS0.Px1.p1.1" class="ltx_p">In <cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Allen-Zhu and Li</span> (<a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>, the authors reported an important finding: LLMs such as GPT-4 may not efficiently reason with their internal knowledge even if they can retrieve information accurately.
As shown in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, when posed with the query, “Was Mother Teresa born on an even month?” GPT-4 might mistakenly assert that August is an odd month.
Based on this observation, <cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Allen-Zhu and Li</span> (<a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite> suggested that GPT-4 instead requires a Chain-of-Thought process—relying on user-led follow-up questions—to correct its previous wrong answers. When posed with the follow-up question “Do you know what even means?”, GPT-4 will correct itself.
However, we take a step further to investigate the intrinsic reason for LLM’s inefficiency in answering such questions.
As shown in the other three conversations in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, when GPT-4 explains its reasoning, it appears that the model has several ambiguities toward the questions.
For example, it may consider February as odd due to its irregular number of days and sometimes consider an even/odd month to be months with an even/odd number of days.</p>
</div>
<div id="S1.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S1.SS0.SSS0.Px1.p2.1" class="ltx_p">Ambiguity in questions is a recognized concern in benchmark datasets.
For instance, it has been observed that the NLI datasets such as MultiNLI <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Williams et al.</span>, <a href="#bib.bib45" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2018</span></a>)</cite> contains ambiguities, which are challenging even for human interpreters <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Liu et al.</span>, <a href="#bib.bib16" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>.
Furthermore, our study uncovers that benchmark datasets commonly used for LLM evaluation <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Wei et al.</span>, <a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>; <span class="ltx_text" style="font-size:70%;">bench authors</span>, <a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite> possess ambiguities that are imperceptible to humans but challenging for language models.
These ambiguities cause LLMs to provide mistaken responses to unintended queries.
To address this issue, it is imperative to reduce ambiguity and contextualize information in a way that aligns with the existing frame of the LLMs.</p>
</div>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2311.04205/assets/x2.png" id="S1.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="131" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>
Comparison with the original question in <cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Allen-Zhu and Li</span> (<a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite> and one possible self-rephrased question using GPT-4.
The rephrased questions are longer and more detailed. Especially, self-rephrased questions resolve the ambiguity of the concept of ”even/odd” months by specifying ”even-numbered” and listing the examples.
We can observe a significant improvement in GPT-4’s accuracy in answering these rephrased questions over the query of <math id="S1.F2.2.m1.1" class="ltx_Math" alttext="105" display="inline"><semantics id="S1.F2.2.m1.1b"><mn id="S1.F2.2.m1.1.1" xref="S1.F2.2.m1.1.1.cmml">105</mn><annotation-xml encoding="MathML-Content" id="S1.F2.2.m1.1c"><cn type="integer" id="S1.F2.2.m1.1.1.cmml" xref="S1.F2.2.m1.1.1">105</cn></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.2.m1.1d">105</annotation></semantics></math> celebrities for this task.</figcaption>
</figure>
</section>
<section id="S1.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Our Approach.</h5>

<div id="S1.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S1.SS0.SSS0.Px2.p1.1" class="ltx_p">In this paper, we highlight an often-overlooked aspect of studies in LLMs: the disparity between human and LLM thought frames.
Our research illustrates that this disparity significantly impacts the performance of LLMs. To tackle this problem, we propose to let the LLM to rephrase the question and incorporate additional details for better answering.
We observe that, as opposed to questions asked casually by human, the rephrased questions tend to enhance <span id="S1.SS0.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_bold">semantic clarity</span> and aid in resolving <span id="S1.SS0.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_bold">inherent ambiguity</span>.
For example, the classification questions in <cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Allen-Zhu and Li</span> (<a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite> tend to be short.
Upon rephrasing by the LLM itself, the newly generated question is more detailed and has a clearer question format, as presented in Figure <a href="#S1.F2" title="Figure 2 ‣ Motivating Example. ‣ 1 Introduction ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
This self-rephrasing technique leads to significant improvement in accuracy compared with <cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Allen-Zhu and Li</span> (<a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>, as shown in the barplot of Figure <a href="#S1.F2" title="Figure 2 ‣ Motivating Example. ‣ 1 Introduction ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
While GPT-4 indeed found the original questions challenging, it demonstrates the ability to effectively answer the rephrased questions it generates.</p>
</div>
<div id="S1.SS0.SSS0.Px2.p2" class="ltx_para">
<p id="S1.SS0.SSS0.Px2.p2.1" class="ltx_p">Building upon these insights, we introduce a method named <span id="S1.SS0.SSS0.Px2.p2.1.1" class="ltx_text ltx_font_bold">Rephrase and Respond</span> (RaR), which prompts the LLM to rearticulate the given question and respond in a single prompt.
In addition to the simple RaR prompt, we also present a variation called Two-step RaR.
Two-step RaR employs a rephrasing LLM to generate reworded questions that can be made available to any responding LLM.
Our empirical results across diverse reasoning tasks show the effectiveness of both approaches.
Notably, Two-step RaR facilitates the transfer of rephrased questions from more capable LLMs to clarify ambiguities for less advanced models.
We also present both theoretical and empirical comparisons with the Chain-of-Thought (CoT) method <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Kojima et al.</span>, <a href="#bib.bib14" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>; <span class="ltx_text" style="font-size:70%;">Wei et al.</span>, <a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>)</cite>.
On the one hand, like CoT, RaR is compatible with the black-box nature of the current powerful GPT-3.5/4 that operate through API services.
On the other hand, while CoT focuses on augmentations either at the beginning or the end of a query, RaR directly modifies the query itself.
Therefore, RaR is complimentary to CoT and can be easily combined for improvement, as confirmed by our experimental results.
Furthermore, unlike methods that employ multiple LLMs for iterative prompt engineering based on accuracy scores <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Zhou et al.</span>, <a href="#bib.bib54" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022b</span></a>; <span class="ltx_text" style="font-size:70%;">Pryzant et al.</span>, <a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>, our method is both <span id="S1.SS0.SSS0.Px2.p2.1.2" class="ltx_text ltx_font_bold">unsupervised</span> and <span id="S1.SS0.SSS0.Px2.p2.1.3" class="ltx_text ltx_font_bold">training-free</span>, making it <span id="S1.SS0.SSS0.Px2.p2.1.4" class="ltx_text ltx_font_bold">economical and applicable to all questions</span>.
Lastly, our work call forth the importance that the design of human-crafted tasks targeting specific LLM capabilities should be rigorously reviewed by both humans and LLMs to ensure clarity in intention.</p>
</div>
<div id="S1.SS0.SSS0.Px2.p3" class="ltx_para">
<p id="S1.SS0.SSS0.Px2.p3.1" class="ltx_p">The remainder of this paper is organized as follows. Section <a href="#S2" title="2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> introduces the RaR method in detail, including One-step RaR (Section <a href="#S2.SS1" title="2.1 One-step RaR: Rephrase and Respond in a Single Prompt ‣ 2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>) and Two-step RaR (Section <a href="#S2.SS2" title="2.2 Two-step RaR: Rephrase the Question and Respond to the Rephrased Question ‣ 2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a>). In Section <a href="#S3" title="3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we present extensive empirical evaluations of the RaR method, including various benchmark tasks (Section <a href="#S3.SS1" title="3.1 Benchmark Tasks ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>), the performance on GPT-4 (Section <a href="#S3.SS2" title="3.2 Performance on GPT-4 ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>) and other GPT models (Section <a href="#S3.SS3" title="3.3 Performance across Various LLMs ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>). We also discuss the use of multiple rephrasing processes (Section <a href="#S3.SS4" title="3.4 Multiple Rephrasings: Will the Questions Converge? ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a>) in this section.
In Section <a href="#S4" title="4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we compare RaR with CoT in detail both theoretically and experimentally. The related works are discussed in Section <a href="#S5" title="5 Related Work ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> and the conclusion is drawn in Section <a href="#S6" title="6 Conclusion ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</p>
</div>
</section>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Rephrase and Respond</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">In this section, we introduce our proposed method in detail, encompassing two principled approaches, namely <span id="S2.p1.1.1" class="ltx_text ltx_font_bold">One-step RaR</span> and <span id="S2.p1.1.2" class="ltx_text ltx_font_bold">Two-step RaR</span> to facilitate better responses from LLMs by letting themselves rephrase the questions. In the following presentation, for the sake of simplicity, we will use <span id="S2.p1.1.3" class="ltx_text ltx_font_bold">RaR</span> to refer to <span id="S2.p1.1.4" class="ltx_text ltx_font_bold">One-step RaR</span>, unless a specific distinction is necessary.</p>
</div>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>One-step RaR: Rephrase and Respond in a Single Prompt</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">In interpersonal communication, rephrasing is a commonly known technique.
People rephrase another person’s question as a process of understanding, to ensure clarity and coherence in responding.
Such a communication strategy can be similarly applied to an LLM, letting it generate a rephrased question first and provide an answer subsequently.
Following this intuition, we propose <span id="S2.SS1.p1.1.1" class="ltx_text ltx_font_bold">RaR</span> to ask the LLMs to <span id="S2.SS1.p1.1.2" class="ltx_text ltx_font_bold">Rephrase and Response</span> the question using a single query. This approach can be viewed as a strategy to directly enhance the quality of the LLM’s response.
In detail, we introduce the following prompt for the question-answering task:</p>
<table id="A2.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.Ex1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S2.Ex1.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">"{question}"</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S2.E1.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">Rephrase and expand the question, and respond.</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2.1)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.p1.2" class="ltx_p">As we will show in experiments, GPT-4 can achieve much better results using RaR prompt (<a href="#S2.E1" title="Equation 2.1 ‣ 2.1 One-step RaR: Rephrase and Respond in a Single Prompt ‣ 2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>) across a wide range of tasks, and especially on human-crafted datasets that exhibit ambiguity to LLMs.</p>
</div>
<figure id="S2.F3" class="ltx_figure"><img src="/html/2311.04205/assets/x3.png" id="S2.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="207" height="111" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Demonstration of One-step RaR: one single prompt to ask the LLM to rephrase, expand and respond. </figcaption>
</figure>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Two-step RaR: Rephrase the Question and Respond to the Rephrased Question</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.2" class="ltx_p">To further leverage the quality improvement of the questions rephrased by larger models, like GPT-4, we introduce a variation of RaR called <span id="S2.SS2.p1.2.1" class="ltx_text ltx_font_bold">Two-step RaR</span>.
Intuitively, even among humans, a more detailed and precise question elicits in more accurate and decisive responses. Two-step RaR follows this intuition by designing a two-step procedure to improve the quality of the questions: in the first step, given a query <span id="S2.SS2.p1.2.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">question</span>, we generate a self-rephrased query <span id="S2.SS2.p1.2.3" class="ltx_text ltx_markedasmath ltx_font_typewriter">rephrased_question</span> by prompting a <em id="S2.SS2.p1.2.4" class="ltx_emph ltx_font_italic">rephrasing LLM</em> with the following prompt:</p>
<table id="A2.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.Ex2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S2.Ex2.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">"{question}"</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.Ex3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S2.Ex3.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">Given the above question, rephrase and expand it to help you</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S2.E2.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">do better answering. Maintain all information in the original question.</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2.2)</span></td>
</tr></tbody>
</table>
<p id="S2.SS2.p1.3" class="ltx_p">Then the original question and the rephrased question are combined to prompt a <em id="S2.SS2.p1.3.1" class="ltx_emph ltx_font_italic">responding LLM</em> with the following prompt:</p>
<table id="A2.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.Ex4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S2.Ex4.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">(original) {question}</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.Ex5"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S2.Ex5.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">(rephrased) {rephrased_question}</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S2.E3.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">Use your answer for the rephrased question to answer the original question.</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2.3)</span></td>
</tr></tbody>
</table>
<p id="S2.SS2.p1.4" class="ltx_p">Notably, the rephrasing LLM and the responding LLM can be either the same or different models. As we will show later in experiments, different LLMs exhibit distinct proficiency in question rephrasing. In particular, a question rephrased by GPT-4 can help a weaker LLM like Vicuna to produce more accurate responses.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.1" class="ltx_p">The enhancement in the response quality can be leveraged to improve the benchmark datasets for a fairer evaluation of LLMs: Existing benchmark datasets, crafted by humans, are designed to assess the performance of LLMs across various reasoning skills. However, as demonstrated in our examples in Figure <a href="#S1.F2" title="Figure 2 ‣ Motivating Example. ‣ 1 Introduction ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, these questions may lack the necessary clarity to fully showcase the specific abilities of LLMs. By the ‘Rephrase’ step in Two-step RaR, we can universally improve the question quality and enable a fairer comparison.</p>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.1" class="ltx_p">In addition, compared with the prompt of One-step RaR, in the Two-step version, we maintain the original context by including the user’s question, while adding the LLM-rephrased question to help better understanding. This prevents the possible divergence of LLMs from the original questions.</p>
</div>
<figure id="S2.F4" class="ltx_figure"><img src="/html/2311.04205/assets/x4.png" id="S2.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="405" height="110" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Demonstration of Two-Step RaR: it involves first rephrasing the question and then using the original and rephrased question to improve the response quality.</figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>RaR Effectively Improves LLM Responses</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.4" class="ltx_p">In this section, we provide a comprehensive assessment of the applicability and efficacy of RaR. The results are presented in four primary dimensions: <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="(1)" display="inline"><semantics id="S3.p1.1.m1.1a"><mrow id="S3.p1.1.m1.1.2.2"><mo stretchy="false" id="S3.p1.1.m1.1.2.2.1">(</mo><mn id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">1</mn><mo stretchy="false" id="S3.p1.1.m1.1.2.2.2">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><cn type="integer" id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">(1)</annotation></semantics></math> One-step RaR is a simple and effective prompt to improve LLM performances; <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="(2)" display="inline"><semantics id="S3.p1.2.m2.1a"><mrow id="S3.p1.2.m2.1.2.2"><mo stretchy="false" id="S3.p1.2.m2.1.2.2.1">(</mo><mn id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml">2</mn><mo stretchy="false" id="S3.p1.2.m2.1.2.2.2">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><cn type="integer" id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">(2)</annotation></semantics></math> Two-step RaR effectively enhances the response accuracy of GPT-4 across diverse tasks; <math id="S3.p1.3.m3.1" class="ltx_Math" alttext="(3)" display="inline"><semantics id="S3.p1.3.m3.1a"><mrow id="S3.p1.3.m3.1.2.2"><mo stretchy="false" id="S3.p1.3.m3.1.2.2.1">(</mo><mn id="S3.p1.3.m3.1.1" xref="S3.p1.3.m3.1.1.cmml">3</mn><mo stretchy="false" id="S3.p1.3.m3.1.2.2.2">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.3.m3.1b"><cn type="integer" id="S3.p1.3.m3.1.1.cmml" xref="S3.p1.3.m3.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m3.1c">(3)</annotation></semantics></math> LLMs, while all benefit from Two-Step RaR, have different proficiency in rephrasing questions; <math id="S3.p1.4.m4.1" class="ltx_Math" alttext="(4)" display="inline"><semantics id="S3.p1.4.m4.1a"><mrow id="S3.p1.4.m4.1.2.2"><mo stretchy="false" id="S3.p1.4.m4.1.2.2.1">(</mo><mn id="S3.p1.4.m4.1.1" xref="S3.p1.4.m4.1.1.cmml">4</mn><mo stretchy="false" id="S3.p1.4.m4.1.2.2.2">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.4.m4.1b"><cn type="integer" id="S3.p1.4.m4.1.1.cmml" xref="S3.p1.4.m4.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.4.m4.1c">(4)</annotation></semantics></math> a weak LLM can benefit more from a question rephrased by a strong LLM.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Benchmark Tasks</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">We first introduce the benchmark tasks we use to evaluate our method.</p>
</div>
<section id="S3.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Tasks.</h5>

<div id="S3.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px1.p1.1" class="ltx_p">We evaluate the capabilities of LLMs across multiple benchmark tasks in different categories.</p>
<ul id="S3.I1" class="ltx_itemize">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p"><span id="S3.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Knowledge Classification</span> <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Allen-Zhu and Li</span>, <a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>. Sampling a pool of individuals with Wikipedia pages, this task challenges the LLM to decide if a renowned person was born on an even day, month, or year.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.1" class="ltx_p"><span id="S3.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Knowledge Comparison</span> <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Allen-Zhu and Li</span>, <a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>. Using the same pool of individuals, this task instructs the LLM to compare the ages of two people and decide who was born earlier.</p>
</div>
</li>
</ul>
<p id="S3.SS1.SSS0.Px1.p1.2" class="ltx_p">As GPT-4 responds poorly to many of these questions<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>As the data are not open-sourced, we let GPT-4 generate <math id="footnote1.m1.1" class="ltx_Math" alttext="105" display="inline"><semantics id="footnote1.m1.1b"><mn id="footnote1.m1.1.1" xref="footnote1.m1.1.1.cmml">105</mn><annotation-xml encoding="MathML-Content" id="footnote1.m1.1c"><cn type="integer" id="footnote1.m1.1.1.cmml" xref="footnote1.m1.1.1">105</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote1.m1.1d">105</annotation></semantics></math> famous individuals with their birth dates and <math id="footnote1.m2.1" class="ltx_Math" alttext="105" display="inline"><semantics id="footnote1.m2.1b"><mn id="footnote1.m2.1.1" xref="footnote1.m2.1.1.cmml">105</mn><annotation-xml encoding="MathML-Content" id="footnote1.m2.1c"><cn type="integer" id="footnote1.m2.1.1.cmml" xref="footnote1.m2.1.1">105</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote1.m2.1d">105</annotation></semantics></math> Chinese idioms in its knowledge.</span></span></span>, it raises a concern that whether GPT-4, despite its proficiency in retrieving knowledge, falls behind in reasoning with its own knowledge. Furthermore, we consider the following widely-used datasets for a comprehensive evaluation, which are also considered in <cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Wei et al.</span> (<a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>)</cite>.</p>
<ul id="S3.I2" class="ltx_itemize">
<li id="S3.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i1.p1" class="ltx_para">
<p id="S3.I2.i1.p1.1" class="ltx_p"><span id="S3.I2.i1.p1.1.1" class="ltx_text ltx_font_bold">CSQA</span> <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Talmor et al.</span>, <a href="#bib.bib37" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2019</span></a>)</cite>. The CommonSense QA data encompasses a range of questions that evaluate the ability of commonsense understanding of the world and involves intricate semantics.</p>
</div>
</li>
<li id="S3.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i2.p1" class="ltx_para">
<p id="S3.I2.i2.p1.1" class="ltx_p"><span id="S3.I2.i2.p1.1.1" class="ltx_text ltx_font_bold">Date Understanding</span> <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">bench authors</span>, <a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>. Sourced from Big-bench <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">bench authors</span>, <a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>, the Date Understanding task emphasizes commonsense reasoning and deducing a date from a provided context. The task is also considered in <cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Wei et al.</span> (<a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>)</cite>. We consider a more difficult version where we do not provide the choices of potential answers and let the LLM answer directly.</p>
</div>
</li>
<li id="S3.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i3.p1" class="ltx_para">
<p id="S3.I2.i3.p1.1" class="ltx_p"><span id="S3.I2.i3.p1.1.1" class="ltx_text ltx_font_bold">Last Letter Concatenation</span> <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Fortes</span>, <a href="#bib.bib9" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>. The task centers on symbolic reasoning, and asks the LLM to concatenate the final letters of a given list of names. We consider concatenation for two names as well as a more difficult task of concatenation for four names.</p>
</div>
</li>
<li id="S3.I2.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i4.p1" class="ltx_para">
<p id="S3.I2.i4.p1.1" class="ltx_p"><span id="S3.I2.i4.p1.1.1" class="ltx_text ltx_font_bold">Coin Flip<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note"><span id="footnote2.1.1.1" class="ltx_text ltx_font_medium">2</span></span><a target="_blank" href="https://huggingface.co/datasets/skrishna/coin_flip" title="" class="ltx_ref ltx_url ltx_font_typewriter ltx_font_medium">https://huggingface.co/datasets/skrishna/coin_flip</a></span></span></span></span>. Sourced from Hugging Face, the task asks the LLM if the coin still heads up, given its initial condition and subsequent actions of people who either flipped or did not flip the coin. We add an additional “Flip means reverse.” to the questions.</p>
</div>
</li>
<li id="S3.I2.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i5.p1" class="ltx_para">
<p id="S3.I2.i5.p1.1" class="ltx_p"><span id="S3.I2.i5.p1.1.1" class="ltx_text ltx_font_bold">Sports</span> <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">bench authors</span>, <a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>. Sourced from Big-bench <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">bench authors</span>, <a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>, the Sports Understanding task primarily asks if a sentence is plausible or implausible, where a prominent sports figure is depicted performing specific sports-related actions.</p>
</div>
</li>
</ul>
</div>
<div id="S3.SS1.SSS0.Px1.p2" class="ltx_para">
<p id="S3.SS1.SSS0.Px1.p2.1" class="ltx_p">The details of all evaluated tasks are summarized in Table <a href="#S3.T1" title="Table 1 ‣ Tasks. ‣ 3.1 Benchmark Tasks ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Details of all the tasks considered in our experiments, including dataset name, category, number of questions, and an example question.</figcaption>
<div id="S3.T1.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:215.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-109.3pt,54.3pt) scale(0.664805374191935,0.664805374191935) ;">
<table id="S3.T1.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S3.T1.1.1.1" class="ltx_tr">
<td id="S3.T1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">Dataset</td>
<td id="S3.T1.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt">Category</td>
<td id="S3.T1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">Number</td>
<td id="S3.T1.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">Question example</td>
</tr>
<tr id="S3.T1.1.1.2" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S3.T1.1.1.2.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T1.1.1.2.1.1" class="ltx_text" style="background-color:#E0FFFF;">Even day</span></td>
<td id="S3.T1.1.1.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.1.2.2.1" class="ltx_text" style="background-color:#E0FFFF;">Knowledge Classification</span></td>
<td id="S3.T1.1.1.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.1.2.3.1" class="ltx_text" style="background-color:#E0FFFF;">105</span></td>
<td id="S3.T1.1.1.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.1.2.4.1" class="ltx_text" style="background-color:#E0FFFF;">Was Donald Trump born in an even day?</span></td>
</tr>
<tr id="S3.T1.1.1.3" class="ltx_tr">
<td id="S3.T1.1.1.3.1" class="ltx_td ltx_align_center ltx_border_r">Even month</td>
<td id="S3.T1.1.1.3.2" class="ltx_td ltx_align_center">Knowledge Classification</td>
<td id="S3.T1.1.1.3.3" class="ltx_td ltx_align_center">105</td>
<td id="S3.T1.1.1.3.4" class="ltx_td ltx_align_center">Was Joe Biden born in an even month?</td>
</tr>
<tr id="S3.T1.1.1.4" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S3.T1.1.1.4.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.1.4.1.1" class="ltx_text" style="background-color:#E0FFFF;">Even year</span></td>
<td id="S3.T1.1.1.4.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.4.2.1" class="ltx_text" style="background-color:#E0FFFF;">Knowledge Classification</span></td>
<td id="S3.T1.1.1.4.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.4.3.1" class="ltx_text" style="background-color:#E0FFFF;">105</span></td>
<td id="S3.T1.1.1.4.4" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.4.4.1" class="ltx_text" style="background-color:#E0FFFF;">Was Elon Musk born in an even year?</span></td>
</tr>
<tr id="S3.T1.1.1.5" class="ltx_tr">
<td id="S3.T1.1.1.5.1" class="ltx_td ltx_align_center ltx_border_r">Compare age</td>
<td id="S3.T1.1.1.5.2" class="ltx_td ltx_align_center">Knowledge Comparison</td>
<td id="S3.T1.1.1.5.3" class="ltx_td ltx_align_center">104</td>
<td id="S3.T1.1.1.5.4" class="ltx_td ltx_align_center">Was Joe Biden born earlier than Donald Trump?</td>
</tr>
<tr id="S3.T1.1.1.6" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S3.T1.1.1.6.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.1.6.1.1" class="ltx_text" style="background-color:#E0FFFF;">Chinese idiom</span></td>
<td id="S3.T1.1.1.6.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.6.2.1" class="ltx_text" style="background-color:#E0FFFF;">Inverse knowledge search</span></td>
<td id="S3.T1.1.1.6.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.6.3.1" class="ltx_text" style="background-color:#E0FFFF;">105</span></td>
<td id="S3.T1.1.1.6.4" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.6.4.1" class="ltx_text" style="background-color:#E0FFFF;">成语”X事求是”的X是什么字？</span></td>
</tr>
<tr id="S3.T1.1.1.7" class="ltx_tr">
<td id="S3.T1.1.1.7.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="2"><span id="S3.T1.1.1.7.1.1" class="ltx_text">CSQA</span></td>
<td id="S3.T1.1.1.7.2" class="ltx_td ltx_align_center" rowspan="2"><span id="S3.T1.1.1.7.2.1" class="ltx_text">Commonsense Reasoning</span></td>
<td id="S3.T1.1.1.7.3" class="ltx_td ltx_align_center" rowspan="2"><span id="S3.T1.1.1.7.3.1" class="ltx_text">220</span></td>
<td id="S3.T1.1.1.7.4" class="ltx_td ltx_align_center">Everybody has some sort of creativity, it’s the best way to what?</td>
</tr>
<tr id="S3.T1.1.1.8" class="ltx_tr">
<td id="S3.T1.1.1.8.1" class="ltx_td ltx_align_center">Choices: A. Create art B. Write C. Sew D. Create idea E. Express yourself</td>
</tr>
<tr id="S3.T1.1.1.9" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S3.T1.1.1.9.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.1.9.1.1" class="ltx_text" style="background-color:#E0FFFF;">Dates</span></td>
<td id="S3.T1.1.1.9.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.9.2.1" class="ltx_text" style="background-color:#E0FFFF;">Commonsense Reasoning</span></td>
<td id="S3.T1.1.1.9.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.9.3.1" class="ltx_text" style="background-color:#E0FFFF;">369</span></td>
<td id="S3.T1.1.1.9.4" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.9.4.1" class="ltx_text" style="background-color:#E0FFFF;">This is the last day of 1899. What is the date tomorrow in MM/DD/YYYY?</span></td>
</tr>
<tr id="S3.T1.1.1.10" class="ltx_tr">
<td id="S3.T1.1.1.10.1" class="ltx_td ltx_align_center ltx_border_r">Last letter (2)</td>
<td id="S3.T1.1.1.10.2" class="ltx_td ltx_align_center">Symbolic Reasoning</td>
<td id="S3.T1.1.1.10.3" class="ltx_td ltx_align_center">220</td>
<td id="S3.T1.1.1.10.4" class="ltx_td ltx_align_center">Take the last letters of the words in “Kristen Andrea” and concatenate them.</td>
</tr>
<tr id="S3.T1.1.1.11" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S3.T1.1.1.11.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.1.11.1.1" class="ltx_text" style="background-color:#E0FFFF;">Last letter (4)</span></td>
<td id="S3.T1.1.1.11.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.11.2.1" class="ltx_text" style="background-color:#E0FFFF;">Symbolic Reasoning</span></td>
<td id="S3.T1.1.1.11.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.11.3.1" class="ltx_text" style="background-color:#E0FFFF;">220</span></td>
<td id="S3.T1.1.1.11.4" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.11.4.1" class="ltx_text" style="background-color:#E0FFFF;">Take the last letters of the words in “Mia Mariana Marcos Xavier” and concatenate them.</span></td>
</tr>
<tr id="S3.T1.1.1.12" class="ltx_tr">
<td id="S3.T1.1.1.12.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="2"><span id="S3.T1.1.1.12.1.1" class="ltx_text">Coin flip</span></td>
<td id="S3.T1.1.1.12.2" class="ltx_td ltx_align_center" rowspan="2"><span id="S3.T1.1.1.12.2.1" class="ltx_text">Symbolic Reasoning</span></td>
<td id="S3.T1.1.1.12.3" class="ltx_td ltx_align_center" rowspan="2"><span id="S3.T1.1.1.12.3.1" class="ltx_text">220</span></td>
<td id="S3.T1.1.1.12.4" class="ltx_td ltx_align_center">A coin is heads up. roxas does not flip the coin.</td>
</tr>
<tr id="S3.T1.1.1.13" class="ltx_tr">
<td id="S3.T1.1.1.13.1" class="ltx_td ltx_align_center">scheideman does not flip the coin. Is the coin still heads up? Flip means reverse.</td>
</tr>
<tr id="S3.T1.1.1.14" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S3.T1.1.1.14.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.1.14.1.1" class="ltx_text" style="background-color:#E0FFFF;">Sports</span></td>
<td id="S3.T1.1.1.14.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.14.2.1" class="ltx_text" style="background-color:#E0FFFF;">Commonsense Reasoning</span></td>
<td id="S3.T1.1.1.14.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.14.3.1" class="ltx_text" style="background-color:#E0FFFF;">220</span></td>
<td id="S3.T1.1.1.14.4" class="ltx_td ltx_align_center"><span id="S3.T1.1.1.14.4.1" class="ltx_text" style="background-color:#E0FFFF;">Is the following sentence plausible? “Juan Soto shot from beyond the arc”</span></td>
</tr>
<tr id="S3.T1.1.1.15" class="ltx_tr">
<td id="S3.T1.1.1.15.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" rowspan="4"><span id="S3.T1.1.1.15.1.1" class="ltx_text">StereoSet</span></td>
<td id="S3.T1.1.1.15.2" class="ltx_td ltx_align_center ltx_border_bb" rowspan="4"><span id="S3.T1.1.1.15.2.1" class="ltx_text">Stereotypical Bias</span></td>
<td id="S3.T1.1.1.15.3" class="ltx_td ltx_align_center ltx_border_bb" rowspan="4"><span id="S3.T1.1.1.15.3.1" class="ltx_text">220</span></td>
<td id="S3.T1.1.1.15.4" class="ltx_td ltx_align_center">Morocco is a country in Africa. Which of these is more accurate?</td>
</tr>
<tr id="S3.T1.1.1.16" class="ltx_tr">
<td id="S3.T1.1.1.16.1" class="ltx_td ltx_align_center">A. It’s a very poor country with low standards of living.</td>
</tr>
<tr id="S3.T1.1.1.17" class="ltx_tr">
<td id="S3.T1.1.1.17.1" class="ltx_td ltx_align_center">B. There are many cultures and classes of people that live there.</td>
</tr>
<tr id="S3.T1.1.1.18" class="ltx_tr">
<td id="S3.T1.1.1.18.1" class="ltx_td ltx_align_center ltx_border_bb">C. Blue is her favorite color.</td>
</tr>
</table>
</span></div>
</figure>
<div id="S3.SS1.SSS0.Px1.p3" class="ltx_para">
<p id="S3.SS1.SSS0.Px1.p3.1" class="ltx_p">We use the entire dataset for <span id="S3.SS1.SSS0.Px1.p3.1.1" class="ltx_text ltx_font_bold">Dates Understanding</span>, and randomly draw subsets of size <math id="S3.SS1.SSS0.Px1.p3.1.m1.1" class="ltx_Math" alttext="220" display="inline"><semantics id="S3.SS1.SSS0.Px1.p3.1.m1.1a"><mn id="S3.SS1.SSS0.Px1.p3.1.m1.1.1" xref="S3.SS1.SSS0.Px1.p3.1.m1.1.1.cmml">220</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px1.p3.1.m1.1b"><cn type="integer" id="S3.SS1.SSS0.Px1.p3.1.m1.1.1.cmml" xref="S3.SS1.SSS0.Px1.p3.1.m1.1.1">220</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px1.p3.1.m1.1c">220</annotation></semantics></math> for the rest tasks.
We use accuracy to evaluate the performance of the LLM. The accuracy is firstly estimated using exact matching on the words generated by the LLM. Specifically, an answer is considered correct if it contains the exact word of the correct response and without any incorrect responses.
We subsequently verify and correct the calculations through manual inspection.
For certain tasks, to constrain the response format (e.g., multiple-choice), we append a consistent prompt when evaluating the original question and RaR, such as “Select the single most appropriate answer”. Details of the prompts are presented in Table <a href="#A1.T8" title="Table 8 ‣ Appendix A Experiment Details ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> in Appendix <a href="#A1" title="Appendix A Experiment Details ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.</p>
</div>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Performance on GPT-4</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">We conduct experiments on the aforementioned benchmark datasets using GPT-4<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>We note that all our experiments accessed GPT-4 during 10/01-10/30. We also include results on <span id="footnote3.1" class="ltx_text ltx_font_typewriter">GPT-4-0613</span>.</span></span></span> <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">OpenAI</span>, <a href="#bib.bib22" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>, both with One-step RaR and Two-step RaR. As shown in Figure <a href="#S3.F5" title="Figure 5 ‣ 3.2 Performance on GPT-4 ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, both One-step RaR and Two-step RaR enjoy superior performance compared with using original questions. We will discuss the findings in the sequel.</p>
</div>
<figure id="S3.F5" class="ltx_figure"><img src="/html/2311.04205/assets/x5.png" id="S3.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="230" height="185" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Accuracy (%) comparison of different prompts using GPT-4. Both One-step RaR and Two-step RaR effectively improve the accuracy of GPT-4 across <math id="S3.F5.2.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.F5.2.m1.1b"><mn id="S3.F5.2.m1.1.1" xref="S3.F5.2.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.F5.2.m1.1c"><cn type="integer" id="S3.F5.2.m1.1.1.cmml" xref="S3.F5.2.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.F5.2.m1.1d">10</annotation></semantics></math> tasks. A larger margin indicates a less well-designed question. For tasks such as CSQA where the question quality is good enough, our method still offers a marginal improvement.</figcaption>
</figure>
<section id="S3.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.1 </span>RaR: A Simple Prompt to Improve LLM Performance</h4>

<div id="S3.SS2.SSS1.p1" class="ltx_para">
<p id="S3.SS2.SSS1.p1.1" class="ltx_p">We investigate the performance of RaR, which allows the LLM to both rephrase and respond to the question in a single query.
Such an approach can be considered as a simple black-box strategy to improve the LLM’s performance on any question.</p>
</div>
<div id="S3.SS2.SSS1.p2" class="ltx_para">
<p id="S3.SS2.SSS1.p2.2" class="ltx_p">In Figure <a href="#S3.F5" title="Figure 5 ‣ 3.2 Performance on GPT-4 ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> and detailed in Table <a href="#A1.T7" title="Table 7 ‣ Appendix A Experiment Details ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> in Appendix <a href="#A1" title="Appendix A Experiment Details ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>, we compare the accuracy of GPT-4 with One-step RaR (i.e., rephrasing and answering a question in a single prompt) and Two-step RaR (answering a pre-rephrased question in separate queries). Notably, One-step RaR improves GPT-4’s accuracy, and outperforms Two-step RaR on <math id="S3.SS2.SSS1.p2.1.m1.1" class="ltx_Math" alttext="6" display="inline"><semantics id="S3.SS2.SSS1.p2.1.m1.1a"><mn id="S3.SS2.SSS1.p2.1.m1.1.1" xref="S3.SS2.SSS1.p2.1.m1.1.1.cmml">6</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.1.m1.1b"><cn type="integer" id="S3.SS2.SSS1.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1">6</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.1.m1.1c">6</annotation></semantics></math> out of <math id="S3.SS2.SSS1.p2.2.m2.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.SS2.SSS1.p2.2.m2.1a"><mn id="S3.SS2.SSS1.p2.2.m2.1.1" xref="S3.SS2.SSS1.p2.2.m2.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.2.m2.1b"><cn type="integer" id="S3.SS2.SSS1.p2.2.m2.1.1.cmml" xref="S3.SS2.SSS1.p2.2.m2.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.2.m2.1c">10</annotation></semantics></math> tasks. Indeed, similar to human communication, rephrasing and elaborating a question and then answering is an effective approach. The key takeaway from this experiment in highlighted below.</p>
</div>
<div id="S3.SS2.SSS1.p3" class="ltx_para ltx_noindent">
<svg id="S3.SS2.SSS1.p3.pic1" class="ltx_picture" height="78.6" overflow="visible" version="1.1" width="600"><g transform="translate(0,78.6) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B3D7E8" fill-opacity="1.0"><path d="M 0 5.91 L 0 72.7 C 0 75.96 2.64 78.6 5.91 78.6 L 594.09 78.6 C 597.36 78.6 600 75.96 600 72.7 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#FAFCFD" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 54.49 L 598.03 54.49 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 60.4)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S3.SS2.SSS1.p3.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S3.SS2.SSS1.p3.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S3.SS2.SSS1.p3.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Takeaway #1</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="28.9" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S3.SS2.SSS1.p3.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S3.SS2.SSS1.p3.pic1.2.2.2.1.1.1" class="ltx_p">(One-step) RaR provides a universal, plug-and-play black-box prompt that allows for efficient and effective performance improvement of LLMs on general tasks.</span>
</span></foreignObject></g></g></svg>
</div>
</section>
<section id="S3.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.2 </span>Two-step RaR: Rephrased Questions Improve Response Quality</h4>

<div id="S3.SS2.SSS2.p1" class="ltx_para">
<p id="S3.SS2.SSS2.p1.1" class="ltx_p">We evaluate the quality improvement of the question using Two-step RaR. In detail, for each query, GPT-4 autonomously generates a rephrased question using the prompt (<a href="#S2.E2" title="Equation 2.2 ‣ 2.2 Two-step RaR: Rephrase the Question and Respond to the Rephrased Question ‣ 2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a>) without any external intervention. The rephrased question is then combined with the original question using (<a href="#S2.E3" title="Equation 2.3 ‣ 2.2 Two-step RaR: Rephrase the Question and Respond to the Rephrased Question ‣ 2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a>) to prompt GPT-4.
We present the accuracy of GPT-4 using Two-step RaR and compare it with GPT-4 using the original questions as shown in Figure <a href="#S3.F5" title="Figure 5 ‣ 3.2 Performance on GPT-4 ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.
Across a diverse span of tasks that emphasize different aspects of LLM’s capabilities, Two-step RaR consistently yields distinguishable improvements for GPT-4.
Notably, for tasks that GPT-4 originally finds highly challenging (e.g., last letter concatenation), the Two-step RaR exhibits remarkable improvement even to almost <math id="S3.SS2.SSS2.p1.1.m1.1" class="ltx_Math" alttext="100\%" display="inline"><semantics id="S3.SS2.SSS2.p1.1.m1.1a"><mrow id="S3.SS2.SSS2.p1.1.m1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.cmml"><mn id="S3.SS2.SSS2.p1.1.m1.1.1.2" xref="S3.SS2.SSS2.p1.1.m1.1.1.2.cmml">100</mn><mo id="S3.SS2.SSS2.p1.1.m1.1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.1.m1.1b"><apply id="S3.SS2.SSS2.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1"><csymbol cd="latexml" id="S3.SS2.SSS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S3.SS2.SSS2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.2">100</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.1.m1.1c">100\%</annotation></semantics></math> accuracy. The numerical details of the accuracy are also presented in Table <a href="#A1.T7" title="Table 7 ‣ Appendix A Experiment Details ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> in Appendix <a href="#A1" title="Appendix A Experiment Details ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.
We conclude this experiment by the following takeaway.</p>
</div>
<div id="S3.SS2.SSS2.p2" class="ltx_para ltx_noindent">
<svg id="S3.SS2.SSS2.p2.pic1" class="ltx_picture" height="96.59" overflow="visible" version="1.1" width="600"><g transform="translate(0,96.59) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B3D7E8" fill-opacity="1.0"><path d="M 0 5.91 L 0 90.69 C 0 93.95 2.64 96.59 5.91 96.59 L 594.09 96.59 C 597.36 96.59 600 93.95 600 90.69 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#FAFCFD" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 72.48 L 598.03 72.48 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 78.39)"><foreignObject width="556.69" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S3.SS2.SSS2.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S3.SS2.SSS2.p2.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S3.SS2.SSS2.p2.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Takeaway #2</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="46.89" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S3.SS2.SSS2.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S3.I3" class="ltx_itemize">
<span id="S3.I3.i1" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S3.I3.i1.p1" class="ltx_para">
<span id="S3.I3.i1.p1.1" class="ltx_p">Examining the question quality is pivotal when evaluating the LLM performance on QA tasks.</span>
</span></span>
<span id="S3.I3.i2" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S3.I3.i2.p1" class="ltx_para">
<span id="S3.I3.i2.p1.1" class="ltx_p">Two-step RaR provides a universal method for LLMs to improve the question quality autonomously by rephrasing the question.</span>
</span></span>
</span>
</span></foreignObject></g></g></svg>
</div>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Performance across Various LLMs</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">We further examine the performance of RaR on various LLMs, including GPT-3.5 and Vicuna <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Chiang et al.</span>, <a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>)</cite>size=<span id="S3.SS3.p1.1.1" class="ltx_text" style="font-size:70%;">,color=orange!20!white,]Quanquan: references. In particular, we employ Two-step RaR to investigate (1) if all these LLMs can provide consistent response improvement by rephrasing the questions; and (2) if the GPT-4-rephrased questions can improve the performance of other LLMs.</span></p>
</div>
<section id="S3.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsubsection">3.3.1 </span>Can All LLMs Rephrase Questions?</h4>

<figure id="S3.F6" class="ltx_figure"><img src="/html/2311.04205/assets/x6.png" id="S3.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="137" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Accuracy (%) of <span id="S3.F6.7.1" class="ltx_text ltx_font_typewriter">GPT-4-0613</span>, <span id="S3.F6.8.2" class="ltx_text ltx_font_typewriter">GPT-3.5-turbo-0613</span> and Vicuna-13b when testing on original and self-rephrased questions using Two-step RaR. GPT-4 enjoys larger improvement on various tasks compared with GPT -3.5 and Vicuna.</figcaption>
</figure>
<div id="S3.SS3.SSS1.p1" class="ltx_para">
<p id="S3.SS3.SSS1.p1.4" class="ltx_p"><span id="S3.SS3.SSS1.p1.4.1" class="ltx_text" style="font-size:70%;">We investigate the rephrasing abilities of different LLMs by employing Two-step RaR to examine the quality of the rephrased questions. We evaluate the performance of several different LLMs, including </span><span id="S3.SS3.SSS1.p1.4.2" class="ltx_text ltx_font_typewriter" style="font-size:70%;">GPT-4-0613</span><span id="S3.SS3.SSS1.p1.4.3" class="ltx_text" style="font-size:70%;">, </span><span id="S3.SS3.SSS1.p1.4.4" class="ltx_text ltx_font_typewriter" style="font-size:70%;">GPT-3.5-turbo-0613</span><span id="S3.SS3.SSS1.p1.4.5" class="ltx_text" style="font-size:70%;">, and </span><span id="S3.SS3.SSS1.p1.4.6" class="ltx_text ltx_font_typewriter" style="font-size:70%;">Vicuna-13b-v1.5</span><span id="S3.SS3.SSS1.p1.4.7" class="ltx_text" style="font-size:70%;">, using Two-step RaR.
We present the experiment results in Figure </span><a href="#S3.F6" title="Figure 6 ‣ 3.3.1 Can All LLMs Rephrase Questions? ‣ 3.3 Performance across Various LLMs ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">6</span></a><span id="S3.SS3.SSS1.p1.4.8" class="ltx_text" style="font-size:70%;">.
Due to </span><span id="S3.SS3.SSS1.p1.4.9" class="ltx_text ltx_font_typewriter" style="font-size:70%;">Vicuna-13b-v1.5</span><span id="S3.SS3.SSS1.p1.4.10" class="ltx_text" style="font-size:70%;">’s near-zero performance on Last Letter Concatenation (4), we exclude this task from the evaluation of </span><span id="S3.SS3.SSS1.p1.4.11" class="ltx_text ltx_font_typewriter" style="font-size:70%;">Vicuna-13b-v1.5</span><span id="S3.SS3.SSS1.p1.4.12" class="ltx_text" style="font-size:70%;">.
Remarkably, all examined LLMs demonstrate enhanced performance with Two-step RaR, resulting in a notable increase in accuracy across the majority of the tasks.
More advanced models, such as GPT-4, benefit from the most significant gains across all tasks, while models of lesser complexity, like Vicuna, achieve modest improvements using our approach.
On certain tasks such as CSQA and Sports, GPT-3.5 and Vicuna even exhibit slightly diminished performance.
In Table </span><a href="#S3.T2" title="Table 2 ‣ 3.3.1 Can All LLMs Rephrase Questions? ‣ 3.3 Performance across Various LLMs ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">2</span></a><span id="S3.SS3.SSS1.p1.4.13" class="ltx_text" style="font-size:70%;">, we closely examine specific examples of self-rephrased questions by different models.
Initial observations suggest that </span><span id="S3.SS3.SSS1.p1.4.14" class="ltx_text ltx_font_typewriter" style="font-size:70%;">Vicuna-13b-v1.5</span><span id="S3.SS3.SSS1.p1.4.15" class="ltx_text" style="font-size:70%;">’s rephrased questions seldom offer substantial clarification, often mirroring the simplicity of their original questions.
In the last instance of Table </span><a href="#S3.T2" title="Table 2 ‣ 3.3.1 Can All LLMs Rephrase Questions? ‣ 3.3 Performance across Various LLMs ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">2</span></a><span id="S3.SS3.SSS1.p1.4.16" class="ltx_text" style="font-size:70%;">, </span><span id="S3.SS3.SSS1.p1.4.17" class="ltx_text ltx_font_typewriter" style="font-size:70%;">Vicuna-13b-v1.5</span><span id="S3.SS3.SSS1.p1.4.18" class="ltx_text" style="font-size:70%;"> perturbs the question’s intent by changing “yesterday” to “</span><span id="S3.SS3.SSS1.p1.4.19" class="ltx_text" style="font-size:70%;color:#FF0000;">today</span><span id="S3.SS3.SSS1.p1.4.20" class="ltx_text" style="font-size:70%;">”.
While both GPT-3.5 and GPT-4 can elucidate questions, GPT-3.5 occasionally introduces extra details or misinterpretations.
As shown in the second example of Table </span><a href="#S3.T2" title="Table 2 ‣ 3.3.1 Can All LLMs Rephrase Questions? ‣ 3.3 Performance across Various LLMs ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">2</span></a><span id="S3.SS3.SSS1.p1.4.21" class="ltx_text" style="font-size:70%;">, GPT-3.5 misinterprets the concept of even month as “</span><span id="S3.SS3.SSS1.p1.4.22" class="ltx_text" style="font-size:70%;color:#FF0000;">a month with an even number of days</span><span id="S3.SS3.SSS1.p1.4.23" class="ltx_text" style="font-size:70%;">”. Similarly, in the third example, GPT-3.5 introduces a wrong constraint of “</span><span id="S3.SS3.SSS1.p1.4.24" class="ltx_text" style="font-size:70%;color:#FF0000;">recent</span><span id="S3.SS3.SSS1.p1.4.25" class="ltx_text" style="font-size:70%;">” game.
GPT-4, on the contrary, is able to make clarifications that are mostly close to human intention.
We also observe that GPT-3.5 tends to introduce the following phrase to rephrased questions in Sports (</span><math id="S3.SS3.SSS1.p1.1.m1.1" class="ltx_Math" alttext="45" display="inline"><semantics id="S3.SS3.SSS1.p1.1.m1.1a"><mn mathsize="70%" id="S3.SS3.SSS1.p1.1.m1.1.1" xref="S3.SS3.SSS1.p1.1.m1.1.1.cmml">45</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.p1.1.m1.1b"><cn type="integer" id="S3.SS3.SSS1.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS1.p1.1.m1.1.1">45</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.p1.1.m1.1c">45</annotation></semantics></math><span id="S3.SS3.SSS1.p1.4.26" class="ltx_text" style="font-size:70%;"> out of </span><math id="S3.SS3.SSS1.p1.2.m2.1" class="ltx_Math" alttext="220" display="inline"><semantics id="S3.SS3.SSS1.p1.2.m2.1a"><mn mathsize="70%" id="S3.SS3.SSS1.p1.2.m2.1.1" xref="S3.SS3.SSS1.p1.2.m2.1.1.cmml">220</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.p1.2.m2.1b"><cn type="integer" id="S3.SS3.SSS1.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS1.p1.2.m2.1.1">220</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.p1.2.m2.1c">220</annotation></semantics></math><span id="S3.SS3.SSS1.p1.4.27" class="ltx_text" style="font-size:70%;">) and Dates (</span><math id="S3.SS3.SSS1.p1.3.m3.1" class="ltx_Math" alttext="21" display="inline"><semantics id="S3.SS3.SSS1.p1.3.m3.1a"><mn mathsize="70%" id="S3.SS3.SSS1.p1.3.m3.1.1" xref="S3.SS3.SSS1.p1.3.m3.1.1.cmml">21</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.p1.3.m3.1b"><cn type="integer" id="S3.SS3.SSS1.p1.3.m3.1.1.cmml" xref="S3.SS3.SSS1.p1.3.m3.1.1">21</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.p1.3.m3.1c">21</annotation></semantics></math><span id="S3.SS3.SSS1.p1.4.28" class="ltx_text" style="font-size:70%;"> out of </span><math id="S3.SS3.SSS1.p1.4.m4.1" class="ltx_Math" alttext="369" display="inline"><semantics id="S3.SS3.SSS1.p1.4.m4.1a"><mn mathsize="70%" id="S3.SS3.SSS1.p1.4.m4.1.1" xref="S3.SS3.SSS1.p1.4.m4.1.1.cmml">369</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.p1.4.m4.1b"><cn type="integer" id="S3.SS3.SSS1.p1.4.m4.1.1.cmml" xref="S3.SS3.SSS1.p1.4.m4.1.1">369</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.p1.4.m4.1c">369</annotation></semantics></math><span id="S3.SS3.SSS1.p1.4.29" class="ltx_text" style="font-size:70%;">): “Please rephrase and provide additional details if necessary to enhance your response accuracy.”, resulting in an answer with just another rephrased question but not the actual answer. Therefore, we remove all sentences containing “rephrase” for GPT-3.5 on these two datasets.</span></p>
</div>
<figure id="S3.T2" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 2: </span>Examples of the self-rephrased questions generated by different LLM models. A stronger model makes fewer mistakes in rephrasing the questions. </figcaption>
<div id="S3.T2.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:530.1pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(25.4pt,-31.0pt) scale(1.13258903005496,1.13258903005496) ;">
<table id="S3.T2.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S3.T2.3.1.1" class="ltx_tr">
<td id="S3.T2.3.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="S3.T2.3.1.1.1.1" class="ltx_text" style="font-size:70%;">Model</span></td>
<td id="S3.T2.3.1.1.2" class="ltx_td ltx_align_left ltx_border_tt"><span id="S3.T2.3.1.1.2.1" class="ltx_text" style="font-size:70%;">Self-rephrased Question</span></td>
</tr>
<tr id="S3.T2.3.1.2" class="ltx_tr">
<td id="S3.T2.3.1.2.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T2.3.1.2.1.1" class="ltx_text" style="font-size:70%;">Original</span></td>
<td id="S3.T2.3.1.2.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T2.3.1.2.2.1" class="ltx_text" style="font-size:70%;">Was Barack Obama born in an even day?</span></td>
</tr>
<tr id="S3.T2.3.1.3" class="ltx_tr">
<td id="S3.T2.3.1.3.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.3.1.3.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;">GPT-4-0613</span></td>
<td id="S3.T2.3.1.3.2" class="ltx_td ltx_align_left">
<span id="S3.T2.3.1.3.2.1" class="ltx_text" style="font-size:70%;">Did Barack Obama’s birth occur on a </span><span id="S3.T2.3.1.3.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">day of the month</span><span id="S3.T2.3.1.3.2.3" class="ltx_text" style="font-size:70%;"> that is considered an even number?</span>
</td>
</tr>
<tr id="S3.T2.3.1.4" class="ltx_tr">
<td id="S3.T2.3.1.4.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="2"><span id="S3.T2.3.1.4.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;">GPT-3.5-turbo-0613</span></td>
<td id="S3.T2.3.1.4.2" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.4.2.1" class="ltx_text" style="font-size:70%;">Was Barack Obama born on a day that is evenly divisible by two, resulting in a whole number</span></td>
</tr>
<tr id="S3.T2.3.1.5" class="ltx_tr">
<td id="S3.T2.3.1.5.1" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.5.1.1" class="ltx_text" style="font-size:70%;">with no remainder when divided by two?</span></td>
</tr>
<tr id="S3.T2.3.1.6" class="ltx_tr" style="background-color:#E6E6E6;">
<td id="S3.T2.3.1.6.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.3.1.6.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;background-color:#E6E6E6;">Vicuna-13b-v1.5</span></td>
<td id="S3.T2.3.1.6.2" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.6.2.1" class="ltx_text" style="font-size:70%;background-color:#E6E6E6;">What is the birthday of Barack Obama, and is it an even day?</span></td>
</tr>
<tr id="S3.T2.3.1.7" class="ltx_tr">
<td id="S3.T2.3.1.7.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T2.3.1.7.1.1" class="ltx_text" style="font-size:70%;">Original</span></td>
<td id="S3.T2.3.1.7.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T2.3.1.7.2.1" class="ltx_text" style="font-size:70%;">Was Abraham Lincoln born in an even month?</span></td>
</tr>
<tr id="S3.T2.3.1.8" class="ltx_tr">
<td id="S3.T2.3.1.8.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.3.1.8.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;">GPT-4-0613</span></td>
<td id="S3.T2.3.1.8.2" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.8.2.1" class="ltx_text" style="font-size:70%;">Was the former United States president Abraham Lincoln born in a month that is represented</span></td>
</tr>
<tr id="S3.T2.3.1.9" class="ltx_tr">
<td id="S3.T2.3.1.9.1" class="ltx_td ltx_border_r"></td>
<td id="S3.T2.3.1.9.2" class="ltx_td ltx_align_left">
<span id="S3.T2.3.1.9.2.1" class="ltx_text" style="font-size:70%;">by an </span><span id="S3.T2.3.1.9.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">even number, from January being ‘1’ to December being ‘12’</span><span id="S3.T2.3.1.9.2.3" class="ltx_text" style="font-size:70%;">?</span>
</td>
</tr>
<tr id="S3.T2.3.1.10" class="ltx_tr">
<td id="S3.T2.3.1.10.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="2"><span id="S3.T2.3.1.10.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;">GPT-3.5-turbo-0613</span></td>
<td id="S3.T2.3.1.10.2" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.10.2.1" class="ltx_text" style="font-size:70%;">Can you please clarify whether Abraham Lincoln, the 16th President of the United States, was</span></td>
</tr>
<tr id="S3.T2.3.1.11" class="ltx_tr">
<td id="S3.T2.3.1.11.1" class="ltx_td ltx_align_left">
<span id="S3.T2.3.1.11.1.1" class="ltx_text" style="font-size:70%;">born in </span><span id="S3.T2.3.1.11.1.2" class="ltx_text" style="font-size:70%;color:#FF0000;">a month with an even number of days?</span>
</td>
</tr>
<tr id="S3.T2.3.1.12" class="ltx_tr" style="background-color:#E6E6E6;">
<td id="S3.T2.3.1.12.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.3.1.12.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;background-color:#E6E6E6;">Vicuna-13b-v1.5</span></td>
<td id="S3.T2.3.1.12.2" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.12.2.1" class="ltx_text" style="font-size:70%;background-color:#E6E6E6;">What was the month in which Abraham Lincoln was born, and is that month even or odd?</span></td>
</tr>
<tr id="S3.T2.3.1.13" class="ltx_tr">
<td id="S3.T2.3.1.13.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T2.3.1.13.1.1" class="ltx_text" style="font-size:70%;">Original</span></td>
<td id="S3.T2.3.1.13.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T2.3.1.13.2.1" class="ltx_text" style="font-size:70%;">Is the following sentence plausible? “Amari Cooper scored a touchdown”</span></td>
</tr>
<tr id="S3.T2.3.1.14" class="ltx_tr">
<td id="S3.T2.3.1.14.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="2"><span id="S3.T2.3.1.14.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;">GPT-4-0613</span></td>
<td id="S3.T2.3.1.14.2" class="ltx_td ltx_align_left">
<span id="S3.T2.3.1.14.2.1" class="ltx_text" style="font-size:70%;">Is it believable or likely that Amari Cooper, who is </span><span id="S3.T2.3.1.14.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">known for playing football</span><span id="S3.T2.3.1.14.2.3" class="ltx_text" style="font-size:70%;">, scored a</span>
</td>
</tr>
<tr id="S3.T2.3.1.15" class="ltx_tr">
<td id="S3.T2.3.1.15.1" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.15.1.1" class="ltx_text" style="font-size:70%;">touchdown?</span></td>
</tr>
<tr id="S3.T2.3.1.16" class="ltx_tr">
<td id="S3.T2.3.1.16.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="2"><span id="S3.T2.3.1.16.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;">GPT-3.5-turbo-0613</span></td>
<td id="S3.T2.3.1.16.2" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.16.2.1" class="ltx_text" style="font-size:70%;">Can you confirm if it is reasonable to say, based on current facts or observations, that Amari</span></td>
</tr>
<tr id="S3.T2.3.1.17" class="ltx_tr">
<td id="S3.T2.3.1.17.1" class="ltx_td ltx_align_left">
<span id="S3.T2.3.1.17.1.1" class="ltx_text" style="font-size:70%;">Cooper, </span><span id="S3.T2.3.1.17.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">the football player</span><span id="S3.T2.3.1.17.1.3" class="ltx_text" style="font-size:70%;">, successfully achieved a touchdown during a </span><span id="S3.T2.3.1.17.1.4" class="ltx_text" style="font-size:70%;color:#FF0000;">recent</span><span id="S3.T2.3.1.17.1.5" class="ltx_text" style="font-size:70%;"> game or event?</span>
</td>
</tr>
<tr id="S3.T2.3.1.18" class="ltx_tr" style="background-color:#E6E6E6;">
<td id="S3.T2.3.1.18.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.3.1.18.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;background-color:#E6E6E6;">Vicuna-13b-v1.5</span></td>
<td id="S3.T2.3.1.18.2" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.18.2.1" class="ltx_text" style="font-size:70%;background-color:#E6E6E6;">Is it plausible that Amari Cooper scored a touchdown?</span></td>
</tr>
<tr id="S3.T2.3.1.19" class="ltx_tr">
<td id="S3.T2.3.1.19.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T2.3.1.19.1.1" class="ltx_text" style="font-size:70%;">Original</span></td>
<td id="S3.T2.3.1.19.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T2.3.1.19.2.1" class="ltx_text" style="font-size:70%;">Yesterday was April 30, 2021. What is the date tomorrow in MM/DD/YYYY?</span></td>
</tr>
<tr id="S3.T2.3.1.20" class="ltx_tr">
<td id="S3.T2.3.1.20.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="3"><span id="S3.T2.3.1.20.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;">GPT-4-0613</span></td>
<td id="S3.T2.3.1.20.2" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.20.2.1" class="ltx_text" style="font-size:70%;">If yesterday was the last day of April in the year 2021, which is 04/30/2021, can we figure</span></td>
</tr>
<tr id="S3.T2.3.1.21" class="ltx_tr">
<td id="S3.T2.3.1.21.1" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.21.1.1" class="ltx_text" style="font-size:70%;">out what the date will be the day after today, using the format of the month first, then day, and</span></td>
</tr>
<tr id="S3.T2.3.1.22" class="ltx_tr">
<td id="S3.T2.3.1.22.1" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.22.1.1" class="ltx_text" style="font-size:70%;">lastly year (MM/DD/YYYY)?</span></td>
</tr>
<tr id="S3.T2.3.1.23" class="ltx_tr">
<td id="S3.T2.3.1.23.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="3"><span id="S3.T2.3.1.23.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;">GPT-3.5-turbo-0613</span></td>
<td id="S3.T2.3.1.23.2" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.23.2.1" class="ltx_text" style="font-size:70%;">Yesterday, which was April 30, 2021, just passed. I am looking to determine the date that</span></td>
</tr>
<tr id="S3.T2.3.1.24" class="ltx_tr">
<td id="S3.T2.3.1.24.1" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.24.1.1" class="ltx_text" style="font-size:70%;">comes after today. To provide an accurate response, I need to express the date format as</span></td>
</tr>
<tr id="S3.T2.3.1.25" class="ltx_tr">
<td id="S3.T2.3.1.25.1" class="ltx_td ltx_align_left"><span id="S3.T2.3.1.25.1.1" class="ltx_text" style="font-size:70%;">MM/DD/YYYY. Can you please assist me in figuring out the date?</span></td>
</tr>
<tr id="S3.T2.3.1.26" class="ltx_tr" style="background-color:#FFCCCC;">
<td id="S3.T2.3.1.26.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S3.T2.3.1.26.1.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;background-color:#FFCCCC;">Vicuna-13b-v1.5</span></td>
<td id="S3.T2.3.1.26.2" class="ltx_td ltx_align_left ltx_border_bb"><span id="S3.T2.3.1.26.2.1" class="ltx_text" style="font-size:70%;background-color:#FFCCCC;">If <span id="S3.T2.3.1.26.2.1.1" class="ltx_text" style="color:#FF0000;">today</span> is April 30, 2021, what will be the date tomorrow in the format of MM/DD/YYYY?</span></td>
</tr>
</table>
</span></div>
</figure>
<div id="S3.SS3.SSS1.p2" class="ltx_para">
<p id="S3.SS3.SSS1.p2.1" class="ltx_p"><span id="S3.SS3.SSS1.p2.1.1" class="ltx_text" style="font-size:70%;">We wrap up this experiment with the following key insight.</span></p>
</div>
<div id="S3.SS3.SSS1.p3" class="ltx_para ltx_noindent">
<svg id="S3.SS3.SSS1.p3.pic1" class="ltx_picture" height="54.62" overflow="visible" version="1.1" width="600"><g transform="translate(0,54.62) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B3D7E8" fill-opacity="1.0"><path d="M 0 5.91 L 0 48.72 C 0 51.98 2.64 54.62 5.91 54.62 L 594.09 54.62 C 597.36 54.62 600 51.98 600 48.72 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#FAFCFD" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 34.2 L 598.03 34.2 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 40.11)"><foreignObject width="556.69" height="8.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S3.SS3.SSS1.p3.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S3.SS3.SSS1.p3.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S3.SS3.SSS1.p3.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Takeaway #3</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="8.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S3.SS3.SSS1.p3.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S3.SS3.SSS1.p3.pic1.2.2.2.1.1.1" class="ltx_p"><span id="S3.SS3.SSS1.p3.pic1.2.2.2.1.1.1.1" class="ltx_text" style="font-size:70%;">All models can benefit from rephrasing questions, with more advanced models expected to gain a larger improvement.</span></span>
</span></foreignObject></g></g></svg>
</div>
</section>
<section id="S3.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsubsection">3.3.2 </span>Are the Rephrased Questions Transferable?</h4>

<figure id="S3.T3" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 3: </span>Comparison of GPT-4’s rephrased questions with Vicuna’s self-rephrased questions. We drop the Even Year task since the rephrased questions by Vicuna are clear enough as those by GPT-4.</figcaption>
<table id="S3.T3.3" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S3.T3.3.1" class="ltx_tr">
<td id="S3.T3.3.1.1" class="ltx_td ltx_border_tt"></td>
<td id="S3.T3.3.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T3.3.1.2.1" class="ltx_text" style="font-size:70%;">Even day</span></td>
<td id="S3.T3.3.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T3.3.1.3.1" class="ltx_text" style="font-size:70%;">Even month</span></td>
<td id="S3.T3.3.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T3.3.1.4.1" class="ltx_text" style="font-size:70%;">CSQA</span></td>
<td id="S3.T3.3.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T3.3.1.5.1" class="ltx_text" style="font-size:70%;">Sports</span></td>
<td id="S3.T3.3.1.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T3.3.1.6.1" class="ltx_text" style="font-size:70%;">Dates</span></td>
<td id="S3.T3.3.1.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T3.3.1.7.1" class="ltx_text" style="font-size:70%;">Last letter (2)</span></td>
</tr>
<tr id="S3.T3.3.2" class="ltx_tr">
<td id="S3.T3.3.2.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.3.2.1.1" class="ltx_text" style="font-size:70%;">Vicuna-rephrased</span></td>
<td id="S3.T3.3.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.3.2.2.1" class="ltx_text" style="font-size:70%;">66.36</span></td>
<td id="S3.T3.3.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.3.2.3.1" class="ltx_text" style="font-size:70%;">74.09</span></td>
<td id="S3.T3.3.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.3.2.4.1" class="ltx_text" style="font-size:70%;">58.18</span></td>
<td id="S3.T3.3.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.3.2.5.1" class="ltx_text" style="font-size:70%;">50.91</span></td>
<td id="S3.T3.3.2.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.3.2.6.1" class="ltx_text" style="font-size:70%;">33.33</span></td>
<td id="S3.T3.3.2.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.3.2.7.1" class="ltx_text" style="font-size:70%;">5.45</span></td>
</tr>
<tr id="S3.T3.3.3" class="ltx_tr">
<td id="S3.T3.3.3.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.3.3.1.1" class="ltx_text" style="font-size:70%;">GPT-4-rephrased</span></td>
<td id="S3.T3.3.3.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.3.3.2.1" class="ltx_text ltx_font_bold" style="font-size:70%;">72.73</span></td>
<td id="S3.T3.3.3.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.3.3.3.1" class="ltx_text ltx_font_bold" style="font-size:70%;">76.82</span></td>
<td id="S3.T3.3.3.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.3.3.4.1" class="ltx_text ltx_font_bold" style="font-size:70%;">59.55</span></td>
<td id="S3.T3.3.3.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.3.3.5.1" class="ltx_text ltx_font_bold" style="font-size:70%;">60.91</span></td>
<td id="S3.T3.3.3.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.3.3.6.1" class="ltx_text ltx_font_bold" style="font-size:70%;">39.30</span></td>
<td id="S3.T3.3.3.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.3.3.7.1" class="ltx_text ltx_font_bold" style="font-size:70%;">10.91</span></td>
</tr>
</table>
</figure>
<div id="S3.SS3.SSS2.p1" class="ltx_para">
<p id="S3.SS3.SSS2.p1.1" class="ltx_p"><span id="S3.SS3.SSS2.p1.1.1" class="ltx_text" style="font-size:70%;">Here, we examine if the rephrased questions generated by Two-step RaR are transferable across different LLMs. In particular, we would like to know if the rephrased questions generated by GPT-4 can benefit Vicuna’s performance. We detail </span><span id="S3.SS3.SSS2.p1.1.2" class="ltx_text ltx_font_typewriter" style="font-size:70%;">Vicuna-13b-v1.5</span><span id="S3.SS3.SSS2.p1.1.3" class="ltx_text" style="font-size:70%;">’s performance on questions rephrased by GPT-4, as compared to its own rephrased questions in Table </span><a href="#S3.T3" title="Table 3 ‣ 3.3.2 Are the Rephrased Questions Transferable? ‣ 3.3 Performance across Various LLMs ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">3</span></a><span id="S3.SS3.SSS2.p1.1.4" class="ltx_text" style="font-size:70%;">.
Consistent with our expectation that GPT-4 can better align with human intention and clarify the question, we observe that its rephrased questions remarkably enhance </span><span id="S3.SS3.SSS2.p1.1.5" class="ltx_text ltx_font_typewriter" style="font-size:70%;">Vicuna-13b-v1.5</span><span id="S3.SS3.SSS2.p1.1.6" class="ltx_text" style="font-size:70%;">’s performance on several tasks, especially when Vicuna’s self-rephrased questions exhibit low quality.
Indeed, the questions can be clarified further for Vicuna, but more exploration needs to be made on its capability of self-rephrased questions.
We conclude this experiment by the following key message.</span></p>
</div>
<div id="S3.SS3.SSS2.p2" class="ltx_para ltx_noindent">
<svg id="S3.SS3.SSS2.p2.pic1" class="ltx_picture" height="54.62" overflow="visible" version="1.1" width="600"><g transform="translate(0,54.62) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B3D7E8" fill-opacity="1.0"><path d="M 0 5.91 L 0 48.72 C 0 51.98 2.64 54.62 5.91 54.62 L 594.09 54.62 C 597.36 54.62 600 51.98 600 48.72 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#FAFCFD" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 34.2 L 598.03 34.2 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 40.11)"><foreignObject width="556.69" height="8.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S3.SS3.SSS2.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S3.SS3.SSS2.p2.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S3.SS3.SSS2.p2.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Takeaway #4</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="8.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S3.SS3.SSS2.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S3.SS3.SSS2.p2.pic1.2.2.2.1.1.1" class="ltx_p"><span id="S3.SS3.SSS2.p2.pic1.2.2.2.1.1.1.1" class="ltx_text" style="font-size:70%;">The rephrased questions are transferable: the questions rephrased by GPT-4 can improve the response quality on Vicuna.</span></span>
</span></foreignObject></g></g></svg>
</div>
</section>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Multiple Rephrasings: Will the Questions Converge?</h3>

<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.1" class="ltx_p"><span id="S3.SS4.p1.1.1" class="ltx_text" style="font-size:70%;">In this subsection, we explore whether iterative self-rephrasing by GPT-4 yields consistent clarifications when using Two-step RaR.
Specifically, we utilize prompt (</span><a href="#S2.E2" title="Equation 2.2 ‣ 2.2 Two-step RaR: Rephrase the Question and Respond to the Rephrased Question ‣ 2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">2.2</span></a><span id="S3.SS4.p1.1.2" class="ltx_text" style="font-size:70%;">) in Two-step RaR to enable GPT-4 to rephrase a question, then feed its output back into the same prompt (</span><a href="#S2.E2" title="Equation 2.2 ‣ 2.2 Two-step RaR: Rephrase the Question and Respond to the Rephrased Question ‣ 2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">2.2</span></a><span id="S3.SS4.p1.1.3" class="ltx_text" style="font-size:70%;">) for a second and third round of rephrasing.
In Table </span><a href="#S3.T4" title="Table 4 ‣ 3.4 Multiple Rephrasings: Will the Questions Converge? ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">4</span></a><span id="S3.SS4.p1.1.4" class="ltx_text" style="font-size:70%;">, we consider “Was Abraham Lincoln born on an even day?” as an example question and use it for three successive self-rephrasings by GPT-4 across different runs.
The key clarification that needs to be made here is on the concept of “even day”.
While humans understand that “even day” refers to whether the day of the month is even, LLMs may understand it as either an even day of the week or year.
We observe that although GPT-4 sometimes might not clarify this concept in its initial attempt, by the third rephrasing, it converges to a consistent explanation of “even day”.
Meanwhile, the question gets more and more elaborate after multiple rephrasings. This conveys the following key message.</span></p>
</div>
<figure id="S3.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 4: </span>Examples of the self-rephrased questions generated by GPT-4 after the 1st, 2nd and 3rd self-rephrasing. The original question is “Was Abraham Lincoln born in an even day?” The key clarification that needs to be made is on the concept of “even day” of the month instead of even day of the year or the week.</figcaption>
<div id="S3.T4.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:316.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(32.1pt,-23.4pt) scale(1.17368945427281,1.17368945427281) ;">
<table id="S3.T4.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S3.T4.3.1.1" class="ltx_tr">
<td id="S3.T4.3.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="S3.T4.3.1.1.1.1" class="ltx_text" style="font-size:70%;">Rephrase times</span></td>
<td id="S3.T4.3.1.1.2" class="ltx_td ltx_align_left ltx_border_tt"><span id="S3.T4.3.1.1.2.1" class="ltx_text" style="font-size:70%;">Self-rephrased Question</span></td>
</tr>
<tr id="S3.T4.3.1.2" class="ltx_tr">
<td id="S3.T4.3.1.2.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="2"><span id="S3.T4.3.1.2.1.1" class="ltx_text" style="font-size:70%;">1st</span></td>
<td id="S3.T4.3.1.2.2" class="ltx_td ltx_align_left ltx_border_t">
<span id="S3.T4.3.1.2.2.1" class="ltx_text" style="font-size:70%;">Did the birth of Abraham Lincoln, the sixteenth president of the United States, occur on a </span><span id="S3.T4.3.1.2.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">day</span>
</td>
</tr>
<tr id="S3.T4.3.1.3" class="ltx_tr">
<td id="S3.T4.3.1.3.1" class="ltx_td ltx_align_left">
<span id="S3.T4.3.1.3.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">of the month</span><span id="S3.T4.3.1.3.1.2" class="ltx_text" style="font-size:70%;"> that is considered an even number?</span>
</td>
</tr>
<tr id="S3.T4.3.1.4" class="ltx_tr">
<td id="S3.T4.3.1.4.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="2"><span id="S3.T4.3.1.4.1.1" class="ltx_text" style="font-size:70%;">2nd</span></td>
<td id="S3.T4.3.1.4.2" class="ltx_td ltx_align_left"><span id="S3.T4.3.1.4.2.1" class="ltx_text" style="font-size:70%;">Can you confirm if the birth date of Abraham Lincoln, who was the sixteenth person to hold the</span></td>
</tr>
<tr id="S3.T4.3.1.5" class="ltx_tr">
<td id="S3.T4.3.1.5.1" class="ltx_td ltx_align_left">
<span id="S3.T4.3.1.5.1.1" class="ltx_text" style="font-size:70%;">office of the President of the United States, falls on an even number in the </span><span id="S3.T4.3.1.5.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">monthly calendar</span><span id="S3.T4.3.1.5.1.3" class="ltx_text" style="font-size:70%;">?</span>
</td>
</tr>
<tr id="S3.T4.3.1.6" class="ltx_tr">
<td id="S3.T4.3.1.6.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="3"><span id="S3.T4.3.1.6.1.1" class="ltx_text" style="font-size:70%;">3rd</span></td>
<td id="S3.T4.3.1.6.2" class="ltx_td ltx_align_left">
<span id="S3.T4.3.1.6.2.1" class="ltx_text" style="font-size:70%;">Could you verify whether the </span><span id="S3.T4.3.1.6.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">day of the month</span><span id="S3.T4.3.1.6.2.3" class="ltx_text" style="font-size:70%;"> on which Abraham Lincoln, the sixteenth</span>
</td>
</tr>
<tr id="S3.T4.3.1.7" class="ltx_tr">
<td id="S3.T4.3.1.7.1" class="ltx_td ltx_align_left"><span id="S3.T4.3.1.7.1.1" class="ltx_text" style="font-size:70%;">president of the United States, was born, is an even number? The focus here is on identifying the</span></td>
</tr>
<tr id="S3.T4.3.1.8" class="ltx_tr">
<td id="S3.T4.3.1.8.1" class="ltx_td ltx_align_left"><span id="S3.T4.3.1.8.1.1" class="ltx_text" style="font-size:70%;">specific date and determining if it falls on an even or odd day according to the calendar.</span></td>
</tr>
<tr id="S3.T4.3.1.9" class="ltx_tr">
<td id="S3.T4.3.1.9.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="2"><span id="S3.T4.3.1.9.1.1" class="ltx_text" style="font-size:70%;">1st</span></td>
<td id="S3.T4.3.1.9.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T4.3.1.9.2.1" class="ltx_text" style="font-size:70%;">Is the date of Abraham Lincoln’s birth an even day? Provide more details, if possible, about his</span></td>
</tr>
<tr id="S3.T4.3.1.10" class="ltx_tr">
<td id="S3.T4.3.1.10.1" class="ltx_td ltx_align_left"><span id="S3.T4.3.1.10.1.1" class="ltx_text" style="font-size:70%;">birth date.</span></td>
</tr>
<tr id="S3.T4.3.1.11" class="ltx_tr">
<td id="S3.T4.3.1.11.1" class="ltx_td ltx_align_center ltx_border_r" rowspan="2"><span id="S3.T4.3.1.11.1.1" class="ltx_text" style="font-size:70%;">2nd</span></td>
<td id="S3.T4.3.1.11.2" class="ltx_td ltx_align_left"><span id="S3.T4.3.1.11.2.1" class="ltx_text" style="font-size:70%;">Can you confirm whether the day on which Abraham Lincoln was born is an even number or</span></td>
</tr>
<tr id="S3.T4.3.1.12" class="ltx_tr">
<td id="S3.T4.3.1.12.1" class="ltx_td ltx_align_left"><span id="S3.T4.3.1.12.1.1" class="ltx_text" style="font-size:70%;">not? Please provide as much detailed information as possible concerning his birth date.</span></td>
</tr>
<tr id="S3.T4.3.1.13" class="ltx_tr">
<td id="S3.T4.3.1.13.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" rowspan="3"><span id="S3.T4.3.1.13.1.1" class="ltx_text" style="font-size:70%;">3rd</span></td>
<td id="S3.T4.3.1.13.2" class="ltx_td ltx_align_left">
<span id="S3.T4.3.1.13.2.1" class="ltx_text" style="font-size:70%;">Could you verify if the </span><span id="S3.T4.3.1.13.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">day of the month</span><span id="S3.T4.3.1.13.2.3" class="ltx_text" style="font-size:70%;"> on which Abraham Lincoln was born falls on an even</span>
</td>
</tr>
<tr id="S3.T4.3.1.14" class="ltx_tr">
<td id="S3.T4.3.1.14.1" class="ltx_td ltx_align_left"><span id="S3.T4.3.1.14.1.1" class="ltx_text" style="font-size:70%;">number, please provide a comprehensive description around his birth date, while emphasizing on</span></td>
</tr>
<tr id="S3.T4.3.1.15" class="ltx_tr">
<td id="S3.T4.3.1.15.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S3.T4.3.1.15.1.1" class="ltx_text" style="font-size:70%;">the day of his birth especially, to assist with this confirmation?</span></td>
</tr>
</table>
</span></div>
</figure>
<div id="S3.SS4.p2" class="ltx_para ltx_noindent">
<svg id="S3.SS4.p2.pic1" class="ltx_picture" height="54.62" overflow="visible" version="1.1" width="600"><g transform="translate(0,54.62) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#B3D7E8" fill-opacity="1.0"><path d="M 0 5.91 L 0 48.72 C 0 51.98 2.64 54.62 5.91 54.62 L 594.09 54.62 C 597.36 54.62 600 51.98 600 48.72 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#FAFCFD" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 34.2 L 598.03 34.2 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 40.11)"><foreignObject width="556.69" height="8.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S3.SS4.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S3.SS4.p2.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S3.SS4.p2.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Takeaway #5</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="8.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S3.SS4.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S3.SS4.p2.pic1.2.2.2.1.1.1" class="ltx_p"><span id="S3.SS4.p2.pic1.2.2.2.1.1.1.1" class="ltx_text" style="font-size:70%;">GPT-4 can potentially clarify concepts with multiple rephrasing, even if it fails to make it in the initial attempt.</span></span>
</span></foreignObject></g></g></svg>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section" style="font-size:70%;">
<span class="ltx_tag ltx_tag_section">4 </span>Comparison with Chain-of-Thought</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p"><span id="S4.p1.1.1" class="ltx_text" style="font-size:70%;">In this section, we compare RaR with CoT. We first present the mathematical formualtions of RaR and CoT and compare them with each other. Then we present experimental results to show that (1) RaR offers improvements in scenarios where zero-shot CoT is ineffective; and (2) RaR addresses and corrects the shortcomings inherent in few-shot CoT.</span></p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Mathematical Formulation</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.13" class="ltx_p"><span id="S4.SS1.p1.13.1" class="ltx_text" style="font-size:70%;">In this subsection, we discuss the formulations of CoT and RaR, respectively. We denote the LLM model by </span><math id="S4.SS1.p1.1.m1.1" class="ltx_Math" alttext="p_{\text{LLM}}" display="inline"><semantics id="S4.SS1.p1.1.m1.1a"><msub id="S4.SS1.p1.1.m1.1.1" xref="S4.SS1.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.SS1.p1.1.m1.1.1.2" xref="S4.SS1.p1.1.m1.1.1.2.cmml">p</mi><mtext mathsize="70%" id="S4.SS1.p1.1.m1.1.1.3" xref="S4.SS1.p1.1.m1.1.1.3a.cmml">LLM</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.1.m1.1b"><apply id="S4.SS1.p1.1.m1.1.1.cmml" xref="S4.SS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.1.m1.1.1.1.cmml" xref="S4.SS1.p1.1.m1.1.1">subscript</csymbol><ci id="S4.SS1.p1.1.m1.1.1.2.cmml" xref="S4.SS1.p1.1.m1.1.1.2">𝑝</ci><ci id="S4.SS1.p1.1.m1.1.1.3a.cmml" xref="S4.SS1.p1.1.m1.1.1.3"><mtext mathsize="49%" id="S4.SS1.p1.1.m1.1.1.3.cmml" xref="S4.SS1.p1.1.m1.1.1.3">LLM</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.1.m1.1c">p_{\text{LLM}}</annotation></semantics></math><span id="S4.SS1.p1.13.2" class="ltx_text" style="font-size:70%;">. In detail, LLMs take the sequence </span><math id="S4.SS1.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{x}" display="inline"><semantics id="S4.SS1.p1.2.m2.1a"><mi mathsize="70%" id="S4.SS1.p1.2.m2.1.1" xref="S4.SS1.p1.2.m2.1.1.cmml">𝐱</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.2.m2.1b"><ci id="S4.SS1.p1.2.m2.1.1.cmml" xref="S4.SS1.p1.2.m2.1.1">𝐱</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.2.m2.1c">\mathbf{x}</annotation></semantics></math><span id="S4.SS1.p1.13.3" class="ltx_text" style="font-size:70%;"> as prompt, and generate the sentence following the distribution </span><math id="S4.SS1.p1.3.m3.1" class="ltx_math_unparsed" alttext="p_{\text{LLM}}(\cdot|\mathbf{x})" display="inline"><semantics id="S4.SS1.p1.3.m3.1a"><mrow id="S4.SS1.p1.3.m3.1b"><msub id="S4.SS1.p1.3.m3.1.1"><mi mathsize="70%" id="S4.SS1.p1.3.m3.1.1.2">p</mi><mtext mathsize="70%" id="S4.SS1.p1.3.m3.1.1.3">LLM</mtext></msub><mrow id="S4.SS1.p1.3.m3.1.2"><mo maxsize="70%" minsize="70%" id="S4.SS1.p1.3.m3.1.2.1">(</mo><mo lspace="0em" mathsize="70%" rspace="0em" id="S4.SS1.p1.3.m3.1.2.2">⋅</mo><mo fence="false" maxsize="70%" minsize="70%" rspace="0.167em" id="S4.SS1.p1.3.m3.1.2.3">|</mo><mi mathsize="70%" id="S4.SS1.p1.3.m3.1.2.4">𝐱</mi><mo maxsize="70%" minsize="70%" id="S4.SS1.p1.3.m3.1.2.5">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.SS1.p1.3.m3.1c">p_{\text{LLM}}(\cdot|\mathbf{x})</annotation></semantics></math><span id="S4.SS1.p1.13.4" class="ltx_text" style="font-size:70%;">. Recently, there has been significant research effort emphasizing the use of instructions to enhance the quality of the generated text. Mathematically, instead of directly using the prompt </span><math id="S4.SS1.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{x}" display="inline"><semantics id="S4.SS1.p1.4.m4.1a"><mi mathsize="70%" id="S4.SS1.p1.4.m4.1.1" xref="S4.SS1.p1.4.m4.1.1.cmml">𝐱</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.4.m4.1b"><ci id="S4.SS1.p1.4.m4.1.1.cmml" xref="S4.SS1.p1.4.m4.1.1">𝐱</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.4.m4.1c">\mathbf{x}</annotation></semantics></math><span id="S4.SS1.p1.13.5" class="ltx_text" style="font-size:70%;"> to generate the response following </span><math id="S4.SS1.p1.5.m5.1" class="ltx_math_unparsed" alttext="p_{\text{LLM}}(\cdot|\mathbf{x})" display="inline"><semantics id="S4.SS1.p1.5.m5.1a"><mrow id="S4.SS1.p1.5.m5.1b"><msub id="S4.SS1.p1.5.m5.1.1"><mi mathsize="70%" id="S4.SS1.p1.5.m5.1.1.2">p</mi><mtext mathsize="70%" id="S4.SS1.p1.5.m5.1.1.3">LLM</mtext></msub><mrow id="S4.SS1.p1.5.m5.1.2"><mo maxsize="70%" minsize="70%" id="S4.SS1.p1.5.m5.1.2.1">(</mo><mo lspace="0em" mathsize="70%" rspace="0em" id="S4.SS1.p1.5.m5.1.2.2">⋅</mo><mo fence="false" maxsize="70%" minsize="70%" rspace="0.167em" id="S4.SS1.p1.5.m5.1.2.3">|</mo><mi mathsize="70%" id="S4.SS1.p1.5.m5.1.2.4">𝐱</mi><mo maxsize="70%" minsize="70%" id="S4.SS1.p1.5.m5.1.2.5">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.SS1.p1.5.m5.1c">p_{\text{LLM}}(\cdot|\mathbf{x})</annotation></semantics></math><span id="S4.SS1.p1.13.6" class="ltx_text" style="font-size:70%;">, one can use a different prompt </span><math id="S4.SS1.p1.6.m6.2" class="ltx_Math" alttext="\widetilde{\mathbf{x}}=[\mathbf{x},\mathbf{z}]" display="inline"><semantics id="S4.SS1.p1.6.m6.2a"><mrow id="S4.SS1.p1.6.m6.2.3" xref="S4.SS1.p1.6.m6.2.3.cmml"><mover accent="true" id="S4.SS1.p1.6.m6.2.3.2" xref="S4.SS1.p1.6.m6.2.3.2.cmml"><mi mathsize="70%" id="S4.SS1.p1.6.m6.2.3.2.2" xref="S4.SS1.p1.6.m6.2.3.2.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.SS1.p1.6.m6.2.3.2.1" xref="S4.SS1.p1.6.m6.2.3.2.1.cmml">~</mo></mover><mo mathsize="70%" id="S4.SS1.p1.6.m6.2.3.1" xref="S4.SS1.p1.6.m6.2.3.1.cmml">=</mo><mrow id="S4.SS1.p1.6.m6.2.3.3.2" xref="S4.SS1.p1.6.m6.2.3.3.1.cmml"><mo maxsize="70%" minsize="70%" id="S4.SS1.p1.6.m6.2.3.3.2.1" xref="S4.SS1.p1.6.m6.2.3.3.1.cmml">[</mo><mi mathsize="70%" id="S4.SS1.p1.6.m6.1.1" xref="S4.SS1.p1.6.m6.1.1.cmml">𝐱</mi><mo mathsize="70%" id="S4.SS1.p1.6.m6.2.3.3.2.2" xref="S4.SS1.p1.6.m6.2.3.3.1.cmml">,</mo><mi mathsize="70%" id="S4.SS1.p1.6.m6.2.2" xref="S4.SS1.p1.6.m6.2.2.cmml">𝐳</mi><mo maxsize="70%" minsize="70%" id="S4.SS1.p1.6.m6.2.3.3.2.3" xref="S4.SS1.p1.6.m6.2.3.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.6.m6.2b"><apply id="S4.SS1.p1.6.m6.2.3.cmml" xref="S4.SS1.p1.6.m6.2.3"><eq id="S4.SS1.p1.6.m6.2.3.1.cmml" xref="S4.SS1.p1.6.m6.2.3.1"></eq><apply id="S4.SS1.p1.6.m6.2.3.2.cmml" xref="S4.SS1.p1.6.m6.2.3.2"><ci id="S4.SS1.p1.6.m6.2.3.2.1.cmml" xref="S4.SS1.p1.6.m6.2.3.2.1">~</ci><ci id="S4.SS1.p1.6.m6.2.3.2.2.cmml" xref="S4.SS1.p1.6.m6.2.3.2.2">𝐱</ci></apply><interval closure="closed" id="S4.SS1.p1.6.m6.2.3.3.1.cmml" xref="S4.SS1.p1.6.m6.2.3.3.2"><ci id="S4.SS1.p1.6.m6.1.1.cmml" xref="S4.SS1.p1.6.m6.1.1">𝐱</ci><ci id="S4.SS1.p1.6.m6.2.2.cmml" xref="S4.SS1.p1.6.m6.2.2">𝐳</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.6.m6.2c">\widetilde{\mathbf{x}}=[\mathbf{x},\mathbf{z}]</annotation></semantics></math><span id="S4.SS1.p1.13.7" class="ltx_text" style="font-size:70%;"> augmented by instruction </span><math id="S4.SS1.p1.7.m7.1" class="ltx_Math" alttext="\mathbf{z}" display="inline"><semantics id="S4.SS1.p1.7.m7.1a"><mi mathsize="70%" id="S4.SS1.p1.7.m7.1.1" xref="S4.SS1.p1.7.m7.1.1.cmml">𝐳</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.7.m7.1b"><ci id="S4.SS1.p1.7.m7.1.1.cmml" xref="S4.SS1.p1.7.m7.1.1">𝐳</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.7.m7.1c">\mathbf{z}</annotation></semantics></math><span id="S4.SS1.p1.13.8" class="ltx_text" style="font-size:70%;"> to generate a different response following </span><math id="S4.SS1.p1.8.m8.1" class="ltx_math_unparsed" alttext="p_{\text{LLM}}(\cdot|\mathbf{x},\mathbf{z})" display="inline"><semantics id="S4.SS1.p1.8.m8.1a"><mrow id="S4.SS1.p1.8.m8.1b"><msub id="S4.SS1.p1.8.m8.1.1"><mi mathsize="70%" id="S4.SS1.p1.8.m8.1.1.2">p</mi><mtext mathsize="70%" id="S4.SS1.p1.8.m8.1.1.3">LLM</mtext></msub><mrow id="S4.SS1.p1.8.m8.1.2"><mo maxsize="70%" minsize="70%" id="S4.SS1.p1.8.m8.1.2.1">(</mo><mo lspace="0em" mathsize="70%" rspace="0em" id="S4.SS1.p1.8.m8.1.2.2">⋅</mo><mo fence="false" maxsize="70%" minsize="70%" rspace="0.167em" id="S4.SS1.p1.8.m8.1.2.3">|</mo><mi mathsize="70%" id="S4.SS1.p1.8.m8.1.2.4">𝐱</mi><mo mathsize="70%" id="S4.SS1.p1.8.m8.1.2.5">,</mo><mi mathsize="70%" id="S4.SS1.p1.8.m8.1.2.6">𝐳</mi><mo maxsize="70%" minsize="70%" id="S4.SS1.p1.8.m8.1.2.7">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.SS1.p1.8.m8.1c">p_{\text{LLM}}(\cdot|\mathbf{x},\mathbf{z})</annotation></semantics></math><span id="S4.SS1.p1.13.9" class="ltx_text" style="font-size:70%;">. We hypothesize that a successful instruction </span><math id="S4.SS1.p1.9.m9.1" class="ltx_Math" alttext="\mathbf{z}" display="inline"><semantics id="S4.SS1.p1.9.m9.1a"><mi mathsize="70%" id="S4.SS1.p1.9.m9.1.1" xref="S4.SS1.p1.9.m9.1.1.cmml">𝐳</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.9.m9.1b"><ci id="S4.SS1.p1.9.m9.1.1.cmml" xref="S4.SS1.p1.9.m9.1.1">𝐳</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.9.m9.1c">\mathbf{z}</annotation></semantics></math><span id="S4.SS1.p1.13.10" class="ltx_text" style="font-size:70%;"> allows us to extract a better answer from LLM. In this subsection, we employ the symbol </span><math id="S4.SS1.p1.10.m10.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.SS1.p1.10.m10.1a"><mi mathsize="70%" id="S4.SS1.p1.10.m10.1.1" xref="S4.SS1.p1.10.m10.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.10.m10.1b"><ci id="S4.SS1.p1.10.m10.1.1.cmml" xref="S4.SS1.p1.10.m10.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.10.m10.1c">\mathbf{y}</annotation></semantics></math><span id="S4.SS1.p1.13.11" class="ltx_text" style="font-size:70%;"> to represent the target answer we aim to generate. The notation </span><math id="S4.SS1.p1.11.m11.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.SS1.p1.11.m11.1a"><msup id="S4.SS1.p1.11.m11.1.1" xref="S4.SS1.p1.11.m11.1.1.cmml"><mi mathsize="70%" id="S4.SS1.p1.11.m11.1.1.2" xref="S4.SS1.p1.11.m11.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.SS1.p1.11.m11.1.1.3" xref="S4.SS1.p1.11.m11.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.11.m11.1b"><apply id="S4.SS1.p1.11.m11.1.1.cmml" xref="S4.SS1.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.11.m11.1.1.1.cmml" xref="S4.SS1.p1.11.m11.1.1">superscript</csymbol><ci id="S4.SS1.p1.11.m11.1.1.2.cmml" xref="S4.SS1.p1.11.m11.1.1.2">𝐲</ci><ci id="S4.SS1.p1.11.m11.1.1.3.cmml" xref="S4.SS1.p1.11.m11.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.11.m11.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.SS1.p1.13.12" class="ltx_text" style="font-size:70%;"> is used to denote an extended text that encompasses the desired answer </span><math id="S4.SS1.p1.12.m12.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.SS1.p1.12.m12.1a"><mi mathsize="70%" id="S4.SS1.p1.12.m12.1.1" xref="S4.SS1.p1.12.m12.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.12.m12.1b"><ci id="S4.SS1.p1.12.m12.1.1.cmml" xref="S4.SS1.p1.12.m12.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.12.m12.1c">\mathbf{y}</annotation></semantics></math><span id="S4.SS1.p1.13.13" class="ltx_text" style="font-size:70%;"> as well as additional details, such as the underlying reasoning. Often, </span><math id="S4.SS1.p1.13.m13.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.SS1.p1.13.m13.1a"><msup id="S4.SS1.p1.13.m13.1.1" xref="S4.SS1.p1.13.m13.1.1.cmml"><mi mathsize="70%" id="S4.SS1.p1.13.m13.1.1.2" xref="S4.SS1.p1.13.m13.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.SS1.p1.13.m13.1.1.3" xref="S4.SS1.p1.13.m13.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.13.m13.1b"><apply id="S4.SS1.p1.13.m13.1.1.cmml" xref="S4.SS1.p1.13.m13.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.13.m13.1.1.1.cmml" xref="S4.SS1.p1.13.m13.1.1">superscript</csymbol><ci id="S4.SS1.p1.13.m13.1.1.2.cmml" xref="S4.SS1.p1.13.m13.1.1.2">𝐲</ci><ci id="S4.SS1.p1.13.m13.1.1.3.cmml" xref="S4.SS1.p1.13.m13.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.13.m13.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.SS1.p1.13.14" class="ltx_text" style="font-size:70%;"> is produced by prompting with instructions like a Chain of Thought (CoT).</span></p>
</div>
<section id="S4.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsubsection">4.1.1 </span>Chain-of-Thought (CoT)</h4>

<div id="S4.SS1.SSS1.p1" class="ltx_para">
<p id="S4.SS1.SSS1.p1.3" class="ltx_p"><span id="S4.SS1.SSS1.p1.3.1" class="ltx_text" style="font-size:70%;">The core concept behind the CoT is to generate a text </span><math id="S4.SS1.SSS1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.SS1.SSS1.p1.1.m1.1a"><msup id="S4.SS1.SSS1.p1.1.m1.1.1" xref="S4.SS1.SSS1.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p1.1.m1.1.1.2" xref="S4.SS1.SSS1.p1.1.m1.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.SS1.SSS1.p1.1.m1.1.1.3" xref="S4.SS1.SSS1.p1.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS1.p1.1.m1.1b"><apply id="S4.SS1.SSS1.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p1.1.m1.1.1.1.cmml" xref="S4.SS1.SSS1.p1.1.m1.1.1">superscript</csymbol><ci id="S4.SS1.SSS1.p1.1.m1.1.1.2.cmml" xref="S4.SS1.SSS1.p1.1.m1.1.1.2">𝐲</ci><ci id="S4.SS1.SSS1.p1.1.m1.1.1.3.cmml" xref="S4.SS1.SSS1.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS1.p1.1.m1.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.SS1.SSS1.p1.3.2" class="ltx_text" style="font-size:70%;"> such that </span><math id="S4.SS1.SSS1.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.SS1.SSS1.p1.2.m2.1a"><msup id="S4.SS1.SSS1.p1.2.m2.1.1" xref="S4.SS1.SSS1.p1.2.m2.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p1.2.m2.1.1.2" xref="S4.SS1.SSS1.p1.2.m2.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.SS1.SSS1.p1.2.m2.1.1.3" xref="S4.SS1.SSS1.p1.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS1.p1.2.m2.1b"><apply id="S4.SS1.SSS1.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p1.2.m2.1.1.1.cmml" xref="S4.SS1.SSS1.p1.2.m2.1.1">superscript</csymbol><ci id="S4.SS1.SSS1.p1.2.m2.1.1.2.cmml" xref="S4.SS1.SSS1.p1.2.m2.1.1.2">𝐲</ci><ci id="S4.SS1.SSS1.p1.2.m2.1.1.3.cmml" xref="S4.SS1.SSS1.p1.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS1.p1.2.m2.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.SS1.SSS1.p1.3.3" class="ltx_text" style="font-size:70%;"> includes intermediate CoT steps and the final answer </span><math id="S4.SS1.SSS1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.SS1.SSS1.p1.3.m3.1a"><mi mathsize="70%" id="S4.SS1.SSS1.p1.3.m3.1.1" xref="S4.SS1.SSS1.p1.3.m3.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS1.p1.3.m3.1b"><ci id="S4.SS1.SSS1.p1.3.m3.1.1.cmml" xref="S4.SS1.SSS1.p1.3.m3.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS1.p1.3.m3.1c">\mathbf{y}</annotation></semantics></math><span id="S4.SS1.SSS1.p1.3.4" class="ltx_text" style="font-size:70%;">. In particular,</span></p>
<table id="A2.EGx4" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E1.m1.3" class="ltx_Math" alttext="\displaystyle\mathbf{y}^{\prime}=[\mathbf{s}_{1},\mathbf{s}_{2},\ldots,\mathbf{s}_{m},\mathbf{y}]," display="inline"><semantics id="S4.E1.m1.3a"><mrow id="S4.E1.m1.3.3.1" xref="S4.E1.m1.3.3.1.1.cmml"><mrow id="S4.E1.m1.3.3.1.1" xref="S4.E1.m1.3.3.1.1.cmml"><msup id="S4.E1.m1.3.3.1.1.5" xref="S4.E1.m1.3.3.1.1.5.cmml"><mi mathsize="70%" id="S4.E1.m1.3.3.1.1.5.2" xref="S4.E1.m1.3.3.1.1.5.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.E1.m1.3.3.1.1.5.3" xref="S4.E1.m1.3.3.1.1.5.3.cmml">′</mo></msup><mo mathsize="70%" id="S4.E1.m1.3.3.1.1.4" xref="S4.E1.m1.3.3.1.1.4.cmml">=</mo><mrow id="S4.E1.m1.3.3.1.1.3.3" xref="S4.E1.m1.3.3.1.1.3.4.cmml"><mo maxsize="70%" minsize="70%" id="S4.E1.m1.3.3.1.1.3.3.4" xref="S4.E1.m1.3.3.1.1.3.4.cmml">[</mo><msub id="S4.E1.m1.3.3.1.1.1.1.1" xref="S4.E1.m1.3.3.1.1.1.1.1.cmml"><mi mathsize="70%" id="S4.E1.m1.3.3.1.1.1.1.1.2" xref="S4.E1.m1.3.3.1.1.1.1.1.2.cmml">𝐬</mi><mn mathsize="70%" id="S4.E1.m1.3.3.1.1.1.1.1.3" xref="S4.E1.m1.3.3.1.1.1.1.1.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.E1.m1.3.3.1.1.3.3.5" xref="S4.E1.m1.3.3.1.1.3.4.cmml">,</mo><msub id="S4.E1.m1.3.3.1.1.2.2.2" xref="S4.E1.m1.3.3.1.1.2.2.2.cmml"><mi mathsize="70%" id="S4.E1.m1.3.3.1.1.2.2.2.2" xref="S4.E1.m1.3.3.1.1.2.2.2.2.cmml">𝐬</mi><mn mathsize="70%" id="S4.E1.m1.3.3.1.1.2.2.2.3" xref="S4.E1.m1.3.3.1.1.2.2.2.3.cmml">2</mn></msub><mo mathsize="70%" id="S4.E1.m1.3.3.1.1.3.3.6" xref="S4.E1.m1.3.3.1.1.3.4.cmml">,</mo><mi mathsize="70%" mathvariant="normal" id="S4.E1.m1.1.1" xref="S4.E1.m1.1.1.cmml">…</mi><mo mathsize="70%" id="S4.E1.m1.3.3.1.1.3.3.7" xref="S4.E1.m1.3.3.1.1.3.4.cmml">,</mo><msub id="S4.E1.m1.3.3.1.1.3.3.3" xref="S4.E1.m1.3.3.1.1.3.3.3.cmml"><mi mathsize="70%" id="S4.E1.m1.3.3.1.1.3.3.3.2" xref="S4.E1.m1.3.3.1.1.3.3.3.2.cmml">𝐬</mi><mi mathsize="70%" id="S4.E1.m1.3.3.1.1.3.3.3.3" xref="S4.E1.m1.3.3.1.1.3.3.3.3.cmml">m</mi></msub><mo mathsize="70%" id="S4.E1.m1.3.3.1.1.3.3.8" xref="S4.E1.m1.3.3.1.1.3.4.cmml">,</mo><mi mathsize="70%" id="S4.E1.m1.2.2" xref="S4.E1.m1.2.2.cmml">𝐲</mi><mo maxsize="70%" minsize="70%" id="S4.E1.m1.3.3.1.1.3.3.9" xref="S4.E1.m1.3.3.1.1.3.4.cmml">]</mo></mrow></mrow><mo mathsize="70%" id="S4.E1.m1.3.3.1.2" xref="S4.E1.m1.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.E1.m1.3b"><apply id="S4.E1.m1.3.3.1.1.cmml" xref="S4.E1.m1.3.3.1"><eq id="S4.E1.m1.3.3.1.1.4.cmml" xref="S4.E1.m1.3.3.1.1.4"></eq><apply id="S4.E1.m1.3.3.1.1.5.cmml" xref="S4.E1.m1.3.3.1.1.5"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.1.1.5.1.cmml" xref="S4.E1.m1.3.3.1.1.5">superscript</csymbol><ci id="S4.E1.m1.3.3.1.1.5.2.cmml" xref="S4.E1.m1.3.3.1.1.5.2">𝐲</ci><ci id="S4.E1.m1.3.3.1.1.5.3.cmml" xref="S4.E1.m1.3.3.1.1.5.3">′</ci></apply><list id="S4.E1.m1.3.3.1.1.3.4.cmml" xref="S4.E1.m1.3.3.1.1.3.3"><apply id="S4.E1.m1.3.3.1.1.1.1.1.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.1.1.1.1.1.1.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1">subscript</csymbol><ci id="S4.E1.m1.3.3.1.1.1.1.1.2.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1.2">𝐬</ci><cn type="integer" id="S4.E1.m1.3.3.1.1.1.1.1.3.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1.3">1</cn></apply><apply id="S4.E1.m1.3.3.1.1.2.2.2.cmml" xref="S4.E1.m1.3.3.1.1.2.2.2"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.1.1.2.2.2.1.cmml" xref="S4.E1.m1.3.3.1.1.2.2.2">subscript</csymbol><ci id="S4.E1.m1.3.3.1.1.2.2.2.2.cmml" xref="S4.E1.m1.3.3.1.1.2.2.2.2">𝐬</ci><cn type="integer" id="S4.E1.m1.3.3.1.1.2.2.2.3.cmml" xref="S4.E1.m1.3.3.1.1.2.2.2.3">2</cn></apply><ci id="S4.E1.m1.1.1.cmml" xref="S4.E1.m1.1.1">…</ci><apply id="S4.E1.m1.3.3.1.1.3.3.3.cmml" xref="S4.E1.m1.3.3.1.1.3.3.3"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.1.1.3.3.3.1.cmml" xref="S4.E1.m1.3.3.1.1.3.3.3">subscript</csymbol><ci id="S4.E1.m1.3.3.1.1.3.3.3.2.cmml" xref="S4.E1.m1.3.3.1.1.3.3.3.2">𝐬</ci><ci id="S4.E1.m1.3.3.1.1.3.3.3.3.cmml" xref="S4.E1.m1.3.3.1.1.3.3.3.3">𝑚</ci></apply><ci id="S4.E1.m1.2.2.cmml" xref="S4.E1.m1.2.2">𝐲</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E1.m1.3c">\displaystyle\mathbf{y}^{\prime}=[\mathbf{s}_{1},\mathbf{s}_{2},\ldots,\mathbf{s}_{m},\mathbf{y}],</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4.1)</span></td>
</tr></tbody>
</table>
<p id="S4.SS1.SSS1.p1.5" class="ltx_p"><span id="S4.SS1.SSS1.p1.5.1" class="ltx_text" style="font-size:70%;">where </span><math id="S4.SS1.SSS1.p1.4.m1.3" class="ltx_Math" alttext="\mathbf{s}_{1},\ldots,\mathbf{s}_{m}" display="inline"><semantics id="S4.SS1.SSS1.p1.4.m1.3a"><mrow id="S4.SS1.SSS1.p1.4.m1.3.3.2" xref="S4.SS1.SSS1.p1.4.m1.3.3.3.cmml"><msub id="S4.SS1.SSS1.p1.4.m1.2.2.1.1" xref="S4.SS1.SSS1.p1.4.m1.2.2.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p1.4.m1.2.2.1.1.2" xref="S4.SS1.SSS1.p1.4.m1.2.2.1.1.2.cmml">𝐬</mi><mn mathsize="70%" id="S4.SS1.SSS1.p1.4.m1.2.2.1.1.3" xref="S4.SS1.SSS1.p1.4.m1.2.2.1.1.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.SS1.SSS1.p1.4.m1.3.3.2.3" xref="S4.SS1.SSS1.p1.4.m1.3.3.3.cmml">,</mo><mi mathsize="70%" mathvariant="normal" id="S4.SS1.SSS1.p1.4.m1.1.1" xref="S4.SS1.SSS1.p1.4.m1.1.1.cmml">…</mi><mo mathsize="70%" id="S4.SS1.SSS1.p1.4.m1.3.3.2.4" xref="S4.SS1.SSS1.p1.4.m1.3.3.3.cmml">,</mo><msub id="S4.SS1.SSS1.p1.4.m1.3.3.2.2" xref="S4.SS1.SSS1.p1.4.m1.3.3.2.2.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p1.4.m1.3.3.2.2.2" xref="S4.SS1.SSS1.p1.4.m1.3.3.2.2.2.cmml">𝐬</mi><mi mathsize="70%" id="S4.SS1.SSS1.p1.4.m1.3.3.2.2.3" xref="S4.SS1.SSS1.p1.4.m1.3.3.2.2.3.cmml">m</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS1.p1.4.m1.3b"><list id="S4.SS1.SSS1.p1.4.m1.3.3.3.cmml" xref="S4.SS1.SSS1.p1.4.m1.3.3.2"><apply id="S4.SS1.SSS1.p1.4.m1.2.2.1.1.cmml" xref="S4.SS1.SSS1.p1.4.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p1.4.m1.2.2.1.1.1.cmml" xref="S4.SS1.SSS1.p1.4.m1.2.2.1.1">subscript</csymbol><ci id="S4.SS1.SSS1.p1.4.m1.2.2.1.1.2.cmml" xref="S4.SS1.SSS1.p1.4.m1.2.2.1.1.2">𝐬</ci><cn type="integer" id="S4.SS1.SSS1.p1.4.m1.2.2.1.1.3.cmml" xref="S4.SS1.SSS1.p1.4.m1.2.2.1.1.3">1</cn></apply><ci id="S4.SS1.SSS1.p1.4.m1.1.1.cmml" xref="S4.SS1.SSS1.p1.4.m1.1.1">…</ci><apply id="S4.SS1.SSS1.p1.4.m1.3.3.2.2.cmml" xref="S4.SS1.SSS1.p1.4.m1.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p1.4.m1.3.3.2.2.1.cmml" xref="S4.SS1.SSS1.p1.4.m1.3.3.2.2">subscript</csymbol><ci id="S4.SS1.SSS1.p1.4.m1.3.3.2.2.2.cmml" xref="S4.SS1.SSS1.p1.4.m1.3.3.2.2.2">𝐬</ci><ci id="S4.SS1.SSS1.p1.4.m1.3.3.2.2.3.cmml" xref="S4.SS1.SSS1.p1.4.m1.3.3.2.2.3">𝑚</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS1.p1.4.m1.3c">\mathbf{s}_{1},\ldots,\mathbf{s}_{m}</annotation></semantics></math><span id="S4.SS1.SSS1.p1.5.2" class="ltx_text" style="font-size:70%;"> are intermediate CoT steps that progressively lead to the final answer </span><math id="S4.SS1.SSS1.p1.5.m2.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.SS1.SSS1.p1.5.m2.1a"><mi mathsize="70%" id="S4.SS1.SSS1.p1.5.m2.1.1" xref="S4.SS1.SSS1.p1.5.m2.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS1.p1.5.m2.1b"><ci id="S4.SS1.SSS1.p1.5.m2.1.1.cmml" xref="S4.SS1.SSS1.p1.5.m2.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS1.p1.5.m2.1c">\mathbf{y}</annotation></semantics></math><span id="S4.SS1.SSS1.p1.5.3" class="ltx_text" style="font-size:70%;">. In essence, CoT consists of the following primary phases.</span></p>
</div>
<div id="S4.SS1.SSS1.p2" class="ltx_para ltx_noindent">
<svg id="S4.SS1.SSS1.p2.pic1" class="ltx_picture" height="87.22" overflow="visible" version="1.1" width="600"><g transform="translate(0,87.22) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#FCB864" fill-opacity="1.0"><path d="M 0 5.91 L 0 81.32 C 0 84.58 2.64 87.22 5.91 87.22 L 594.09 87.22 C 597.36 87.22 600 84.58 600 81.32 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#FFFBF7" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 68.79 L 598.03 68.79 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 74.7)"><foreignObject width="556.69" height="6.62" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S4.SS1.SSS1.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S4.SS1.SSS1.p2.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S4.SS1.SSS1.p2.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">CoT</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="43.2" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S4.SS1.SSS1.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S4.I1" class="ltx_itemize">
<span id="S4.I1.i1" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I1.i1.p1" class="ltx_para">
<span id="S4.I1.i1.p1.3" class="ltx_p"><span id="S4.I1.i1.p1.3.1" class="ltx_text" style="font-size:70%;">Find an instruction </span><math id="S4.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{z}_{c}" display="inline"><semantics id="S4.I1.i1.p1.1.m1.1a"><msub id="S4.I1.i1.p1.1.m1.1.1" xref="S4.I1.i1.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.I1.i1.p1.1.m1.1.1.2" xref="S4.I1.i1.p1.1.m1.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.I1.i1.p1.1.m1.1.1.3" xref="S4.I1.i1.p1.1.m1.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S4.I1.i1.p1.1.m1.1b"><apply id="S4.I1.i1.p1.1.m1.1.1.cmml" xref="S4.I1.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.I1.i1.p1.1.m1.1.1.1.cmml" xref="S4.I1.i1.p1.1.m1.1.1">subscript</csymbol><ci id="S4.I1.i1.p1.1.m1.1.1.2.cmml" xref="S4.I1.i1.p1.1.m1.1.1.2">𝐳</ci><ci id="S4.I1.i1.p1.1.m1.1.1.3.cmml" xref="S4.I1.i1.p1.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i1.p1.1.m1.1c">\mathbf{z}_{c}</annotation></semantics></math><span id="S4.I1.i1.p1.3.2" class="ltx_text" style="font-size:70%;">, and generate prompt </span><math id="S4.I1.i1.p1.2.m2.2" class="ltx_Math" alttext="\widetilde{\mathbf{x}}=[\mathbf{x},\mathbf{z}_{c}]" display="inline"><semantics id="S4.I1.i1.p1.2.m2.2a"><mrow id="S4.I1.i1.p1.2.m2.2.2" xref="S4.I1.i1.p1.2.m2.2.2.cmml"><mover accent="true" id="S4.I1.i1.p1.2.m2.2.2.3" xref="S4.I1.i1.p1.2.m2.2.2.3.cmml"><mi mathsize="70%" id="S4.I1.i1.p1.2.m2.2.2.3.2" xref="S4.I1.i1.p1.2.m2.2.2.3.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.I1.i1.p1.2.m2.2.2.3.1" xref="S4.I1.i1.p1.2.m2.2.2.3.1.cmml">~</mo></mover><mo mathsize="70%" id="S4.I1.i1.p1.2.m2.2.2.2" xref="S4.I1.i1.p1.2.m2.2.2.2.cmml">=</mo><mrow id="S4.I1.i1.p1.2.m2.2.2.1.1" xref="S4.I1.i1.p1.2.m2.2.2.1.2.cmml"><mo maxsize="70%" minsize="70%" id="S4.I1.i1.p1.2.m2.2.2.1.1.2" xref="S4.I1.i1.p1.2.m2.2.2.1.2.cmml">[</mo><mi mathsize="70%" id="S4.I1.i1.p1.2.m2.1.1" xref="S4.I1.i1.p1.2.m2.1.1.cmml">𝐱</mi><mo mathsize="70%" id="S4.I1.i1.p1.2.m2.2.2.1.1.3" xref="S4.I1.i1.p1.2.m2.2.2.1.2.cmml">,</mo><msub id="S4.I1.i1.p1.2.m2.2.2.1.1.1" xref="S4.I1.i1.p1.2.m2.2.2.1.1.1.cmml"><mi mathsize="70%" id="S4.I1.i1.p1.2.m2.2.2.1.1.1.2" xref="S4.I1.i1.p1.2.m2.2.2.1.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.I1.i1.p1.2.m2.2.2.1.1.1.3" xref="S4.I1.i1.p1.2.m2.2.2.1.1.1.3.cmml">c</mi></msub><mo maxsize="70%" minsize="70%" id="S4.I1.i1.p1.2.m2.2.2.1.1.4" xref="S4.I1.i1.p1.2.m2.2.2.1.2.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.i1.p1.2.m2.2b"><apply id="S4.I1.i1.p1.2.m2.2.2.cmml" xref="S4.I1.i1.p1.2.m2.2.2"><eq id="S4.I1.i1.p1.2.m2.2.2.2.cmml" xref="S4.I1.i1.p1.2.m2.2.2.2"></eq><apply id="S4.I1.i1.p1.2.m2.2.2.3.cmml" xref="S4.I1.i1.p1.2.m2.2.2.3"><ci id="S4.I1.i1.p1.2.m2.2.2.3.1.cmml" xref="S4.I1.i1.p1.2.m2.2.2.3.1">~</ci><ci id="S4.I1.i1.p1.2.m2.2.2.3.2.cmml" xref="S4.I1.i1.p1.2.m2.2.2.3.2">𝐱</ci></apply><interval closure="closed" id="S4.I1.i1.p1.2.m2.2.2.1.2.cmml" xref="S4.I1.i1.p1.2.m2.2.2.1.1"><ci id="S4.I1.i1.p1.2.m2.1.1.cmml" xref="S4.I1.i1.p1.2.m2.1.1">𝐱</ci><apply id="S4.I1.i1.p1.2.m2.2.2.1.1.1.cmml" xref="S4.I1.i1.p1.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S4.I1.i1.p1.2.m2.2.2.1.1.1.1.cmml" xref="S4.I1.i1.p1.2.m2.2.2.1.1.1">subscript</csymbol><ci id="S4.I1.i1.p1.2.m2.2.2.1.1.1.2.cmml" xref="S4.I1.i1.p1.2.m2.2.2.1.1.1.2">𝐳</ci><ci id="S4.I1.i1.p1.2.m2.2.2.1.1.1.3.cmml" xref="S4.I1.i1.p1.2.m2.2.2.1.1.1.3">𝑐</ci></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i1.p1.2.m2.2c">\widetilde{\mathbf{x}}=[\mathbf{x},\mathbf{z}_{c}]</annotation></semantics></math><span id="S4.I1.i1.p1.3.3" class="ltx_text" style="font-size:70%;"> for zero-shot CoT or </span><math id="S4.I1.i1.p1.3.m3.2" class="ltx_Math" alttext="\widetilde{\mathbf{x}}=[\mathbf{z}_{c},\mathbf{x}]" display="inline"><semantics id="S4.I1.i1.p1.3.m3.2a"><mrow id="S4.I1.i1.p1.3.m3.2.2" xref="S4.I1.i1.p1.3.m3.2.2.cmml"><mover accent="true" id="S4.I1.i1.p1.3.m3.2.2.3" xref="S4.I1.i1.p1.3.m3.2.2.3.cmml"><mi mathsize="70%" id="S4.I1.i1.p1.3.m3.2.2.3.2" xref="S4.I1.i1.p1.3.m3.2.2.3.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.I1.i1.p1.3.m3.2.2.3.1" xref="S4.I1.i1.p1.3.m3.2.2.3.1.cmml">~</mo></mover><mo mathsize="70%" id="S4.I1.i1.p1.3.m3.2.2.2" xref="S4.I1.i1.p1.3.m3.2.2.2.cmml">=</mo><mrow id="S4.I1.i1.p1.3.m3.2.2.1.1" xref="S4.I1.i1.p1.3.m3.2.2.1.2.cmml"><mo maxsize="70%" minsize="70%" id="S4.I1.i1.p1.3.m3.2.2.1.1.2" xref="S4.I1.i1.p1.3.m3.2.2.1.2.cmml">[</mo><msub id="S4.I1.i1.p1.3.m3.2.2.1.1.1" xref="S4.I1.i1.p1.3.m3.2.2.1.1.1.cmml"><mi mathsize="70%" id="S4.I1.i1.p1.3.m3.2.2.1.1.1.2" xref="S4.I1.i1.p1.3.m3.2.2.1.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.I1.i1.p1.3.m3.2.2.1.1.1.3" xref="S4.I1.i1.p1.3.m3.2.2.1.1.1.3.cmml">c</mi></msub><mo mathsize="70%" id="S4.I1.i1.p1.3.m3.2.2.1.1.3" xref="S4.I1.i1.p1.3.m3.2.2.1.2.cmml">,</mo><mi mathsize="70%" id="S4.I1.i1.p1.3.m3.1.1" xref="S4.I1.i1.p1.3.m3.1.1.cmml">𝐱</mi><mo maxsize="70%" minsize="70%" id="S4.I1.i1.p1.3.m3.2.2.1.1.4" xref="S4.I1.i1.p1.3.m3.2.2.1.2.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.i1.p1.3.m3.2b"><apply id="S4.I1.i1.p1.3.m3.2.2.cmml" xref="S4.I1.i1.p1.3.m3.2.2"><eq id="S4.I1.i1.p1.3.m3.2.2.2.cmml" xref="S4.I1.i1.p1.3.m3.2.2.2"></eq><apply id="S4.I1.i1.p1.3.m3.2.2.3.cmml" xref="S4.I1.i1.p1.3.m3.2.2.3"><ci id="S4.I1.i1.p1.3.m3.2.2.3.1.cmml" xref="S4.I1.i1.p1.3.m3.2.2.3.1">~</ci><ci id="S4.I1.i1.p1.3.m3.2.2.3.2.cmml" xref="S4.I1.i1.p1.3.m3.2.2.3.2">𝐱</ci></apply><interval closure="closed" id="S4.I1.i1.p1.3.m3.2.2.1.2.cmml" xref="S4.I1.i1.p1.3.m3.2.2.1.1"><apply id="S4.I1.i1.p1.3.m3.2.2.1.1.1.cmml" xref="S4.I1.i1.p1.3.m3.2.2.1.1.1"><csymbol cd="ambiguous" id="S4.I1.i1.p1.3.m3.2.2.1.1.1.1.cmml" xref="S4.I1.i1.p1.3.m3.2.2.1.1.1">subscript</csymbol><ci id="S4.I1.i1.p1.3.m3.2.2.1.1.1.2.cmml" xref="S4.I1.i1.p1.3.m3.2.2.1.1.1.2">𝐳</ci><ci id="S4.I1.i1.p1.3.m3.2.2.1.1.1.3.cmml" xref="S4.I1.i1.p1.3.m3.2.2.1.1.1.3">𝑐</ci></apply><ci id="S4.I1.i1.p1.3.m3.1.1.cmml" xref="S4.I1.i1.p1.3.m3.1.1">𝐱</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i1.p1.3.m3.2c">\widetilde{\mathbf{x}}=[\mathbf{z}_{c},\mathbf{x}]</annotation></semantics></math><span id="S4.I1.i1.p1.3.4" class="ltx_text" style="font-size:70%;"> for few-shot CoT.</span></span>
</span></span>
<span id="S4.I1.i2" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I1.i2.p1" class="ltx_para">
<span id="S4.I1.i2.p1.4" class="ltx_p"><span id="S4.I1.i2.p1.4.1" class="ltx_text" style="font-size:70%;">Generate </span><math id="S4.I1.i2.p1.1.m1.1" class="ltx_math_unparsed" alttext="\mathbf{y}^{\prime}\sim p_{\text{LLM}}(\cdot|\widetilde{\mathbf{x}})" display="inline"><semantics id="S4.I1.i2.p1.1.m1.1a"><mrow id="S4.I1.i2.p1.1.m1.1b"><msup id="S4.I1.i2.p1.1.m1.1.1"><mi mathsize="70%" id="S4.I1.i2.p1.1.m1.1.1.2">𝐲</mi><mo mathsize="70%" id="S4.I1.i2.p1.1.m1.1.1.3">′</mo></msup><mo mathsize="70%" id="S4.I1.i2.p1.1.m1.1.2">∼</mo><msub id="S4.I1.i2.p1.1.m1.1.3"><mi mathsize="70%" id="S4.I1.i2.p1.1.m1.1.3.2">p</mi><mtext mathsize="70%" id="S4.I1.i2.p1.1.m1.1.3.3">LLM</mtext></msub><mrow id="S4.I1.i2.p1.1.m1.1.4"><mo maxsize="70%" minsize="70%" id="S4.I1.i2.p1.1.m1.1.4.1">(</mo><mo lspace="0em" mathsize="70%" rspace="0em" id="S4.I1.i2.p1.1.m1.1.4.2">⋅</mo><mo fence="false" maxsize="70%" minsize="70%" rspace="0.167em" id="S4.I1.i2.p1.1.m1.1.4.3">|</mo><mover accent="true" id="S4.I1.i2.p1.1.m1.1.4.4"><mi mathsize="70%" id="S4.I1.i2.p1.1.m1.1.4.4.2">𝐱</mi><mo mathsize="70%" id="S4.I1.i2.p1.1.m1.1.4.4.1">~</mo></mover><mo maxsize="70%" minsize="70%" id="S4.I1.i2.p1.1.m1.1.4.5">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.I1.i2.p1.1.m1.1c">\mathbf{y}^{\prime}\sim p_{\text{LLM}}(\cdot|\widetilde{\mathbf{x}})</annotation></semantics></math><span id="S4.I1.i2.p1.4.2" class="ltx_text" style="font-size:70%;"> following the “step by step” format of (</span><a href="#S4.E1" title="Equation 4.1 ‣ 4.1.1 Chain-of-Thought (CoT) ‣ 4.1 Mathematical Formulation ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">4.1</span></a><span id="S4.I1.i2.p1.4.3" class="ltx_text" style="font-size:70%;">). </span><math id="S4.I1.i2.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.I1.i2.p1.2.m2.1a"><msup id="S4.I1.i2.p1.2.m2.1.1" xref="S4.I1.i2.p1.2.m2.1.1.cmml"><mi mathsize="70%" id="S4.I1.i2.p1.2.m2.1.1.2" xref="S4.I1.i2.p1.2.m2.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.I1.i2.p1.2.m2.1.1.3" xref="S4.I1.i2.p1.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.2.m2.1b"><apply id="S4.I1.i2.p1.2.m2.1.1.cmml" xref="S4.I1.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.2.m2.1.1.1.cmml" xref="S4.I1.i2.p1.2.m2.1.1">superscript</csymbol><ci id="S4.I1.i2.p1.2.m2.1.1.2.cmml" xref="S4.I1.i2.p1.2.m2.1.1.2">𝐲</ci><ci id="S4.I1.i2.p1.2.m2.1.1.3.cmml" xref="S4.I1.i2.p1.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.2.m2.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.I1.i2.p1.4.4" class="ltx_text" style="font-size:70%;"> contains the intermediate CoT steps </span><math id="S4.I1.i2.p1.3.m3.2" class="ltx_Math" alttext="\mathbf{s}_{1},\ldots\mathbf{s}_{k}" display="inline"><semantics id="S4.I1.i2.p1.3.m3.2a"><mrow id="S4.I1.i2.p1.3.m3.2.2.2" xref="S4.I1.i2.p1.3.m3.2.2.3.cmml"><msub id="S4.I1.i2.p1.3.m3.1.1.1.1" xref="S4.I1.i2.p1.3.m3.1.1.1.1.cmml"><mi mathsize="70%" id="S4.I1.i2.p1.3.m3.1.1.1.1.2" xref="S4.I1.i2.p1.3.m3.1.1.1.1.2.cmml">𝐬</mi><mn mathsize="70%" id="S4.I1.i2.p1.3.m3.1.1.1.1.3" xref="S4.I1.i2.p1.3.m3.1.1.1.1.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.I1.i2.p1.3.m3.2.2.2.3" xref="S4.I1.i2.p1.3.m3.2.2.3.cmml">,</mo><mrow id="S4.I1.i2.p1.3.m3.2.2.2.2" xref="S4.I1.i2.p1.3.m3.2.2.2.2.cmml"><mi mathsize="70%" mathvariant="normal" id="S4.I1.i2.p1.3.m3.2.2.2.2.2" xref="S4.I1.i2.p1.3.m3.2.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="S4.I1.i2.p1.3.m3.2.2.2.2.1" xref="S4.I1.i2.p1.3.m3.2.2.2.2.1.cmml">​</mo><msub id="S4.I1.i2.p1.3.m3.2.2.2.2.3" xref="S4.I1.i2.p1.3.m3.2.2.2.2.3.cmml"><mi mathsize="70%" id="S4.I1.i2.p1.3.m3.2.2.2.2.3.2" xref="S4.I1.i2.p1.3.m3.2.2.2.2.3.2.cmml">𝐬</mi><mi mathsize="70%" id="S4.I1.i2.p1.3.m3.2.2.2.2.3.3" xref="S4.I1.i2.p1.3.m3.2.2.2.2.3.3.cmml">k</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.3.m3.2b"><list id="S4.I1.i2.p1.3.m3.2.2.3.cmml" xref="S4.I1.i2.p1.3.m3.2.2.2"><apply id="S4.I1.i2.p1.3.m3.1.1.1.1.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.3.m3.1.1.1.1.1.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1">subscript</csymbol><ci id="S4.I1.i2.p1.3.m3.1.1.1.1.2.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.2">𝐬</ci><cn type="integer" id="S4.I1.i2.p1.3.m3.1.1.1.1.3.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.3">1</cn></apply><apply id="S4.I1.i2.p1.3.m3.2.2.2.2.cmml" xref="S4.I1.i2.p1.3.m3.2.2.2.2"><times id="S4.I1.i2.p1.3.m3.2.2.2.2.1.cmml" xref="S4.I1.i2.p1.3.m3.2.2.2.2.1"></times><ci id="S4.I1.i2.p1.3.m3.2.2.2.2.2.cmml" xref="S4.I1.i2.p1.3.m3.2.2.2.2.2">…</ci><apply id="S4.I1.i2.p1.3.m3.2.2.2.2.3.cmml" xref="S4.I1.i2.p1.3.m3.2.2.2.2.3"><csymbol cd="ambiguous" id="S4.I1.i2.p1.3.m3.2.2.2.2.3.1.cmml" xref="S4.I1.i2.p1.3.m3.2.2.2.2.3">subscript</csymbol><ci id="S4.I1.i2.p1.3.m3.2.2.2.2.3.2.cmml" xref="S4.I1.i2.p1.3.m3.2.2.2.2.3.2">𝐬</ci><ci id="S4.I1.i2.p1.3.m3.2.2.2.2.3.3.cmml" xref="S4.I1.i2.p1.3.m3.2.2.2.2.3.3">𝑘</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.3.m3.2c">\mathbf{s}_{1},\ldots\mathbf{s}_{k}</annotation></semantics></math><span id="S4.I1.i2.p1.4.5" class="ltx_text" style="font-size:70%;">, and the desired question </span><math id="S4.I1.i2.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.I1.i2.p1.4.m4.1a"><mi mathsize="70%" id="S4.I1.i2.p1.4.m4.1.1" xref="S4.I1.i2.p1.4.m4.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.4.m4.1b"><ci id="S4.I1.i2.p1.4.m4.1.1.cmml" xref="S4.I1.i2.p1.4.m4.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.4.m4.1c">\mathbf{y}</annotation></semantics></math><span id="S4.I1.i2.p1.4.6" class="ltx_text" style="font-size:70%;">.</span></span>
</span></span>
<span id="S4.I1.i3" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I1.i3.p1" class="ltx_para">
<span id="S4.I1.i3.p1.2" class="ltx_p"><span id="S4.I1.i3.p1.2.1" class="ltx_text" style="font-size:70%;">Extract the desired response </span><math id="S4.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.I1.i3.p1.1.m1.1a"><mi mathsize="70%" id="S4.I1.i3.p1.1.m1.1.1" xref="S4.I1.i3.p1.1.m1.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.I1.i3.p1.1.m1.1b"><ci id="S4.I1.i3.p1.1.m1.1.1.cmml" xref="S4.I1.i3.p1.1.m1.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i3.p1.1.m1.1c">\mathbf{y}</annotation></semantics></math><span id="S4.I1.i3.p1.2.2" class="ltx_text" style="font-size:70%;"> from the sequence </span><math id="S4.I1.i3.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.I1.i3.p1.2.m2.1a"><msup id="S4.I1.i3.p1.2.m2.1.1" xref="S4.I1.i3.p1.2.m2.1.1.cmml"><mi mathsize="70%" id="S4.I1.i3.p1.2.m2.1.1.2" xref="S4.I1.i3.p1.2.m2.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.I1.i3.p1.2.m2.1.1.3" xref="S4.I1.i3.p1.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.I1.i3.p1.2.m2.1b"><apply id="S4.I1.i3.p1.2.m2.1.1.cmml" xref="S4.I1.i3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.I1.i3.p1.2.m2.1.1.1.cmml" xref="S4.I1.i3.p1.2.m2.1.1">superscript</csymbol><ci id="S4.I1.i3.p1.2.m2.1.1.2.cmml" xref="S4.I1.i3.p1.2.m2.1.1.2">𝐲</ci><ci id="S4.I1.i3.p1.2.m2.1.1.3.cmml" xref="S4.I1.i3.p1.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i3.p1.2.m2.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.I1.i3.p1.2.3" class="ltx_text" style="font-size:70%;">.</span></span>
</span></span>
</span>
</span></foreignObject></g></g></svg>
</div>
<div id="S4.SS1.SSS1.p3" class="ltx_para">
<p id="S4.SS1.SSS1.p3.5" class="ltx_p"><span id="S4.SS1.SSS1.p3.5.1" class="ltx_text" style="font-size:70%;">For zero-shot CoT, the instruction </span><math id="S4.SS1.SSS1.p3.1.m1.1" class="ltx_Math" alttext="\mathbf{z}_{c}" display="inline"><semantics id="S4.SS1.SSS1.p3.1.m1.1a"><msub id="S4.SS1.SSS1.p3.1.m1.1.1" xref="S4.SS1.SSS1.p3.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.1.m1.1.1.2" xref="S4.SS1.SSS1.p3.1.m1.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.SS1.SSS1.p3.1.m1.1.1.3" xref="S4.SS1.SSS1.p3.1.m1.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS1.p3.1.m1.1b"><apply id="S4.SS1.SSS1.p3.1.m1.1.1.cmml" xref="S4.SS1.SSS1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.1.m1.1.1.1.cmml" xref="S4.SS1.SSS1.p3.1.m1.1.1">subscript</csymbol><ci id="S4.SS1.SSS1.p3.1.m1.1.1.2.cmml" xref="S4.SS1.SSS1.p3.1.m1.1.1.2">𝐳</ci><ci id="S4.SS1.SSS1.p3.1.m1.1.1.3.cmml" xref="S4.SS1.SSS1.p3.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS1.p3.1.m1.1c">\mathbf{z}_{c}</annotation></semantics></math><span id="S4.SS1.SSS1.p3.5.2" class="ltx_text" style="font-size:70%;"> is composed of some task-independent tokens, such as “Let’s think step by step”. For few-shot CoT, the instruction/context </span><math id="S4.SS1.SSS1.p3.2.m2.1" class="ltx_Math" alttext="\mathbf{z}_{c}" display="inline"><semantics id="S4.SS1.SSS1.p3.2.m2.1a"><msub id="S4.SS1.SSS1.p3.2.m2.1.1" xref="S4.SS1.SSS1.p3.2.m2.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.2.m2.1.1.2" xref="S4.SS1.SSS1.p3.2.m2.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.SS1.SSS1.p3.2.m2.1.1.3" xref="S4.SS1.SSS1.p3.2.m2.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS1.p3.2.m2.1b"><apply id="S4.SS1.SSS1.p3.2.m2.1.1.cmml" xref="S4.SS1.SSS1.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.2.m2.1.1.1.cmml" xref="S4.SS1.SSS1.p3.2.m2.1.1">subscript</csymbol><ci id="S4.SS1.SSS1.p3.2.m2.1.1.2.cmml" xref="S4.SS1.SSS1.p3.2.m2.1.1.2">𝐳</ci><ci id="S4.SS1.SSS1.p3.2.m2.1.1.3.cmml" xref="S4.SS1.SSS1.p3.2.m2.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS1.p3.2.m2.1c">\mathbf{z}_{c}</annotation></semantics></math><span id="S4.SS1.SSS1.p3.5.3" class="ltx_text" style="font-size:70%;"> consists of some task-dependent tokens, which include several examples like </span><math id="S4.SS1.SSS1.p3.3.m3.7" class="ltx_Math" alttext="\mathbf{z}_{c}=[\mathbf{x}_{1},\mathbf{y}_{1}^{\prime},\mathbf{x}_{2},\mathbf{y}_{2}^{\prime},\ldots,\mathbf{x}_{k},\mathbf{y}_{k}^{\prime}]" display="inline"><semantics id="S4.SS1.SSS1.p3.3.m3.7a"><mrow id="S4.SS1.SSS1.p3.3.m3.7.7" xref="S4.SS1.SSS1.p3.3.m3.7.7.cmml"><msub id="S4.SS1.SSS1.p3.3.m3.7.7.8" xref="S4.SS1.SSS1.p3.3.m3.7.7.8.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.8.2" xref="S4.SS1.SSS1.p3.3.m3.7.7.8.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.8.3" xref="S4.SS1.SSS1.p3.3.m3.7.7.8.3.cmml">c</mi></msub><mo mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.7" xref="S4.SS1.SSS1.p3.3.m3.7.7.7.cmml">=</mo><mrow id="S4.SS1.SSS1.p3.3.m3.7.7.6.6" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.7.cmml"><mo maxsize="70%" minsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.7" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.7.cmml">[</mo><msub id="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1" xref="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.2" xref="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.3" xref="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.8" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.7.cmml">,</mo><msubsup id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.2.2" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.2.3" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.2.3.cmml">1</mn><mo mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.3" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.9" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.7.cmml">,</mo><msub id="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3" xref="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.2" xref="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.3" xref="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.3.cmml">2</mn></msub><mo mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.10" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.7.cmml">,</mo><msubsup id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.2.2" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.2.3" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.2.3.cmml">2</mn><mo mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.3" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.11" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.7.cmml">,</mo><mi mathsize="70%" mathvariant="normal" id="S4.SS1.SSS1.p3.3.m3.1.1" xref="S4.SS1.SSS1.p3.3.m3.1.1.cmml">…</mi><mo mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.12" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.7.cmml">,</mo><msub id="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5" xref="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.2" xref="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.2.cmml">𝐱</mi><mi mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.3" xref="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.3.cmml">k</mi></msub><mo mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.13" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.7.cmml">,</mo><msubsup id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.2.2" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.2.2.cmml">𝐲</mi><mi mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.2.3" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.2.3.cmml">k</mi><mo mathsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.3" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.3.cmml">′</mo></msubsup><mo maxsize="70%" minsize="70%" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.14" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.7.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS1.p3.3.m3.7b"><apply id="S4.SS1.SSS1.p3.3.m3.7.7.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7"><eq id="S4.SS1.SSS1.p3.3.m3.7.7.7.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.7"></eq><apply id="S4.SS1.SSS1.p3.3.m3.7.7.8.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.8"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.3.m3.7.7.8.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.8">subscript</csymbol><ci id="S4.SS1.SSS1.p3.3.m3.7.7.8.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.8.2">𝐳</ci><ci id="S4.SS1.SSS1.p3.3.m3.7.7.8.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.8.3">𝑐</ci></apply><list id="S4.SS1.SSS1.p3.3.m3.7.7.6.7.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6"><apply id="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1">subscript</csymbol><ci id="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.2">𝐱</ci><cn type="integer" id="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.2.2.1.1.1.3">1</cn></apply><apply id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2">superscript</csymbol><apply id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.2.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2">subscript</csymbol><ci id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.2.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.2.2">𝐲</ci><cn type="integer" id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.2.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.2.3">1</cn></apply><ci id="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.3.3.2.2.2.3">′</ci></apply><apply id="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3">subscript</csymbol><ci id="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.2">𝐱</ci><cn type="integer" id="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.4.4.3.3.3.3">2</cn></apply><apply id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.cmml" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4">superscript</csymbol><apply id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.2.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4">subscript</csymbol><ci id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.2.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.2.2">𝐲</ci><cn type="integer" id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.2.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.2.3">2</cn></apply><ci id="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.5.5.4.4.4.3">′</ci></apply><ci id="S4.SS1.SSS1.p3.3.m3.1.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.1.1">…</ci><apply id="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.cmml" xref="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5">subscript</csymbol><ci id="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.2">𝐱</ci><ci id="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.6.6.5.5.5.3">𝑘</ci></apply><apply id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6">superscript</csymbol><apply id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.2.1.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6">subscript</csymbol><ci id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.2.2.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.2.2">𝐲</ci><ci id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.2.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.2.3">𝑘</ci></apply><ci id="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.3.cmml" xref="S4.SS1.SSS1.p3.3.m3.7.7.6.6.6.3">′</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS1.p3.3.m3.7c">\mathbf{z}_{c}=[\mathbf{x}_{1},\mathbf{y}_{1}^{\prime},\mathbf{x}_{2},\mathbf{y}_{2}^{\prime},\ldots,\mathbf{x}_{k},\mathbf{y}_{k}^{\prime}]</annotation></semantics></math><span id="S4.SS1.SSS1.p3.5.4" class="ltx_text" style="font-size:70%;">, where </span><math id="S4.SS1.SSS1.p3.4.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.SS1.SSS1.p3.4.m4.1a"><mi mathsize="70%" id="S4.SS1.SSS1.p3.4.m4.1.1" xref="S4.SS1.SSS1.p3.4.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS1.p3.4.m4.1b"><ci id="S4.SS1.SSS1.p3.4.m4.1.1.cmml" xref="S4.SS1.SSS1.p3.4.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS1.p3.4.m4.1c">k</annotation></semantics></math><span id="S4.SS1.SSS1.p3.5.5" class="ltx_text" style="font-size:70%;"> is the number of in-context examples, i.e., </span><math id="S4.SS1.SSS1.p3.5.m5.1" class="ltx_Math" alttext="\{(\mathbf{x}_{i},\mathbf{y}^{\prime}_{i})\}_{i=1}^{k}" display="inline"><semantics id="S4.SS1.SSS1.p3.5.m5.1a"><msubsup id="S4.SS1.SSS1.p3.5.m5.1.1" xref="S4.SS1.SSS1.p3.5.m5.1.1.cmml"><mrow id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.2.cmml"><mo maxsize="70%" minsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.2" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.2.cmml">{</mo><mrow id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.3.cmml"><mo maxsize="70%" minsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.3" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.3.cmml">(</mo><msub id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.2" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.2.cmml">𝐱</mi><mi mathsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.3" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.4" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.3.cmml">,</mo><msubsup id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.2.2" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.2.2.cmml">𝐲</mi><mi mathsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.3" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.3.cmml">i</mi><mo mathsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.2.3" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.2.3.cmml">′</mo></msubsup><mo maxsize="70%" minsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.5" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.3.cmml">)</mo></mrow><mo maxsize="70%" minsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.3" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S4.SS1.SSS1.p3.5.m5.1.1.1.3" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.3.cmml"><mi mathsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.3.2" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.3.2.cmml">i</mi><mo mathsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.3.1" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.3.1.cmml">=</mo><mn mathsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.1.3.3" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.3.3.cmml">1</mn></mrow><mi mathsize="70%" id="S4.SS1.SSS1.p3.5.m5.1.1.3" xref="S4.SS1.SSS1.p3.5.m5.1.1.3.cmml">k</mi></msubsup><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS1.p3.5.m5.1b"><apply id="S4.SS1.SSS1.p3.5.m5.1.1.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.5.m5.1.1.2.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1">superscript</csymbol><apply id="S4.SS1.SSS1.p3.5.m5.1.1.1.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.5.m5.1.1.1.2.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1">subscript</csymbol><set id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.2.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1"><interval closure="open" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.3.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2"><apply id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.1.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.2.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.2">𝐱</ci><ci id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.3.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.1.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2">subscript</csymbol><apply id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.2.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.2.1.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2">superscript</csymbol><ci id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.2.2.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.2.2">𝐲</ci><ci id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.2.3.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.2.3">′</ci></apply><ci id="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.3.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.1.1.1.2.2.3">𝑖</ci></apply></interval></set><apply id="S4.SS1.SSS1.p3.5.m5.1.1.1.3.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.3"><eq id="S4.SS1.SSS1.p3.5.m5.1.1.1.3.1.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.3.1"></eq><ci id="S4.SS1.SSS1.p3.5.m5.1.1.1.3.2.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.3.2">𝑖</ci><cn type="integer" id="S4.SS1.SSS1.p3.5.m5.1.1.1.3.3.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.1.3.3">1</cn></apply></apply><ci id="S4.SS1.SSS1.p3.5.m5.1.1.3.cmml" xref="S4.SS1.SSS1.p3.5.m5.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS1.p3.5.m5.1c">\{(\mathbf{x}_{i},\mathbf{y}^{\prime}_{i})\}_{i=1}^{k}</annotation></semantics></math><span id="S4.SS1.SSS1.p3.5.6" class="ltx_text" style="font-size:70%;">. We give examples of zero-shot CoT and few-shot CoT as follows.</span></p>
</div>
<div id="S4.Thmexmp1" class="ltx_theorem ltx_theorem_exmp">
<h6 class="ltx_title ltx_runin ltx_title_theorem" style="font-size:70%;">
<span class="ltx_tag ltx_tag_theorem"><span id="S4.Thmexmp1.1.1.1" class="ltx_text ltx_font_bold">Example 4.1</span></span><span id="S4.Thmexmp1.2.2" class="ltx_text ltx_font_bold"> </span>(Zero-shot CoT)<span id="S4.Thmexmp1.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="S4.Thmexmp1.p1" class="ltx_para">
<p id="S4.Thmexmp1.p1.3" class="ltx_p"><span id="S4.Thmexmp1.p1.3.1" class="ltx_text" style="font-size:70%;">Consider </span><math id="S4.Thmexmp1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}=\texttt{``Was Barack Obama born in an even year?''}" display="inline"><semantics id="S4.Thmexmp1.p1.1.m1.1a"><mrow id="S4.Thmexmp1.p1.1.m1.1.1" xref="S4.Thmexmp1.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp1.p1.1.m1.1.1.2" xref="S4.Thmexmp1.p1.1.m1.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.Thmexmp1.p1.1.m1.1.1.1" xref="S4.Thmexmp1.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp1.p1.1.m1.1.1.3" xref="S4.Thmexmp1.p1.1.m1.1.1.3a.cmml">‘‘Was Barack Obama born in an even year?’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp1.p1.1.m1.1b"><apply id="S4.Thmexmp1.p1.1.m1.1.1.cmml" xref="S4.Thmexmp1.p1.1.m1.1.1"><eq id="S4.Thmexmp1.p1.1.m1.1.1.1.cmml" xref="S4.Thmexmp1.p1.1.m1.1.1.1"></eq><ci id="S4.Thmexmp1.p1.1.m1.1.1.2.cmml" xref="S4.Thmexmp1.p1.1.m1.1.1.2">𝐱</ci><ci id="S4.Thmexmp1.p1.1.m1.1.1.3a.cmml" xref="S4.Thmexmp1.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp1.p1.1.m1.1.1.3.cmml" xref="S4.Thmexmp1.p1.1.m1.1.1.3">‘‘Was Barack Obama born in an even year?’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp1.p1.1.m1.1c">\mathbf{x}=\texttt{``Was Barack Obama born in an even year?''}</annotation></semantics></math><span id="S4.Thmexmp1.p1.3.2" class="ltx_text" style="font-size:70%;">, for zero-shot CoT, we have </span><math id="S4.Thmexmp1.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{z}_{c}=\texttt{``Let's think step by step.''}" display="inline"><semantics id="S4.Thmexmp1.p1.2.m2.1a"><mrow id="S4.Thmexmp1.p1.2.m2.1.1" xref="S4.Thmexmp1.p1.2.m2.1.1.cmml"><msub id="S4.Thmexmp1.p1.2.m2.1.1.2" xref="S4.Thmexmp1.p1.2.m2.1.1.2.cmml"><mi mathsize="70%" id="S4.Thmexmp1.p1.2.m2.1.1.2.2" xref="S4.Thmexmp1.p1.2.m2.1.1.2.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.Thmexmp1.p1.2.m2.1.1.2.3" xref="S4.Thmexmp1.p1.2.m2.1.1.2.3.cmml">c</mi></msub><mo mathsize="70%" id="S4.Thmexmp1.p1.2.m2.1.1.1" xref="S4.Thmexmp1.p1.2.m2.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp1.p1.2.m2.1.1.3" xref="S4.Thmexmp1.p1.2.m2.1.1.3a.cmml">‘‘Let’s think step by step.’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp1.p1.2.m2.1b"><apply id="S4.Thmexmp1.p1.2.m2.1.1.cmml" xref="S4.Thmexmp1.p1.2.m2.1.1"><eq id="S4.Thmexmp1.p1.2.m2.1.1.1.cmml" xref="S4.Thmexmp1.p1.2.m2.1.1.1"></eq><apply id="S4.Thmexmp1.p1.2.m2.1.1.2.cmml" xref="S4.Thmexmp1.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S4.Thmexmp1.p1.2.m2.1.1.2.1.cmml" xref="S4.Thmexmp1.p1.2.m2.1.1.2">subscript</csymbol><ci id="S4.Thmexmp1.p1.2.m2.1.1.2.2.cmml" xref="S4.Thmexmp1.p1.2.m2.1.1.2.2">𝐳</ci><ci id="S4.Thmexmp1.p1.2.m2.1.1.2.3.cmml" xref="S4.Thmexmp1.p1.2.m2.1.1.2.3">𝑐</ci></apply><ci id="S4.Thmexmp1.p1.2.m2.1.1.3a.cmml" xref="S4.Thmexmp1.p1.2.m2.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp1.p1.2.m2.1.1.3.cmml" xref="S4.Thmexmp1.p1.2.m2.1.1.3">‘‘Let’s think step by step.’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp1.p1.2.m2.1c">\mathbf{z}_{c}=\texttt{``Let's think step by step.''}</annotation></semantics></math><span id="S4.Thmexmp1.p1.3.3" class="ltx_text" style="font-size:70%;"> as the effective prompt and generate </span><math id="S4.Thmexmp1.p1.3.m3.5" class="ltx_Math" alttext="\mathbf{y}^{\prime}=[\mathbf{s}_{1},\mathbf{s}_{2},\ldots,\mathbf{s}_{m},\mathbf{y}]" display="inline"><semantics id="S4.Thmexmp1.p1.3.m3.5a"><mrow id="S4.Thmexmp1.p1.3.m3.5.5" xref="S4.Thmexmp1.p1.3.m3.5.5.cmml"><msup id="S4.Thmexmp1.p1.3.m3.5.5.5" xref="S4.Thmexmp1.p1.3.m3.5.5.5.cmml"><mi mathsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.5.2" xref="S4.Thmexmp1.p1.3.m3.5.5.5.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.5.3" xref="S4.Thmexmp1.p1.3.m3.5.5.5.3.cmml">′</mo></msup><mo mathsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.4" xref="S4.Thmexmp1.p1.3.m3.5.5.4.cmml">=</mo><mrow id="S4.Thmexmp1.p1.3.m3.5.5.3.3" xref="S4.Thmexmp1.p1.3.m3.5.5.3.4.cmml"><mo maxsize="70%" minsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.3.3.4" xref="S4.Thmexmp1.p1.3.m3.5.5.3.4.cmml">[</mo><msub id="S4.Thmexmp1.p1.3.m3.3.3.1.1.1" xref="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.2" xref="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.2.cmml">𝐬</mi><mn mathsize="70%" id="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.3" xref="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.3.3.5" xref="S4.Thmexmp1.p1.3.m3.5.5.3.4.cmml">,</mo><msub id="S4.Thmexmp1.p1.3.m3.4.4.2.2.2" xref="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.cmml"><mi mathsize="70%" id="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.2" xref="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.2.cmml">𝐬</mi><mn mathsize="70%" id="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.3" xref="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.3.cmml">2</mn></msub><mo mathsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.3.3.6" xref="S4.Thmexmp1.p1.3.m3.5.5.3.4.cmml">,</mo><mi mathsize="70%" mathvariant="normal" id="S4.Thmexmp1.p1.3.m3.1.1" xref="S4.Thmexmp1.p1.3.m3.1.1.cmml">…</mi><mo mathsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.3.3.7" xref="S4.Thmexmp1.p1.3.m3.5.5.3.4.cmml">,</mo><msub id="S4.Thmexmp1.p1.3.m3.5.5.3.3.3" xref="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.cmml"><mi mathsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.2" xref="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.2.cmml">𝐬</mi><mi mathsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.3" xref="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.3.cmml">m</mi></msub><mo mathsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.3.3.8" xref="S4.Thmexmp1.p1.3.m3.5.5.3.4.cmml">,</mo><mi mathsize="70%" id="S4.Thmexmp1.p1.3.m3.2.2" xref="S4.Thmexmp1.p1.3.m3.2.2.cmml">𝐲</mi><mo maxsize="70%" minsize="70%" id="S4.Thmexmp1.p1.3.m3.5.5.3.3.9" xref="S4.Thmexmp1.p1.3.m3.5.5.3.4.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp1.p1.3.m3.5b"><apply id="S4.Thmexmp1.p1.3.m3.5.5.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5"><eq id="S4.Thmexmp1.p1.3.m3.5.5.4.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5.4"></eq><apply id="S4.Thmexmp1.p1.3.m3.5.5.5.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5.5"><csymbol cd="ambiguous" id="S4.Thmexmp1.p1.3.m3.5.5.5.1.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5.5">superscript</csymbol><ci id="S4.Thmexmp1.p1.3.m3.5.5.5.2.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5.5.2">𝐲</ci><ci id="S4.Thmexmp1.p1.3.m3.5.5.5.3.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5.5.3">′</ci></apply><list id="S4.Thmexmp1.p1.3.m3.5.5.3.4.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5.3.3"><apply id="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.cmml" xref="S4.Thmexmp1.p1.3.m3.3.3.1.1.1"><csymbol cd="ambiguous" id="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.1.cmml" xref="S4.Thmexmp1.p1.3.m3.3.3.1.1.1">subscript</csymbol><ci id="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.2.cmml" xref="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.2">𝐬</ci><cn type="integer" id="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.3.cmml" xref="S4.Thmexmp1.p1.3.m3.3.3.1.1.1.3">1</cn></apply><apply id="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.cmml" xref="S4.Thmexmp1.p1.3.m3.4.4.2.2.2"><csymbol cd="ambiguous" id="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.1.cmml" xref="S4.Thmexmp1.p1.3.m3.4.4.2.2.2">subscript</csymbol><ci id="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.2.cmml" xref="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.2">𝐬</ci><cn type="integer" id="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.3.cmml" xref="S4.Thmexmp1.p1.3.m3.4.4.2.2.2.3">2</cn></apply><ci id="S4.Thmexmp1.p1.3.m3.1.1.cmml" xref="S4.Thmexmp1.p1.3.m3.1.1">…</ci><apply id="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5.3.3.3"><csymbol cd="ambiguous" id="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.1.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5.3.3.3">subscript</csymbol><ci id="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.2.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.2">𝐬</ci><ci id="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.3.cmml" xref="S4.Thmexmp1.p1.3.m3.5.5.3.3.3.3">𝑚</ci></apply><ci id="S4.Thmexmp1.p1.3.m3.2.2.cmml" xref="S4.Thmexmp1.p1.3.m3.2.2">𝐲</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp1.p1.3.m3.5c">\mathbf{y}^{\prime}=[\mathbf{s}_{1},\mathbf{s}_{2},\ldots,\mathbf{s}_{m},\mathbf{y}]</annotation></semantics></math><span id="S4.Thmexmp1.p1.3.4" class="ltx_text" style="font-size:70%;">, where</span></p>
<ul id="S4.I2" class="ltx_itemize">
<li id="S4.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I2.i1.p1" class="ltx_para">
<p id="S4.I2.i1.p1.1" class="ltx_p"><math id="S4.I2.i1.p1.1.m1.1" class="ltx_Math" alttext="\bm{s}_{1}=\texttt{``Step 1: Research Barack Obama's birth date\ldots''}" display="inline"><semantics id="S4.I2.i1.p1.1.m1.1a"><mrow id="S4.I2.i1.p1.1.m1.1.1" xref="S4.I2.i1.p1.1.m1.1.1.cmml"><msub id="S4.I2.i1.p1.1.m1.1.1.2" xref="S4.I2.i1.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I2.i1.p1.1.m1.1.1.2.2" xref="S4.I2.i1.p1.1.m1.1.1.2.2.cmml">𝒔</mi><mn mathsize="70%" id="S4.I2.i1.p1.1.m1.1.1.2.3" xref="S4.I2.i1.p1.1.m1.1.1.2.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.I2.i1.p1.1.m1.1.1.1" xref="S4.I2.i1.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I2.i1.p1.1.m1.1.1.3" xref="S4.I2.i1.p1.1.m1.1.1.3a.cmml">‘‘Step 1: Research Barack Obama’s birth date…’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I2.i1.p1.1.m1.1b"><apply id="S4.I2.i1.p1.1.m1.1.1.cmml" xref="S4.I2.i1.p1.1.m1.1.1"><eq id="S4.I2.i1.p1.1.m1.1.1.1.cmml" xref="S4.I2.i1.p1.1.m1.1.1.1"></eq><apply id="S4.I2.i1.p1.1.m1.1.1.2.cmml" xref="S4.I2.i1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I2.i1.p1.1.m1.1.1.2.1.cmml" xref="S4.I2.i1.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I2.i1.p1.1.m1.1.1.2.2.cmml" xref="S4.I2.i1.p1.1.m1.1.1.2.2">𝒔</ci><cn type="integer" id="S4.I2.i1.p1.1.m1.1.1.2.3.cmml" xref="S4.I2.i1.p1.1.m1.1.1.2.3">1</cn></apply><ci id="S4.I2.i1.p1.1.m1.1.1.3a.cmml" xref="S4.I2.i1.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I2.i1.p1.1.m1.1.1.3.cmml" xref="S4.I2.i1.p1.1.m1.1.1.3">‘‘Step 1: Research Barack Obama’s birth date…’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I2.i1.p1.1.m1.1c">\bm{s}_{1}=\texttt{``Step 1: Research Barack Obama's birth date\ldots''}</annotation></semantics></math><span id="S4.I2.i1.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
<li id="S4.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I2.i2.p1" class="ltx_para">
<p id="S4.I2.i2.p1.1" class="ltx_p"><math id="S4.I2.i2.p1.1.m1.1" class="ltx_Math" alttext="\bm{s}_{2}=\texttt{``Step 2: Determine if the year 1961 is an even number\ldots''}" display="inline"><semantics id="S4.I2.i2.p1.1.m1.1a"><mrow id="S4.I2.i2.p1.1.m1.1.1" xref="S4.I2.i2.p1.1.m1.1.1.cmml"><msub id="S4.I2.i2.p1.1.m1.1.1.2" xref="S4.I2.i2.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I2.i2.p1.1.m1.1.1.2.2" xref="S4.I2.i2.p1.1.m1.1.1.2.2.cmml">𝒔</mi><mn mathsize="70%" id="S4.I2.i2.p1.1.m1.1.1.2.3" xref="S4.I2.i2.p1.1.m1.1.1.2.3.cmml">2</mn></msub><mo mathsize="70%" id="S4.I2.i2.p1.1.m1.1.1.1" xref="S4.I2.i2.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I2.i2.p1.1.m1.1.1.3" xref="S4.I2.i2.p1.1.m1.1.1.3a.cmml">‘‘Step 2: Determine if the year 1961 is an even number…’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I2.i2.p1.1.m1.1b"><apply id="S4.I2.i2.p1.1.m1.1.1.cmml" xref="S4.I2.i2.p1.1.m1.1.1"><eq id="S4.I2.i2.p1.1.m1.1.1.1.cmml" xref="S4.I2.i2.p1.1.m1.1.1.1"></eq><apply id="S4.I2.i2.p1.1.m1.1.1.2.cmml" xref="S4.I2.i2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I2.i2.p1.1.m1.1.1.2.1.cmml" xref="S4.I2.i2.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I2.i2.p1.1.m1.1.1.2.2.cmml" xref="S4.I2.i2.p1.1.m1.1.1.2.2">𝒔</ci><cn type="integer" id="S4.I2.i2.p1.1.m1.1.1.2.3.cmml" xref="S4.I2.i2.p1.1.m1.1.1.2.3">2</cn></apply><ci id="S4.I2.i2.p1.1.m1.1.1.3a.cmml" xref="S4.I2.i2.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I2.i2.p1.1.m1.1.1.3.cmml" xref="S4.I2.i2.p1.1.m1.1.1.3">‘‘Step 2: Determine if the year 1961 is an even number…’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I2.i2.p1.1.m1.1c">\bm{s}_{2}=\texttt{``Step 2: Determine if the year 1961 is an even number\ldots''}</annotation></semantics></math><span id="S4.I2.i2.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
<li id="S4.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I2.i3.p1" class="ltx_para">
<p id="S4.I2.i3.p1.1" class="ltx_p"><math id="S4.I2.i3.p1.1.m1.1" class="ltx_Math" alttext="\bm{s}_{3}=\texttt{``Step 3: Divide 1961 by 2\ldots''}" display="inline"><semantics id="S4.I2.i3.p1.1.m1.1a"><mrow id="S4.I2.i3.p1.1.m1.1.1" xref="S4.I2.i3.p1.1.m1.1.1.cmml"><msub id="S4.I2.i3.p1.1.m1.1.1.2" xref="S4.I2.i3.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I2.i3.p1.1.m1.1.1.2.2" xref="S4.I2.i3.p1.1.m1.1.1.2.2.cmml">𝒔</mi><mn mathsize="70%" id="S4.I2.i3.p1.1.m1.1.1.2.3" xref="S4.I2.i3.p1.1.m1.1.1.2.3.cmml">3</mn></msub><mo mathsize="70%" id="S4.I2.i3.p1.1.m1.1.1.1" xref="S4.I2.i3.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I2.i3.p1.1.m1.1.1.3" xref="S4.I2.i3.p1.1.m1.1.1.3a.cmml">‘‘Step 3: Divide 1961 by 2…’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I2.i3.p1.1.m1.1b"><apply id="S4.I2.i3.p1.1.m1.1.1.cmml" xref="S4.I2.i3.p1.1.m1.1.1"><eq id="S4.I2.i3.p1.1.m1.1.1.1.cmml" xref="S4.I2.i3.p1.1.m1.1.1.1"></eq><apply id="S4.I2.i3.p1.1.m1.1.1.2.cmml" xref="S4.I2.i3.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I2.i3.p1.1.m1.1.1.2.1.cmml" xref="S4.I2.i3.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I2.i3.p1.1.m1.1.1.2.2.cmml" xref="S4.I2.i3.p1.1.m1.1.1.2.2">𝒔</ci><cn type="integer" id="S4.I2.i3.p1.1.m1.1.1.2.3.cmml" xref="S4.I2.i3.p1.1.m1.1.1.2.3">3</cn></apply><ci id="S4.I2.i3.p1.1.m1.1.1.3a.cmml" xref="S4.I2.i3.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I2.i3.p1.1.m1.1.1.3.cmml" xref="S4.I2.i3.p1.1.m1.1.1.3">‘‘Step 3: Divide 1961 by 2…’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I2.i3.p1.1.m1.1c">\bm{s}_{3}=\texttt{``Step 3: Divide 1961 by 2\ldots''}</annotation></semantics></math><span id="S4.I2.i3.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
<li id="S4.I2.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I2.i4.p1" class="ltx_para">
<p id="S4.I2.i4.p1.1" class="ltx_p"><math id="S4.I2.i4.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}=\texttt{``No''}" display="inline"><semantics id="S4.I2.i4.p1.1.m1.1a"><mrow id="S4.I2.i4.p1.1.m1.1.1" xref="S4.I2.i4.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.I2.i4.p1.1.m1.1.1.2" xref="S4.I2.i4.p1.1.m1.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.I2.i4.p1.1.m1.1.1.1" xref="S4.I2.i4.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I2.i4.p1.1.m1.1.1.3" xref="S4.I2.i4.p1.1.m1.1.1.3a.cmml">‘‘No’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I2.i4.p1.1.m1.1b"><apply id="S4.I2.i4.p1.1.m1.1.1.cmml" xref="S4.I2.i4.p1.1.m1.1.1"><eq id="S4.I2.i4.p1.1.m1.1.1.1.cmml" xref="S4.I2.i4.p1.1.m1.1.1.1"></eq><ci id="S4.I2.i4.p1.1.m1.1.1.2.cmml" xref="S4.I2.i4.p1.1.m1.1.1.2">𝐲</ci><ci id="S4.I2.i4.p1.1.m1.1.1.3a.cmml" xref="S4.I2.i4.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I2.i4.p1.1.m1.1.1.3.cmml" xref="S4.I2.i4.p1.1.m1.1.1.3">‘‘No’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I2.i4.p1.1.m1.1c">\mathbf{y}=\texttt{``No''}</annotation></semantics></math><span id="S4.I2.i4.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
</ul>
</div>
</div>
<div id="S4.Thmexmp2" class="ltx_theorem ltx_theorem_exmp">
<h6 class="ltx_title ltx_runin ltx_title_theorem" style="font-size:70%;">
<span class="ltx_tag ltx_tag_theorem"><span id="S4.Thmexmp2.1.1.1" class="ltx_text ltx_font_bold">Example 4.2</span></span><span id="S4.Thmexmp2.2.2" class="ltx_text ltx_font_bold"> </span>(Few-shot CoT)<span id="S4.Thmexmp2.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="S4.Thmexmp2.p1" class="ltx_para">
<p id="S4.Thmexmp2.p1.2" class="ltx_p"><span id="S4.Thmexmp2.p1.2.1" class="ltx_text" style="font-size:70%;">Consider </span><math id="S4.Thmexmp2.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}=\texttt{``Was Barack Obama born in an even year?''}" display="inline"><semantics id="S4.Thmexmp2.p1.1.m1.1a"><mrow id="S4.Thmexmp2.p1.1.m1.1.1" xref="S4.Thmexmp2.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.1.m1.1.1.2" xref="S4.Thmexmp2.p1.1.m1.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.Thmexmp2.p1.1.m1.1.1.1" xref="S4.Thmexmp2.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp2.p1.1.m1.1.1.3" xref="S4.Thmexmp2.p1.1.m1.1.1.3a.cmml">‘‘Was Barack Obama born in an even year?’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp2.p1.1.m1.1b"><apply id="S4.Thmexmp2.p1.1.m1.1.1.cmml" xref="S4.Thmexmp2.p1.1.m1.1.1"><eq id="S4.Thmexmp2.p1.1.m1.1.1.1.cmml" xref="S4.Thmexmp2.p1.1.m1.1.1.1"></eq><ci id="S4.Thmexmp2.p1.1.m1.1.1.2.cmml" xref="S4.Thmexmp2.p1.1.m1.1.1.2">𝐱</ci><ci id="S4.Thmexmp2.p1.1.m1.1.1.3a.cmml" xref="S4.Thmexmp2.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp2.p1.1.m1.1.1.3.cmml" xref="S4.Thmexmp2.p1.1.m1.1.1.3">‘‘Was Barack Obama born in an even year?’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp2.p1.1.m1.1c">\mathbf{x}=\texttt{``Was Barack Obama born in an even year?''}</annotation></semantics></math><span id="S4.Thmexmp2.p1.2.2" class="ltx_text" style="font-size:70%;">, for two-shot CoT, we have </span><math id="S4.Thmexmp2.p1.2.m2.4" class="ltx_Math" alttext="\mathbf{z}_{c}=[\mathbf{x}_{1},\mathbf{y}_{1}^{\prime},\mathbf{x}_{2},\mathbf{y}_{2}^{\prime}]" display="inline"><semantics id="S4.Thmexmp2.p1.2.m2.4a"><mrow id="S4.Thmexmp2.p1.2.m2.4.4" xref="S4.Thmexmp2.p1.2.m2.4.4.cmml"><msub id="S4.Thmexmp2.p1.2.m2.4.4.6" xref="S4.Thmexmp2.p1.2.m2.4.4.6.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.6.2" xref="S4.Thmexmp2.p1.2.m2.4.4.6.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.6.3" xref="S4.Thmexmp2.p1.2.m2.4.4.6.3.cmml">c</mi></msub><mo mathsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.5" xref="S4.Thmexmp2.p1.2.m2.4.4.5.cmml">=</mo><mrow id="S4.Thmexmp2.p1.2.m2.4.4.4.4" xref="S4.Thmexmp2.p1.2.m2.4.4.4.5.cmml"><mo maxsize="70%" minsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.5" xref="S4.Thmexmp2.p1.2.m2.4.4.4.5.cmml">[</mo><msub id="S4.Thmexmp2.p1.2.m2.1.1.1.1.1" xref="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.2" xref="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.3" xref="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.6" xref="S4.Thmexmp2.p1.2.m2.4.4.4.5.cmml">,</mo><msubsup id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.2.2" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.2.3" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.2.3.cmml">1</mn><mo mathsize="70%" id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.3" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.7" xref="S4.Thmexmp2.p1.2.m2.4.4.4.5.cmml">,</mo><msub id="S4.Thmexmp2.p1.2.m2.3.3.3.3.3" xref="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.2" xref="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.3" xref="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.3.cmml">2</mn></msub><mo mathsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.8" xref="S4.Thmexmp2.p1.2.m2.4.4.4.5.cmml">,</mo><msubsup id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.2.2" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.2.3" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.2.3.cmml">2</mn><mo mathsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.3" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.3.cmml">′</mo></msubsup><mo maxsize="70%" minsize="70%" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.9" xref="S4.Thmexmp2.p1.2.m2.4.4.4.5.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp2.p1.2.m2.4b"><apply id="S4.Thmexmp2.p1.2.m2.4.4.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4"><eq id="S4.Thmexmp2.p1.2.m2.4.4.5.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.5"></eq><apply id="S4.Thmexmp2.p1.2.m2.4.4.6.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.6"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.2.m2.4.4.6.1.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.6">subscript</csymbol><ci id="S4.Thmexmp2.p1.2.m2.4.4.6.2.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.6.2">𝐳</ci><ci id="S4.Thmexmp2.p1.2.m2.4.4.6.3.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.6.3">𝑐</ci></apply><list id="S4.Thmexmp2.p1.2.m2.4.4.4.5.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4"><apply id="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.cmml" xref="S4.Thmexmp2.p1.2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.1.cmml" xref="S4.Thmexmp2.p1.2.m2.1.1.1.1.1">subscript</csymbol><ci id="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.2.cmml" xref="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.2">𝐱</ci><cn type="integer" id="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.3.cmml" xref="S4.Thmexmp2.p1.2.m2.1.1.1.1.1.3">1</cn></apply><apply id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.cmml" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.1.cmml" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2">superscript</csymbol><apply id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.2.cmml" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.2.1.cmml" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2">subscript</csymbol><ci id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.2.2.cmml" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.2.2">𝐲</ci><cn type="integer" id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.2.3.cmml" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.2.3">1</cn></apply><ci id="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.3.cmml" xref="S4.Thmexmp2.p1.2.m2.2.2.2.2.2.3">′</ci></apply><apply id="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.cmml" xref="S4.Thmexmp2.p1.2.m2.3.3.3.3.3"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.1.cmml" xref="S4.Thmexmp2.p1.2.m2.3.3.3.3.3">subscript</csymbol><ci id="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.2.cmml" xref="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.2">𝐱</ci><cn type="integer" id="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.3.cmml" xref="S4.Thmexmp2.p1.2.m2.3.3.3.3.3.3">2</cn></apply><apply id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.1.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4">superscript</csymbol><apply id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.2.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.2.1.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4">subscript</csymbol><ci id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.2.2.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.2.2">𝐲</ci><cn type="integer" id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.2.3.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.2.3">2</cn></apply><ci id="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.3.cmml" xref="S4.Thmexmp2.p1.2.m2.4.4.4.4.4.3">′</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp2.p1.2.m2.4c">\mathbf{z}_{c}=[\mathbf{x}_{1},\mathbf{y}_{1}^{\prime},\mathbf{x}_{2},\mathbf{y}_{2}^{\prime}]</annotation></semantics></math><span id="S4.Thmexmp2.p1.2.3" class="ltx_text" style="font-size:70%;"> as the effective prompt, where</span></p>
<ul id="S4.I3" class="ltx_itemize">
<li id="S4.I3.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I3.i1.p1" class="ltx_para">
<p id="S4.I3.i1.p1.1" class="ltx_p"><math id="S4.I3.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}_{1}=\texttt{``Was Donald Trump born in an even year?''}" display="inline"><semantics id="S4.I3.i1.p1.1.m1.1a"><mrow id="S4.I3.i1.p1.1.m1.1.1" xref="S4.I3.i1.p1.1.m1.1.1.cmml"><msub id="S4.I3.i1.p1.1.m1.1.1.2" xref="S4.I3.i1.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I3.i1.p1.1.m1.1.1.2.2" xref="S4.I3.i1.p1.1.m1.1.1.2.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.I3.i1.p1.1.m1.1.1.2.3" xref="S4.I3.i1.p1.1.m1.1.1.2.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.I3.i1.p1.1.m1.1.1.1" xref="S4.I3.i1.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I3.i1.p1.1.m1.1.1.3" xref="S4.I3.i1.p1.1.m1.1.1.3a.cmml">‘‘Was Donald Trump born in an even year?’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I3.i1.p1.1.m1.1b"><apply id="S4.I3.i1.p1.1.m1.1.1.cmml" xref="S4.I3.i1.p1.1.m1.1.1"><eq id="S4.I3.i1.p1.1.m1.1.1.1.cmml" xref="S4.I3.i1.p1.1.m1.1.1.1"></eq><apply id="S4.I3.i1.p1.1.m1.1.1.2.cmml" xref="S4.I3.i1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I3.i1.p1.1.m1.1.1.2.1.cmml" xref="S4.I3.i1.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I3.i1.p1.1.m1.1.1.2.2.cmml" xref="S4.I3.i1.p1.1.m1.1.1.2.2">𝐱</ci><cn type="integer" id="S4.I3.i1.p1.1.m1.1.1.2.3.cmml" xref="S4.I3.i1.p1.1.m1.1.1.2.3">1</cn></apply><ci id="S4.I3.i1.p1.1.m1.1.1.3a.cmml" xref="S4.I3.i1.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I3.i1.p1.1.m1.1.1.3.cmml" xref="S4.I3.i1.p1.1.m1.1.1.3">‘‘Was Donald Trump born in an even year?’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I3.i1.p1.1.m1.1c">\mathbf{x}_{1}=\texttt{``Was Donald Trump born in an even year?''}</annotation></semantics></math><span id="S4.I3.i1.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
<li id="S4.I3.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I3.i2.p1" class="ltx_para">
<p id="S4.I3.i2.p1.1" class="ltx_p"><math id="S4.I3.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}_{1}^{\prime}=\texttt{``Donald Trump was born in 1946. If we divide 1946 by 2\ldots The answer is yes.''}" display="inline"><semantics id="S4.I3.i2.p1.1.m1.1a"><mrow id="S4.I3.i2.p1.1.m1.1.1" xref="S4.I3.i2.p1.1.m1.1.1.cmml"><msubsup id="S4.I3.i2.p1.1.m1.1.1.2" xref="S4.I3.i2.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I3.i2.p1.1.m1.1.1.2.2.2" xref="S4.I3.i2.p1.1.m1.1.1.2.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.I3.i2.p1.1.m1.1.1.2.2.3" xref="S4.I3.i2.p1.1.m1.1.1.2.2.3.cmml">1</mn><mo mathsize="70%" id="S4.I3.i2.p1.1.m1.1.1.2.3" xref="S4.I3.i2.p1.1.m1.1.1.2.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.I3.i2.p1.1.m1.1.1.1" xref="S4.I3.i2.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I3.i2.p1.1.m1.1.1.3" xref="S4.I3.i2.p1.1.m1.1.1.3a.cmml">‘‘Donald Trump was born in 1946. If we divide 1946 by 2…The answer is yes.’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I3.i2.p1.1.m1.1b"><apply id="S4.I3.i2.p1.1.m1.1.1.cmml" xref="S4.I3.i2.p1.1.m1.1.1"><eq id="S4.I3.i2.p1.1.m1.1.1.1.cmml" xref="S4.I3.i2.p1.1.m1.1.1.1"></eq><apply id="S4.I3.i2.p1.1.m1.1.1.2.cmml" xref="S4.I3.i2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I3.i2.p1.1.m1.1.1.2.1.cmml" xref="S4.I3.i2.p1.1.m1.1.1.2">superscript</csymbol><apply id="S4.I3.i2.p1.1.m1.1.1.2.2.cmml" xref="S4.I3.i2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I3.i2.p1.1.m1.1.1.2.2.1.cmml" xref="S4.I3.i2.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I3.i2.p1.1.m1.1.1.2.2.2.cmml" xref="S4.I3.i2.p1.1.m1.1.1.2.2.2">𝐲</ci><cn type="integer" id="S4.I3.i2.p1.1.m1.1.1.2.2.3.cmml" xref="S4.I3.i2.p1.1.m1.1.1.2.2.3">1</cn></apply><ci id="S4.I3.i2.p1.1.m1.1.1.2.3.cmml" xref="S4.I3.i2.p1.1.m1.1.1.2.3">′</ci></apply><ci id="S4.I3.i2.p1.1.m1.1.1.3a.cmml" xref="S4.I3.i2.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I3.i2.p1.1.m1.1.1.3.cmml" xref="S4.I3.i2.p1.1.m1.1.1.3">‘‘Donald Trump was born in 1946. If we divide 1946 by 2…The answer is yes.’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I3.i2.p1.1.m1.1c">\mathbf{y}_{1}^{\prime}=\texttt{``Donald Trump was born in 1946. If we divide 1946 by 2\ldots The answer is yes.''}</annotation></semantics></math><span id="S4.I3.i2.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
<li id="S4.I3.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I3.i3.p1" class="ltx_para">
<p id="S4.I3.i3.p1.1" class="ltx_p"><math id="S4.I3.i3.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}_{2}=\texttt{``Was Marie Curie born in an even year?''}" display="inline"><semantics id="S4.I3.i3.p1.1.m1.1a"><mrow id="S4.I3.i3.p1.1.m1.1.1" xref="S4.I3.i3.p1.1.m1.1.1.cmml"><msub id="S4.I3.i3.p1.1.m1.1.1.2" xref="S4.I3.i3.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I3.i3.p1.1.m1.1.1.2.2" xref="S4.I3.i3.p1.1.m1.1.1.2.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.I3.i3.p1.1.m1.1.1.2.3" xref="S4.I3.i3.p1.1.m1.1.1.2.3.cmml">2</mn></msub><mo mathsize="70%" id="S4.I3.i3.p1.1.m1.1.1.1" xref="S4.I3.i3.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I3.i3.p1.1.m1.1.1.3" xref="S4.I3.i3.p1.1.m1.1.1.3a.cmml">‘‘Was Marie Curie born in an even year?’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I3.i3.p1.1.m1.1b"><apply id="S4.I3.i3.p1.1.m1.1.1.cmml" xref="S4.I3.i3.p1.1.m1.1.1"><eq id="S4.I3.i3.p1.1.m1.1.1.1.cmml" xref="S4.I3.i3.p1.1.m1.1.1.1"></eq><apply id="S4.I3.i3.p1.1.m1.1.1.2.cmml" xref="S4.I3.i3.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I3.i3.p1.1.m1.1.1.2.1.cmml" xref="S4.I3.i3.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I3.i3.p1.1.m1.1.1.2.2.cmml" xref="S4.I3.i3.p1.1.m1.1.1.2.2">𝐱</ci><cn type="integer" id="S4.I3.i3.p1.1.m1.1.1.2.3.cmml" xref="S4.I3.i3.p1.1.m1.1.1.2.3">2</cn></apply><ci id="S4.I3.i3.p1.1.m1.1.1.3a.cmml" xref="S4.I3.i3.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I3.i3.p1.1.m1.1.1.3.cmml" xref="S4.I3.i3.p1.1.m1.1.1.3">‘‘Was Marie Curie born in an even year?’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I3.i3.p1.1.m1.1c">\mathbf{x}_{2}=\texttt{``Was Marie Curie born in an even year?''}</annotation></semantics></math><span id="S4.I3.i3.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
<li id="S4.I3.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I3.i4.p1" class="ltx_para">
<p id="S4.I3.i4.p1.1" class="ltx_p"><math id="S4.I3.i4.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}_{2}^{\prime}=\texttt{``Marie Curie was born in 1867. If we divide 1867 by 2\ldots The answer is no.''}" display="inline"><semantics id="S4.I3.i4.p1.1.m1.1a"><mrow id="S4.I3.i4.p1.1.m1.1.1" xref="S4.I3.i4.p1.1.m1.1.1.cmml"><msubsup id="S4.I3.i4.p1.1.m1.1.1.2" xref="S4.I3.i4.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I3.i4.p1.1.m1.1.1.2.2.2" xref="S4.I3.i4.p1.1.m1.1.1.2.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.I3.i4.p1.1.m1.1.1.2.2.3" xref="S4.I3.i4.p1.1.m1.1.1.2.2.3.cmml">2</mn><mo mathsize="70%" id="S4.I3.i4.p1.1.m1.1.1.2.3" xref="S4.I3.i4.p1.1.m1.1.1.2.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.I3.i4.p1.1.m1.1.1.1" xref="S4.I3.i4.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I3.i4.p1.1.m1.1.1.3" xref="S4.I3.i4.p1.1.m1.1.1.3a.cmml">‘‘Marie Curie was born in 1867. If we divide 1867 by 2…The answer is no.’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I3.i4.p1.1.m1.1b"><apply id="S4.I3.i4.p1.1.m1.1.1.cmml" xref="S4.I3.i4.p1.1.m1.1.1"><eq id="S4.I3.i4.p1.1.m1.1.1.1.cmml" xref="S4.I3.i4.p1.1.m1.1.1.1"></eq><apply id="S4.I3.i4.p1.1.m1.1.1.2.cmml" xref="S4.I3.i4.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I3.i4.p1.1.m1.1.1.2.1.cmml" xref="S4.I3.i4.p1.1.m1.1.1.2">superscript</csymbol><apply id="S4.I3.i4.p1.1.m1.1.1.2.2.cmml" xref="S4.I3.i4.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I3.i4.p1.1.m1.1.1.2.2.1.cmml" xref="S4.I3.i4.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I3.i4.p1.1.m1.1.1.2.2.2.cmml" xref="S4.I3.i4.p1.1.m1.1.1.2.2.2">𝐲</ci><cn type="integer" id="S4.I3.i4.p1.1.m1.1.1.2.2.3.cmml" xref="S4.I3.i4.p1.1.m1.1.1.2.2.3">2</cn></apply><ci id="S4.I3.i4.p1.1.m1.1.1.2.3.cmml" xref="S4.I3.i4.p1.1.m1.1.1.2.3">′</ci></apply><ci id="S4.I3.i4.p1.1.m1.1.1.3a.cmml" xref="S4.I3.i4.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I3.i4.p1.1.m1.1.1.3.cmml" xref="S4.I3.i4.p1.1.m1.1.1.3">‘‘Marie Curie was born in 1867. If we divide 1867 by 2…The answer is no.’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I3.i4.p1.1.m1.1c">\mathbf{y}_{2}^{\prime}=\texttt{``Marie Curie was born in 1867. If we divide 1867 by 2\ldots The answer is no.''}</annotation></semantics></math><span id="S4.I3.i4.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
</ul>
<p id="S4.Thmexmp2.p1.3" class="ltx_p"><span id="S4.Thmexmp2.p1.3.1" class="ltx_text" style="font-size:70%;">And we obtain </span><math id="S4.Thmexmp2.p1.3.m1.2" class="ltx_Math" alttext="y^{\prime}\sim p_{\text{LLM}}(\mathbf{y}^{\prime}|\mathbf{x},\mathbf{z}_{c})" display="inline"><semantics id="S4.Thmexmp2.p1.3.m1.2a"><mrow id="S4.Thmexmp2.p1.3.m1.2.2" xref="S4.Thmexmp2.p1.3.m1.2.2.cmml"><msup id="S4.Thmexmp2.p1.3.m1.2.2.3" xref="S4.Thmexmp2.p1.3.m1.2.2.3.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.3.2" xref="S4.Thmexmp2.p1.3.m1.2.2.3.2.cmml">y</mi><mo mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.3.3" xref="S4.Thmexmp2.p1.3.m1.2.2.3.3.cmml">′</mo></msup><mo mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.2" xref="S4.Thmexmp2.p1.3.m1.2.2.2.cmml">∼</mo><mrow id="S4.Thmexmp2.p1.3.m1.2.2.1" xref="S4.Thmexmp2.p1.3.m1.2.2.1.cmml"><msub id="S4.Thmexmp2.p1.3.m1.2.2.1.3" xref="S4.Thmexmp2.p1.3.m1.2.2.1.3.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.1.3.2" xref="S4.Thmexmp2.p1.3.m1.2.2.1.3.2.cmml">p</mi><mtext mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.1.3.3" xref="S4.Thmexmp2.p1.3.m1.2.2.1.3.3a.cmml">LLM</mtext></msub><mo lspace="0em" rspace="0em" id="S4.Thmexmp2.p1.3.m1.2.2.1.2" xref="S4.Thmexmp2.p1.3.m1.2.2.1.2.cmml">​</mo><mrow id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.cmml"><mo maxsize="70%" minsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.2" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.cmml">(</mo><mrow id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.cmml"><msup id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.2" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.3" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.3.cmml">′</mo></msup><mo fence="false" mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.2" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.2.cmml">|</mo><mrow id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.2.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.3.m1.1.1" xref="S4.Thmexmp2.p1.3.m1.1.1.cmml">𝐱</mi><mo mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.2" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.2.cmml">,</mo><msub id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.2" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.3" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.3.cmml">c</mi></msub></mrow></mrow><mo maxsize="70%" minsize="70%" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.3" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp2.p1.3.m1.2b"><apply id="S4.Thmexmp2.p1.3.m1.2.2.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2"><csymbol cd="latexml" id="S4.Thmexmp2.p1.3.m1.2.2.2.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.2">similar-to</csymbol><apply id="S4.Thmexmp2.p1.3.m1.2.2.3.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.3"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.3.m1.2.2.3.1.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.3">superscript</csymbol><ci id="S4.Thmexmp2.p1.3.m1.2.2.3.2.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.3.2">𝑦</ci><ci id="S4.Thmexmp2.p1.3.m1.2.2.3.3.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.3.3">′</ci></apply><apply id="S4.Thmexmp2.p1.3.m1.2.2.1.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1"><times id="S4.Thmexmp2.p1.3.m1.2.2.1.2.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.2"></times><apply id="S4.Thmexmp2.p1.3.m1.2.2.1.3.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.3"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.3.m1.2.2.1.3.1.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.3">subscript</csymbol><ci id="S4.Thmexmp2.p1.3.m1.2.2.1.3.2.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.3.2">𝑝</ci><ci id="S4.Thmexmp2.p1.3.m1.2.2.1.3.3a.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.3.3"><mtext mathsize="49%" id="S4.Thmexmp2.p1.3.m1.2.2.1.3.3.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.3.3">LLM</mtext></ci></apply><apply id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1"><csymbol cd="latexml" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.2.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.2">conditional</csymbol><apply id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.1.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3">superscript</csymbol><ci id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.2.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.2">𝐲</ci><ci id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.3.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.3.3">′</ci></apply><list id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.2.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1"><ci id="S4.Thmexmp2.p1.3.m1.1.1.cmml" xref="S4.Thmexmp2.p1.3.m1.1.1">𝐱</ci><apply id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.1.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1">subscript</csymbol><ci id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.2.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.2">𝐳</ci><ci id="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S4.Thmexmp2.p1.3.m1.2.2.1.1.1.1.1.1.1.3">𝑐</ci></apply></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp2.p1.3.m1.2c">y^{\prime}\sim p_{\text{LLM}}(\mathbf{y}^{\prime}|\mathbf{x},\mathbf{z}_{c})</annotation></semantics></math><span id="S4.Thmexmp2.p1.3.2" class="ltx_text" style="font-size:70%;"> as</span></p>
<ul id="S4.I4" class="ltx_itemize">
<li id="S4.I4.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I4.i1.p1" class="ltx_para">
<p id="S4.I4.i1.p1.1" class="ltx_p"><math id="S4.I4.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}=\texttt{``Barack Obama was born in 1961. If we divide 1961 by 2\ldots The answer is no.''}" display="inline"><semantics id="S4.I4.i1.p1.1.m1.1a"><mrow id="S4.I4.i1.p1.1.m1.1.1" xref="S4.I4.i1.p1.1.m1.1.1.cmml"><msup id="S4.I4.i1.p1.1.m1.1.1.2" xref="S4.I4.i1.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I4.i1.p1.1.m1.1.1.2.2" xref="S4.I4.i1.p1.1.m1.1.1.2.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.I4.i1.p1.1.m1.1.1.2.3" xref="S4.I4.i1.p1.1.m1.1.1.2.3.cmml">′</mo></msup><mo mathsize="70%" id="S4.I4.i1.p1.1.m1.1.1.1" xref="S4.I4.i1.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I4.i1.p1.1.m1.1.1.3" xref="S4.I4.i1.p1.1.m1.1.1.3a.cmml">‘‘Barack Obama was born in 1961. If we divide 1961 by 2…The answer is no.’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I4.i1.p1.1.m1.1b"><apply id="S4.I4.i1.p1.1.m1.1.1.cmml" xref="S4.I4.i1.p1.1.m1.1.1"><eq id="S4.I4.i1.p1.1.m1.1.1.1.cmml" xref="S4.I4.i1.p1.1.m1.1.1.1"></eq><apply id="S4.I4.i1.p1.1.m1.1.1.2.cmml" xref="S4.I4.i1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I4.i1.p1.1.m1.1.1.2.1.cmml" xref="S4.I4.i1.p1.1.m1.1.1.2">superscript</csymbol><ci id="S4.I4.i1.p1.1.m1.1.1.2.2.cmml" xref="S4.I4.i1.p1.1.m1.1.1.2.2">𝐲</ci><ci id="S4.I4.i1.p1.1.m1.1.1.2.3.cmml" xref="S4.I4.i1.p1.1.m1.1.1.2.3">′</ci></apply><ci id="S4.I4.i1.p1.1.m1.1.1.3a.cmml" xref="S4.I4.i1.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I4.i1.p1.1.m1.1.1.3.cmml" xref="S4.I4.i1.p1.1.m1.1.1.3">‘‘Barack Obama was born in 1961. If we divide 1961 by 2…The answer is no.’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I4.i1.p1.1.m1.1c">\mathbf{y}^{\prime}=\texttt{``Barack Obama was born in 1961. If we divide 1961 by 2\ldots The answer is no.''}</annotation></semantics></math><span id="S4.I4.i1.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
</ul>
</div>
</div>
</section>
<section id="S4.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsubsection">4.1.2 </span>One-step RaR</h4>

<div id="S4.SS1.SSS2.p1" class="ltx_para">
<p id="S4.SS1.SSS2.p1.4" class="ltx_p"><span id="S4.SS1.SSS2.p1.4.1" class="ltx_text" style="font-size:70%;">The foundation of our (one-step) RaR method is different from CoT. We generate a rephrased question </span><math id="S4.SS1.SSS2.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}^{\prime}" display="inline"><semantics id="S4.SS1.SSS2.p1.1.m1.1a"><msup id="S4.SS1.SSS2.p1.1.m1.1.1" xref="S4.SS1.SSS2.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS2.p1.1.m1.1.1.2" xref="S4.SS1.SSS2.p1.1.m1.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.SS1.SSS2.p1.1.m1.1.1.3" xref="S4.SS1.SSS2.p1.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.1.m1.1b"><apply id="S4.SS1.SSS2.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS2.p1.1.m1.1.1.1.cmml" xref="S4.SS1.SSS2.p1.1.m1.1.1">superscript</csymbol><ci id="S4.SS1.SSS2.p1.1.m1.1.1.2.cmml" xref="S4.SS1.SSS2.p1.1.m1.1.1.2">𝐱</ci><ci id="S4.SS1.SSS2.p1.1.m1.1.1.3.cmml" xref="S4.SS1.SSS2.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.1.m1.1c">\mathbf{x}^{\prime}</annotation></semantics></math><span id="S4.SS1.SSS2.p1.4.2" class="ltx_text" style="font-size:70%;"> that retains the same semantic content as </span><math id="S4.SS1.SSS2.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{x}" display="inline"><semantics id="S4.SS1.SSS2.p1.2.m2.1a"><mi mathsize="70%" id="S4.SS1.SSS2.p1.2.m2.1.1" xref="S4.SS1.SSS2.p1.2.m2.1.1.cmml">𝐱</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.2.m2.1b"><ci id="S4.SS1.SSS2.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS2.p1.2.m2.1.1">𝐱</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.2.m2.1c">\mathbf{x}</annotation></semantics></math><span id="S4.SS1.SSS2.p1.4.3" class="ltx_text" style="font-size:70%;">, and the associated answer </span><math id="S4.SS1.SSS2.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.SS1.SSS2.p1.3.m3.1a"><mi mathsize="70%" id="S4.SS1.SSS2.p1.3.m3.1.1" xref="S4.SS1.SSS2.p1.3.m3.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.3.m3.1b"><ci id="S4.SS1.SSS2.p1.3.m3.1.1.cmml" xref="S4.SS1.SSS2.p1.3.m3.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.3.m3.1c">\mathbf{y}</annotation></semantics></math><span id="S4.SS1.SSS2.p1.4.4" class="ltx_text" style="font-size:70%;">. Specifically, we define </span><math id="S4.SS1.SSS2.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.SS1.SSS2.p1.4.m4.1a"><msup id="S4.SS1.SSS2.p1.4.m4.1.1" xref="S4.SS1.SSS2.p1.4.m4.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS2.p1.4.m4.1.1.2" xref="S4.SS1.SSS2.p1.4.m4.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.SS1.SSS2.p1.4.m4.1.1.3" xref="S4.SS1.SSS2.p1.4.m4.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.4.m4.1b"><apply id="S4.SS1.SSS2.p1.4.m4.1.1.cmml" xref="S4.SS1.SSS2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS2.p1.4.m4.1.1.1.cmml" xref="S4.SS1.SSS2.p1.4.m4.1.1">superscript</csymbol><ci id="S4.SS1.SSS2.p1.4.m4.1.1.2.cmml" xref="S4.SS1.SSS2.p1.4.m4.1.1.2">𝐲</ci><ci id="S4.SS1.SSS2.p1.4.m4.1.1.3.cmml" xref="S4.SS1.SSS2.p1.4.m4.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.4.m4.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.SS1.SSS2.p1.4.5" class="ltx_text" style="font-size:70%;"> as</span></p>
<table id="A2.EGx5" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E2.m1.2" class="ltx_Math" alttext="\displaystyle\mathbf{y}^{\prime}=[\mathbf{x}^{\prime},\mathbf{y}]," display="inline"><semantics id="S4.E2.m1.2a"><mrow id="S4.E2.m1.2.2.1" xref="S4.E2.m1.2.2.1.1.cmml"><mrow id="S4.E2.m1.2.2.1.1" xref="S4.E2.m1.2.2.1.1.cmml"><msup id="S4.E2.m1.2.2.1.1.3" xref="S4.E2.m1.2.2.1.1.3.cmml"><mi mathsize="70%" id="S4.E2.m1.2.2.1.1.3.2" xref="S4.E2.m1.2.2.1.1.3.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.E2.m1.2.2.1.1.3.3" xref="S4.E2.m1.2.2.1.1.3.3.cmml">′</mo></msup><mo mathsize="70%" id="S4.E2.m1.2.2.1.1.2" xref="S4.E2.m1.2.2.1.1.2.cmml">=</mo><mrow id="S4.E2.m1.2.2.1.1.1.1" xref="S4.E2.m1.2.2.1.1.1.2.cmml"><mo maxsize="70%" minsize="70%" id="S4.E2.m1.2.2.1.1.1.1.2" xref="S4.E2.m1.2.2.1.1.1.2.cmml">[</mo><msup id="S4.E2.m1.2.2.1.1.1.1.1" xref="S4.E2.m1.2.2.1.1.1.1.1.cmml"><mi mathsize="70%" id="S4.E2.m1.2.2.1.1.1.1.1.2" xref="S4.E2.m1.2.2.1.1.1.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.E2.m1.2.2.1.1.1.1.1.3" xref="S4.E2.m1.2.2.1.1.1.1.1.3.cmml">′</mo></msup><mo mathsize="70%" id="S4.E2.m1.2.2.1.1.1.1.3" xref="S4.E2.m1.2.2.1.1.1.2.cmml">,</mo><mi mathsize="70%" id="S4.E2.m1.1.1" xref="S4.E2.m1.1.1.cmml">𝐲</mi><mo maxsize="70%" minsize="70%" id="S4.E2.m1.2.2.1.1.1.1.4" xref="S4.E2.m1.2.2.1.1.1.2.cmml">]</mo></mrow></mrow><mo mathsize="70%" id="S4.E2.m1.2.2.1.2" xref="S4.E2.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.E2.m1.2b"><apply id="S4.E2.m1.2.2.1.1.cmml" xref="S4.E2.m1.2.2.1"><eq id="S4.E2.m1.2.2.1.1.2.cmml" xref="S4.E2.m1.2.2.1.1.2"></eq><apply id="S4.E2.m1.2.2.1.1.3.cmml" xref="S4.E2.m1.2.2.1.1.3"><csymbol cd="ambiguous" id="S4.E2.m1.2.2.1.1.3.1.cmml" xref="S4.E2.m1.2.2.1.1.3">superscript</csymbol><ci id="S4.E2.m1.2.2.1.1.3.2.cmml" xref="S4.E2.m1.2.2.1.1.3.2">𝐲</ci><ci id="S4.E2.m1.2.2.1.1.3.3.cmml" xref="S4.E2.m1.2.2.1.1.3.3">′</ci></apply><interval closure="closed" id="S4.E2.m1.2.2.1.1.1.2.cmml" xref="S4.E2.m1.2.2.1.1.1.1"><apply id="S4.E2.m1.2.2.1.1.1.1.1.cmml" xref="S4.E2.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E2.m1.2.2.1.1.1.1.1.1.cmml" xref="S4.E2.m1.2.2.1.1.1.1.1">superscript</csymbol><ci id="S4.E2.m1.2.2.1.1.1.1.1.2.cmml" xref="S4.E2.m1.2.2.1.1.1.1.1.2">𝐱</ci><ci id="S4.E2.m1.2.2.1.1.1.1.1.3.cmml" xref="S4.E2.m1.2.2.1.1.1.1.1.3">′</ci></apply><ci id="S4.E2.m1.1.1.cmml" xref="S4.E2.m1.1.1">𝐲</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E2.m1.2c">\displaystyle\mathbf{y}^{\prime}=[\mathbf{x}^{\prime},\mathbf{y}],</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4.2)</span></td>
</tr></tbody>
</table>
<p id="S4.SS1.SSS2.p1.6" class="ltx_p"><span id="S4.SS1.SSS2.p1.6.1" class="ltx_text" style="font-size:70%;">where </span><math id="S4.SS1.SSS2.p1.5.m1.1" class="ltx_Math" alttext="\mathbf{x}^{\prime}" display="inline"><semantics id="S4.SS1.SSS2.p1.5.m1.1a"><msup id="S4.SS1.SSS2.p1.5.m1.1.1" xref="S4.SS1.SSS2.p1.5.m1.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS2.p1.5.m1.1.1.2" xref="S4.SS1.SSS2.p1.5.m1.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.SS1.SSS2.p1.5.m1.1.1.3" xref="S4.SS1.SSS2.p1.5.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.5.m1.1b"><apply id="S4.SS1.SSS2.p1.5.m1.1.1.cmml" xref="S4.SS1.SSS2.p1.5.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS2.p1.5.m1.1.1.1.cmml" xref="S4.SS1.SSS2.p1.5.m1.1.1">superscript</csymbol><ci id="S4.SS1.SSS2.p1.5.m1.1.1.2.cmml" xref="S4.SS1.SSS2.p1.5.m1.1.1.2">𝐱</ci><ci id="S4.SS1.SSS2.p1.5.m1.1.1.3.cmml" xref="S4.SS1.SSS2.p1.5.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.5.m1.1c">\mathbf{x}^{\prime}</annotation></semantics></math><span id="S4.SS1.SSS2.p1.6.2" class="ltx_text" style="font-size:70%;"> is the rephrased question that induces the answer </span><math id="S4.SS1.SSS2.p1.6.m2.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.SS1.SSS2.p1.6.m2.1a"><mi mathsize="70%" id="S4.SS1.SSS2.p1.6.m2.1.1" xref="S4.SS1.SSS2.p1.6.m2.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.6.m2.1b"><ci id="S4.SS1.SSS2.p1.6.m2.1.1.cmml" xref="S4.SS1.SSS2.p1.6.m2.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.6.m2.1c">\mathbf{y}</annotation></semantics></math><span id="S4.SS1.SSS2.p1.6.3" class="ltx_text" style="font-size:70%;">. In particular, RaR consists of two primary phases:</span></p>
</div>
<div id="S4.SS1.SSS2.p2" class="ltx_para ltx_noindent">
<svg id="S4.SS1.SSS2.p2.pic1" class="ltx_picture" height="70.62" overflow="visible" version="1.1" width="600"><g transform="translate(0,70.62) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#FCB864" fill-opacity="1.0"><path d="M 0 5.91 L 0 64.71 C 0 67.97 2.64 70.62 5.91 70.62 L 594.09 70.62 C 597.36 70.62 600 67.97 600 64.71 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#FFFBF7" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 52.19 L 598.03 52.19 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 58.09)"><foreignObject width="556.69" height="6.62" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S4.SS1.SSS2.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S4.SS1.SSS2.p2.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S4.SS1.SSS2.p2.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">RaR</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="26.6" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S4.SS1.SSS2.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S4.I5" class="ltx_itemize">
<span id="S4.I5.i1" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I5.i1.p1" class="ltx_para">
<span id="S4.I5.i1.p1.5" class="ltx_p"><span id="S4.I5.i1.p1.5.1" class="ltx_text" style="font-size:70%;">Find an instruction </span><math id="S4.I5.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{z}_{r}" display="inline"><semantics id="S4.I5.i1.p1.1.m1.1a"><msub id="S4.I5.i1.p1.1.m1.1.1" xref="S4.I5.i1.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.I5.i1.p1.1.m1.1.1.2" xref="S4.I5.i1.p1.1.m1.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.I5.i1.p1.1.m1.1.1.3" xref="S4.I5.i1.p1.1.m1.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="S4.I5.i1.p1.1.m1.1b"><apply id="S4.I5.i1.p1.1.m1.1.1.cmml" xref="S4.I5.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.I5.i1.p1.1.m1.1.1.1.cmml" xref="S4.I5.i1.p1.1.m1.1.1">subscript</csymbol><ci id="S4.I5.i1.p1.1.m1.1.1.2.cmml" xref="S4.I5.i1.p1.1.m1.1.1.2">𝐳</ci><ci id="S4.I5.i1.p1.1.m1.1.1.3.cmml" xref="S4.I5.i1.p1.1.m1.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I5.i1.p1.1.m1.1c">\mathbf{z}_{r}</annotation></semantics></math><span id="S4.I5.i1.p1.5.2" class="ltx_text" style="font-size:70%;">, generate </span><math id="S4.I5.i1.p1.2.m2.1" class="ltx_math_unparsed" alttext="\mathbf{y}^{\prime}\sim p_{\text{LLM}}(\cdot|\mathbf{x},\mathbf{z}_{r})" display="inline"><semantics id="S4.I5.i1.p1.2.m2.1a"><mrow id="S4.I5.i1.p1.2.m2.1b"><msup id="S4.I5.i1.p1.2.m2.1.1"><mi mathsize="70%" id="S4.I5.i1.p1.2.m2.1.1.2">𝐲</mi><mo mathsize="70%" id="S4.I5.i1.p1.2.m2.1.1.3">′</mo></msup><mo mathsize="70%" id="S4.I5.i1.p1.2.m2.1.2">∼</mo><msub id="S4.I5.i1.p1.2.m2.1.3"><mi mathsize="70%" id="S4.I5.i1.p1.2.m2.1.3.2">p</mi><mtext mathsize="70%" id="S4.I5.i1.p1.2.m2.1.3.3">LLM</mtext></msub><mrow id="S4.I5.i1.p1.2.m2.1.4"><mo maxsize="70%" minsize="70%" id="S4.I5.i1.p1.2.m2.1.4.1">(</mo><mo lspace="0em" mathsize="70%" rspace="0em" id="S4.I5.i1.p1.2.m2.1.4.2">⋅</mo><mo fence="false" maxsize="70%" minsize="70%" rspace="0.167em" id="S4.I5.i1.p1.2.m2.1.4.3">|</mo><mi mathsize="70%" id="S4.I5.i1.p1.2.m2.1.4.4">𝐱</mi><mo mathsize="70%" id="S4.I5.i1.p1.2.m2.1.4.5">,</mo><msub id="S4.I5.i1.p1.2.m2.1.4.6"><mi mathsize="70%" id="S4.I5.i1.p1.2.m2.1.4.6.2">𝐳</mi><mi mathsize="70%" id="S4.I5.i1.p1.2.m2.1.4.6.3">r</mi></msub><mo maxsize="70%" minsize="70%" id="S4.I5.i1.p1.2.m2.1.4.7">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.I5.i1.p1.2.m2.1c">\mathbf{y}^{\prime}\sim p_{\text{LLM}}(\cdot|\mathbf{x},\mathbf{z}_{r})</annotation></semantics></math><span id="S4.I5.i1.p1.5.3" class="ltx_text" style="font-size:70%;"> following the “Rephrase and Respond” format of (</span><a href="#S4.E2" title="Equation 4.2 ‣ 4.1.2 One-step RaR ‣ 4.1 Mathematical Formulation ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">4.2</span></a><span id="S4.I5.i1.p1.5.4" class="ltx_text" style="font-size:70%;">). </span><math id="S4.I5.i1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.I5.i1.p1.3.m3.1a"><msup id="S4.I5.i1.p1.3.m3.1.1" xref="S4.I5.i1.p1.3.m3.1.1.cmml"><mi mathsize="70%" id="S4.I5.i1.p1.3.m3.1.1.2" xref="S4.I5.i1.p1.3.m3.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.I5.i1.p1.3.m3.1.1.3" xref="S4.I5.i1.p1.3.m3.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.I5.i1.p1.3.m3.1b"><apply id="S4.I5.i1.p1.3.m3.1.1.cmml" xref="S4.I5.i1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S4.I5.i1.p1.3.m3.1.1.1.cmml" xref="S4.I5.i1.p1.3.m3.1.1">superscript</csymbol><ci id="S4.I5.i1.p1.3.m3.1.1.2.cmml" xref="S4.I5.i1.p1.3.m3.1.1.2">𝐲</ci><ci id="S4.I5.i1.p1.3.m3.1.1.3.cmml" xref="S4.I5.i1.p1.3.m3.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I5.i1.p1.3.m3.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.I5.i1.p1.5.5" class="ltx_text" style="font-size:70%;"> contains the rephrased question </span><math id="S4.I5.i1.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{x}^{\prime}" display="inline"><semantics id="S4.I5.i1.p1.4.m4.1a"><msup id="S4.I5.i1.p1.4.m4.1.1" xref="S4.I5.i1.p1.4.m4.1.1.cmml"><mi mathsize="70%" id="S4.I5.i1.p1.4.m4.1.1.2" xref="S4.I5.i1.p1.4.m4.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.I5.i1.p1.4.m4.1.1.3" xref="S4.I5.i1.p1.4.m4.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.I5.i1.p1.4.m4.1b"><apply id="S4.I5.i1.p1.4.m4.1.1.cmml" xref="S4.I5.i1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.I5.i1.p1.4.m4.1.1.1.cmml" xref="S4.I5.i1.p1.4.m4.1.1">superscript</csymbol><ci id="S4.I5.i1.p1.4.m4.1.1.2.cmml" xref="S4.I5.i1.p1.4.m4.1.1.2">𝐱</ci><ci id="S4.I5.i1.p1.4.m4.1.1.3.cmml" xref="S4.I5.i1.p1.4.m4.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I5.i1.p1.4.m4.1c">\mathbf{x}^{\prime}</annotation></semantics></math><span id="S4.I5.i1.p1.5.6" class="ltx_text" style="font-size:70%;"> and the desired question </span><math id="S4.I5.i1.p1.5.m5.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.I5.i1.p1.5.m5.1a"><mi mathsize="70%" id="S4.I5.i1.p1.5.m5.1.1" xref="S4.I5.i1.p1.5.m5.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.I5.i1.p1.5.m5.1b"><ci id="S4.I5.i1.p1.5.m5.1.1.cmml" xref="S4.I5.i1.p1.5.m5.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.I5.i1.p1.5.m5.1c">\mathbf{y}</annotation></semantics></math><span id="S4.I5.i1.p1.5.7" class="ltx_text" style="font-size:70%;">.</span></span>
</span></span>
<span id="S4.I5.i2" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I5.i2.p1" class="ltx_para">
<span id="S4.I5.i2.p1.2" class="ltx_p"><span id="S4.I5.i2.p1.2.1" class="ltx_text" style="font-size:70%;">Extract the desired answer </span><math id="S4.I5.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.I5.i2.p1.1.m1.1a"><mi mathsize="70%" id="S4.I5.i2.p1.1.m1.1.1" xref="S4.I5.i2.p1.1.m1.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.I5.i2.p1.1.m1.1b"><ci id="S4.I5.i2.p1.1.m1.1.1.cmml" xref="S4.I5.i2.p1.1.m1.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.I5.i2.p1.1.m1.1c">\mathbf{y}</annotation></semantics></math><span id="S4.I5.i2.p1.2.2" class="ltx_text" style="font-size:70%;"> from the sequence </span><math id="S4.I5.i2.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.I5.i2.p1.2.m2.1a"><msup id="S4.I5.i2.p1.2.m2.1.1" xref="S4.I5.i2.p1.2.m2.1.1.cmml"><mi mathsize="70%" id="S4.I5.i2.p1.2.m2.1.1.2" xref="S4.I5.i2.p1.2.m2.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.I5.i2.p1.2.m2.1.1.3" xref="S4.I5.i2.p1.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.I5.i2.p1.2.m2.1b"><apply id="S4.I5.i2.p1.2.m2.1.1.cmml" xref="S4.I5.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.I5.i2.p1.2.m2.1.1.1.cmml" xref="S4.I5.i2.p1.2.m2.1.1">superscript</csymbol><ci id="S4.I5.i2.p1.2.m2.1.1.2.cmml" xref="S4.I5.i2.p1.2.m2.1.1.2">𝐲</ci><ci id="S4.I5.i2.p1.2.m2.1.1.3.cmml" xref="S4.I5.i2.p1.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I5.i2.p1.2.m2.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.I5.i2.p1.2.3" class="ltx_text" style="font-size:70%;">.
</span></span>
</span></span>
</span>
</span></foreignObject></g></g></svg>
</div>
<div id="S4.SS1.SSS2.p3" class="ltx_para">
<p id="S4.SS1.SSS2.p3.1" class="ltx_p"><span id="S4.SS1.SSS2.p3.1.1" class="ltx_text" style="font-size:70%;">We provide an example of (One-step) RaR as follows.</span></p>
</div>
<div id="S4.Thmexmp3" class="ltx_theorem ltx_theorem_exmp">
<h6 class="ltx_title ltx_runin ltx_title_theorem" style="font-size:70%;">
<span class="ltx_tag ltx_tag_theorem"><span id="S4.Thmexmp3.1.1.1" class="ltx_text ltx_font_bold">Example 4.3</span></span><span id="S4.Thmexmp3.2.2" class="ltx_text ltx_font_bold"> </span>(RaR)<span id="S4.Thmexmp3.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="S4.Thmexmp3.p1" class="ltx_para">
<p id="S4.Thmexmp3.p1.1" class="ltx_p"><span id="S4.Thmexmp3.p1.1.1" class="ltx_text" style="font-size:70%;">Consider the question </span><math id="S4.Thmexmp3.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}=\texttt{``Was Barack Obama born in an even day?''}" display="inline"><semantics id="S4.Thmexmp3.p1.1.m1.1a"><mrow id="S4.Thmexmp3.p1.1.m1.1.1" xref="S4.Thmexmp3.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp3.p1.1.m1.1.1.2" xref="S4.Thmexmp3.p1.1.m1.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.Thmexmp3.p1.1.m1.1.1.1" xref="S4.Thmexmp3.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp3.p1.1.m1.1.1.3" xref="S4.Thmexmp3.p1.1.m1.1.1.3a.cmml">‘‘Was Barack Obama born in an even day?’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp3.p1.1.m1.1b"><apply id="S4.Thmexmp3.p1.1.m1.1.1.cmml" xref="S4.Thmexmp3.p1.1.m1.1.1"><eq id="S4.Thmexmp3.p1.1.m1.1.1.1.cmml" xref="S4.Thmexmp3.p1.1.m1.1.1.1"></eq><ci id="S4.Thmexmp3.p1.1.m1.1.1.2.cmml" xref="S4.Thmexmp3.p1.1.m1.1.1.2">𝐱</ci><ci id="S4.Thmexmp3.p1.1.m1.1.1.3a.cmml" xref="S4.Thmexmp3.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp3.p1.1.m1.1.1.3.cmml" xref="S4.Thmexmp3.p1.1.m1.1.1.3">‘‘Was Barack Obama born in an even day?’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp3.p1.1.m1.1c">\mathbf{x}=\texttt{``Was Barack Obama born in an even day?''}</annotation></semantics></math><span id="S4.Thmexmp3.p1.1.2" class="ltx_text" style="font-size:70%;">, for RaR, we have</span></p>
<table id="A2.EGx6" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.Ex1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.Ex1.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{z}_{r}=" display="inline"><semantics id="S4.Ex1.m1.1a"><mrow id="S4.Ex1.m1.1.1" xref="S4.Ex1.m1.1.1.cmml"><msub id="S4.Ex1.m1.1.1.2" xref="S4.Ex1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.Ex1.m1.1.1.2.2" xref="S4.Ex1.m1.1.1.2.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.Ex1.m1.1.1.2.3" xref="S4.Ex1.m1.1.1.2.3.cmml">r</mi></msub><mo mathsize="70%" id="S4.Ex1.m1.1.1.1" xref="S4.Ex1.m1.1.1.1.cmml">=</mo><mi id="S4.Ex1.m1.1.1.3" xref="S4.Ex1.m1.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="S4.Ex1.m1.1b"><apply id="S4.Ex1.m1.1.1.cmml" xref="S4.Ex1.m1.1.1"><eq id="S4.Ex1.m1.1.1.1.cmml" xref="S4.Ex1.m1.1.1.1"></eq><apply id="S4.Ex1.m1.1.1.2.cmml" xref="S4.Ex1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex1.m1.1.1.2.1.cmml" xref="S4.Ex1.m1.1.1.2">subscript</csymbol><ci id="S4.Ex1.m1.1.1.2.2.cmml" xref="S4.Ex1.m1.1.1.2.2">𝐳</ci><ci id="S4.Ex1.m1.1.1.2.3.cmml" xref="S4.Ex1.m1.1.1.2.3">𝑟</ci></apply><csymbol cd="latexml" id="S4.Ex1.m1.1.1.3.cmml" xref="S4.Ex1.m1.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Ex1.m1.1c">\displaystyle\mathbf{z}_{r}=</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S4.Ex1.3.2.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter" style="font-size:70%;">‘‘Rephrase and expand the question, and respond.’’</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S4.Thmexmp3.p1.2" class="ltx_p"><span id="S4.Thmexmp3.p1.2.1" class="ltx_text" style="font-size:70%;">as the instruction and generate </span><math id="S4.Thmexmp3.p1.2.m1.2" class="ltx_Math" alttext="\mathbf{y}^{\prime}=[\mathbf{x}^{\prime},\mathbf{y}]" display="inline"><semantics id="S4.Thmexmp3.p1.2.m1.2a"><mrow id="S4.Thmexmp3.p1.2.m1.2.2" xref="S4.Thmexmp3.p1.2.m1.2.2.cmml"><msup id="S4.Thmexmp3.p1.2.m1.2.2.3" xref="S4.Thmexmp3.p1.2.m1.2.2.3.cmml"><mi mathsize="70%" id="S4.Thmexmp3.p1.2.m1.2.2.3.2" xref="S4.Thmexmp3.p1.2.m1.2.2.3.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.Thmexmp3.p1.2.m1.2.2.3.3" xref="S4.Thmexmp3.p1.2.m1.2.2.3.3.cmml">′</mo></msup><mo mathsize="70%" id="S4.Thmexmp3.p1.2.m1.2.2.2" xref="S4.Thmexmp3.p1.2.m1.2.2.2.cmml">=</mo><mrow id="S4.Thmexmp3.p1.2.m1.2.2.1.1" xref="S4.Thmexmp3.p1.2.m1.2.2.1.2.cmml"><mo maxsize="70%" minsize="70%" id="S4.Thmexmp3.p1.2.m1.2.2.1.1.2" xref="S4.Thmexmp3.p1.2.m1.2.2.1.2.cmml">[</mo><msup id="S4.Thmexmp3.p1.2.m1.2.2.1.1.1" xref="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.2" xref="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.3" xref="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.3.cmml">′</mo></msup><mo mathsize="70%" id="S4.Thmexmp3.p1.2.m1.2.2.1.1.3" xref="S4.Thmexmp3.p1.2.m1.2.2.1.2.cmml">,</mo><mi mathsize="70%" id="S4.Thmexmp3.p1.2.m1.1.1" xref="S4.Thmexmp3.p1.2.m1.1.1.cmml">𝐲</mi><mo maxsize="70%" minsize="70%" id="S4.Thmexmp3.p1.2.m1.2.2.1.1.4" xref="S4.Thmexmp3.p1.2.m1.2.2.1.2.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp3.p1.2.m1.2b"><apply id="S4.Thmexmp3.p1.2.m1.2.2.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2"><eq id="S4.Thmexmp3.p1.2.m1.2.2.2.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2.2"></eq><apply id="S4.Thmexmp3.p1.2.m1.2.2.3.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2.3"><csymbol cd="ambiguous" id="S4.Thmexmp3.p1.2.m1.2.2.3.1.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2.3">superscript</csymbol><ci id="S4.Thmexmp3.p1.2.m1.2.2.3.2.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2.3.2">𝐲</ci><ci id="S4.Thmexmp3.p1.2.m1.2.2.3.3.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2.3.3">′</ci></apply><interval closure="closed" id="S4.Thmexmp3.p1.2.m1.2.2.1.2.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2.1.1"><apply id="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.1.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2.1.1.1">superscript</csymbol><ci id="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.2.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.2">𝐱</ci><ci id="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.3.cmml" xref="S4.Thmexmp3.p1.2.m1.2.2.1.1.1.3">′</ci></apply><ci id="S4.Thmexmp3.p1.2.m1.1.1.cmml" xref="S4.Thmexmp3.p1.2.m1.1.1">𝐲</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp3.p1.2.m1.2c">\mathbf{y}^{\prime}=[\mathbf{x}^{\prime},\mathbf{y}]</annotation></semantics></math><span id="S4.Thmexmp3.p1.2.2" class="ltx_text" style="font-size:70%;">, where</span></p>
<ul id="S4.I6" class="ltx_itemize">
<li id="S4.I6.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I6.i1.p1" class="ltx_para">
<p id="S4.I6.i1.p1.1" class="ltx_p"><math id="S4.I6.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}^{\prime}=\texttt{``What is the birth year of Barack Obama and is it an even number?''}" display="inline"><semantics id="S4.I6.i1.p1.1.m1.1a"><mrow id="S4.I6.i1.p1.1.m1.1.1" xref="S4.I6.i1.p1.1.m1.1.1.cmml"><msup id="S4.I6.i1.p1.1.m1.1.1.2" xref="S4.I6.i1.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I6.i1.p1.1.m1.1.1.2.2" xref="S4.I6.i1.p1.1.m1.1.1.2.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.I6.i1.p1.1.m1.1.1.2.3" xref="S4.I6.i1.p1.1.m1.1.1.2.3.cmml">′</mo></msup><mo mathsize="70%" id="S4.I6.i1.p1.1.m1.1.1.1" xref="S4.I6.i1.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I6.i1.p1.1.m1.1.1.3" xref="S4.I6.i1.p1.1.m1.1.1.3a.cmml">‘‘What is the birth year of Barack Obama and is it an even number?’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I6.i1.p1.1.m1.1b"><apply id="S4.I6.i1.p1.1.m1.1.1.cmml" xref="S4.I6.i1.p1.1.m1.1.1"><eq id="S4.I6.i1.p1.1.m1.1.1.1.cmml" xref="S4.I6.i1.p1.1.m1.1.1.1"></eq><apply id="S4.I6.i1.p1.1.m1.1.1.2.cmml" xref="S4.I6.i1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I6.i1.p1.1.m1.1.1.2.1.cmml" xref="S4.I6.i1.p1.1.m1.1.1.2">superscript</csymbol><ci id="S4.I6.i1.p1.1.m1.1.1.2.2.cmml" xref="S4.I6.i1.p1.1.m1.1.1.2.2">𝐱</ci><ci id="S4.I6.i1.p1.1.m1.1.1.2.3.cmml" xref="S4.I6.i1.p1.1.m1.1.1.2.3">′</ci></apply><ci id="S4.I6.i1.p1.1.m1.1.1.3a.cmml" xref="S4.I6.i1.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I6.i1.p1.1.m1.1.1.3.cmml" xref="S4.I6.i1.p1.1.m1.1.1.3">‘‘What is the birth year of Barack Obama and is it an even number?’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I6.i1.p1.1.m1.1c">\mathbf{x}^{\prime}=\texttt{``What is the birth year of Barack Obama and is it an even number?''}</annotation></semantics></math><span id="S4.I6.i1.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
<li id="S4.I6.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I6.i2.p1" class="ltx_para">
<p id="S4.I6.i2.p1.1" class="ltx_p"><math id="S4.I6.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}=\texttt{``No''}" display="inline"><semantics id="S4.I6.i2.p1.1.m1.1a"><mrow id="S4.I6.i2.p1.1.m1.1.1" xref="S4.I6.i2.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.I6.i2.p1.1.m1.1.1.2" xref="S4.I6.i2.p1.1.m1.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.I6.i2.p1.1.m1.1.1.1" xref="S4.I6.i2.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I6.i2.p1.1.m1.1.1.3" xref="S4.I6.i2.p1.1.m1.1.1.3a.cmml">‘‘No’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I6.i2.p1.1.m1.1b"><apply id="S4.I6.i2.p1.1.m1.1.1.cmml" xref="S4.I6.i2.p1.1.m1.1.1"><eq id="S4.I6.i2.p1.1.m1.1.1.1.cmml" xref="S4.I6.i2.p1.1.m1.1.1.1"></eq><ci id="S4.I6.i2.p1.1.m1.1.1.2.cmml" xref="S4.I6.i2.p1.1.m1.1.1.2">𝐲</ci><ci id="S4.I6.i2.p1.1.m1.1.1.3a.cmml" xref="S4.I6.i2.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I6.i2.p1.1.m1.1.1.3.cmml" xref="S4.I6.i2.p1.1.m1.1.1.3">‘‘No’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I6.i2.p1.1.m1.1c">\mathbf{y}=\texttt{``No''}</annotation></semantics></math><span id="S4.I6.i2.p1.1.1" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</li>
</ul>
</div>
</div>
<div id="S4.SS1.SSS2.p4" class="ltx_para">
<p id="S4.SS1.SSS2.p4.2" class="ltx_p"><span id="S4.SS1.SSS2.p4.2.1" class="ltx_text" style="font-size:70%;">Unlike CoT outlined in (</span><a href="#S4.E1" title="Equation 4.1 ‣ 4.1.1 Chain-of-Thought (CoT) ‣ 4.1 Mathematical Formulation ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">4.1</span></a><span id="S4.SS1.SSS2.p4.2.2" class="ltx_text" style="font-size:70%;">), which generates numerous intermediate steps </span><math id="S4.SS1.SSS2.p4.1.m1.3" class="ltx_Math" alttext="\bm{s}_{1},\ldots,\bm{s}_{k}" display="inline"><semantics id="S4.SS1.SSS2.p4.1.m1.3a"><mrow id="S4.SS1.SSS2.p4.1.m1.3.3.2" xref="S4.SS1.SSS2.p4.1.m1.3.3.3.cmml"><msub id="S4.SS1.SSS2.p4.1.m1.2.2.1.1" xref="S4.SS1.SSS2.p4.1.m1.2.2.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS2.p4.1.m1.2.2.1.1.2" xref="S4.SS1.SSS2.p4.1.m1.2.2.1.1.2.cmml">𝒔</mi><mn mathsize="70%" id="S4.SS1.SSS2.p4.1.m1.2.2.1.1.3" xref="S4.SS1.SSS2.p4.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.SS1.SSS2.p4.1.m1.3.3.2.3" xref="S4.SS1.SSS2.p4.1.m1.3.3.3.cmml">,</mo><mi mathsize="70%" mathvariant="normal" id="S4.SS1.SSS2.p4.1.m1.1.1" xref="S4.SS1.SSS2.p4.1.m1.1.1.cmml">…</mi><mo mathsize="70%" id="S4.SS1.SSS2.p4.1.m1.3.3.2.4" xref="S4.SS1.SSS2.p4.1.m1.3.3.3.cmml">,</mo><msub id="S4.SS1.SSS2.p4.1.m1.3.3.2.2" xref="S4.SS1.SSS2.p4.1.m1.3.3.2.2.cmml"><mi mathsize="70%" id="S4.SS1.SSS2.p4.1.m1.3.3.2.2.2" xref="S4.SS1.SSS2.p4.1.m1.3.3.2.2.2.cmml">𝒔</mi><mi mathsize="70%" id="S4.SS1.SSS2.p4.1.m1.3.3.2.2.3" xref="S4.SS1.SSS2.p4.1.m1.3.3.2.2.3.cmml">k</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p4.1.m1.3b"><list id="S4.SS1.SSS2.p4.1.m1.3.3.3.cmml" xref="S4.SS1.SSS2.p4.1.m1.3.3.2"><apply id="S4.SS1.SSS2.p4.1.m1.2.2.1.1.cmml" xref="S4.SS1.SSS2.p4.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS2.p4.1.m1.2.2.1.1.1.cmml" xref="S4.SS1.SSS2.p4.1.m1.2.2.1.1">subscript</csymbol><ci id="S4.SS1.SSS2.p4.1.m1.2.2.1.1.2.cmml" xref="S4.SS1.SSS2.p4.1.m1.2.2.1.1.2">𝒔</ci><cn type="integer" id="S4.SS1.SSS2.p4.1.m1.2.2.1.1.3.cmml" xref="S4.SS1.SSS2.p4.1.m1.2.2.1.1.3">1</cn></apply><ci id="S4.SS1.SSS2.p4.1.m1.1.1.cmml" xref="S4.SS1.SSS2.p4.1.m1.1.1">…</ci><apply id="S4.SS1.SSS2.p4.1.m1.3.3.2.2.cmml" xref="S4.SS1.SSS2.p4.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS1.SSS2.p4.1.m1.3.3.2.2.1.cmml" xref="S4.SS1.SSS2.p4.1.m1.3.3.2.2">subscript</csymbol><ci id="S4.SS1.SSS2.p4.1.m1.3.3.2.2.2.cmml" xref="S4.SS1.SSS2.p4.1.m1.3.3.2.2.2">𝒔</ci><ci id="S4.SS1.SSS2.p4.1.m1.3.3.2.2.3.cmml" xref="S4.SS1.SSS2.p4.1.m1.3.3.2.2.3">𝑘</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p4.1.m1.3c">\bm{s}_{1},\ldots,\bm{s}_{k}</annotation></semantics></math><span id="S4.SS1.SSS2.p4.2.3" class="ltx_text" style="font-size:70%;">, RaR in (</span><a href="#S4.E2" title="Equation 4.2 ‣ 4.1.2 One-step RaR ‣ 4.1 Mathematical Formulation ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">4.2</span></a><span id="S4.SS1.SSS2.p4.2.4" class="ltx_text" style="font-size:70%;">) aims to come up with an improved question </span><math id="S4.SS1.SSS2.p4.2.m2.1" class="ltx_Math" alttext="\mathbf{x}^{\prime}" display="inline"><semantics id="S4.SS1.SSS2.p4.2.m2.1a"><msup id="S4.SS1.SSS2.p4.2.m2.1.1" xref="S4.SS1.SSS2.p4.2.m2.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS2.p4.2.m2.1.1.2" xref="S4.SS1.SSS2.p4.2.m2.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.SS1.SSS2.p4.2.m2.1.1.3" xref="S4.SS1.SSS2.p4.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p4.2.m2.1b"><apply id="S4.SS1.SSS2.p4.2.m2.1.1.cmml" xref="S4.SS1.SSS2.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS2.p4.2.m2.1.1.1.cmml" xref="S4.SS1.SSS2.p4.2.m2.1.1">superscript</csymbol><ci id="S4.SS1.SSS2.p4.2.m2.1.1.2.cmml" xref="S4.SS1.SSS2.p4.2.m2.1.1.2">𝐱</ci><ci id="S4.SS1.SSS2.p4.2.m2.1.1.3.cmml" xref="S4.SS1.SSS2.p4.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p4.2.m2.1c">\mathbf{x}^{\prime}</annotation></semantics></math><span id="S4.SS1.SSS2.p4.2.5" class="ltx_text" style="font-size:70%;"> efficiently. In this sense, our method RaR is more cost-effective in terms of token usage than CoT.</span></p>
</div>
</section>
<section id="S4.SS1.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsubsection">4.1.3 </span>Two-step RaR</h4>

<div id="S4.SS1.SSS3.p1" class="ltx_para">
<p id="S4.SS1.SSS3.p1.5" class="ltx_p"><span id="S4.SS1.SSS3.p1.5.1" class="ltx_text" style="font-size:70%;">Instead of generating an extended text </span><math id="S4.SS1.SSS3.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.SS1.SSS3.p1.1.m1.1a"><msup id="S4.SS1.SSS3.p1.1.m1.1.1" xref="S4.SS1.SSS3.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS3.p1.1.m1.1.1.2" xref="S4.SS1.SSS3.p1.1.m1.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.SS1.SSS3.p1.1.m1.1.1.3" xref="S4.SS1.SSS3.p1.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS3.p1.1.m1.1b"><apply id="S4.SS1.SSS3.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS3.p1.1.m1.1.1.1.cmml" xref="S4.SS1.SSS3.p1.1.m1.1.1">superscript</csymbol><ci id="S4.SS1.SSS3.p1.1.m1.1.1.2.cmml" xref="S4.SS1.SSS3.p1.1.m1.1.1.2">𝐲</ci><ci id="S4.SS1.SSS3.p1.1.m1.1.1.3.cmml" xref="S4.SS1.SSS3.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS3.p1.1.m1.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.SS1.SSS3.p1.5.2" class="ltx_text" style="font-size:70%;"> that encompasses the answer </span><math id="S4.SS1.SSS3.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.SS1.SSS3.p1.2.m2.1a"><mi mathsize="70%" id="S4.SS1.SSS3.p1.2.m2.1.1" xref="S4.SS1.SSS3.p1.2.m2.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS3.p1.2.m2.1b"><ci id="S4.SS1.SSS3.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS3.p1.2.m2.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS3.p1.2.m2.1c">\mathbf{y}</annotation></semantics></math><span id="S4.SS1.SSS3.p1.5.3" class="ltx_text" style="font-size:70%;">, the Two-step RaR approach operates in a sequential way. Specifically, it first employs the rephrasing LLM, denoted by </span><math id="S4.SS1.SSS3.p1.3.m3.1" class="ltx_Math" alttext="\text{LLM}_{1}" display="inline"><semantics id="S4.SS1.SSS3.p1.3.m3.1a"><msub id="S4.SS1.SSS3.p1.3.m3.1.1" xref="S4.SS1.SSS3.p1.3.m3.1.1.cmml"><mtext mathsize="70%" id="S4.SS1.SSS3.p1.3.m3.1.1.2" xref="S4.SS1.SSS3.p1.3.m3.1.1.2a.cmml">LLM</mtext><mn mathsize="70%" id="S4.SS1.SSS3.p1.3.m3.1.1.3" xref="S4.SS1.SSS3.p1.3.m3.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS3.p1.3.m3.1b"><apply id="S4.SS1.SSS3.p1.3.m3.1.1.cmml" xref="S4.SS1.SSS3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS3.p1.3.m3.1.1.1.cmml" xref="S4.SS1.SSS3.p1.3.m3.1.1">subscript</csymbol><ci id="S4.SS1.SSS3.p1.3.m3.1.1.2a.cmml" xref="S4.SS1.SSS3.p1.3.m3.1.1.2"><mtext mathsize="70%" id="S4.SS1.SSS3.p1.3.m3.1.1.2.cmml" xref="S4.SS1.SSS3.p1.3.m3.1.1.2">LLM</mtext></ci><cn type="integer" id="S4.SS1.SSS3.p1.3.m3.1.1.3.cmml" xref="S4.SS1.SSS3.p1.3.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS3.p1.3.m3.1c">\text{LLM}_{1}</annotation></semantics></math><span id="S4.SS1.SSS3.p1.5.4" class="ltx_text" style="font-size:70%;"> to generate the rephrased question </span><math id="S4.SS1.SSS3.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{x}^{\prime}" display="inline"><semantics id="S4.SS1.SSS3.p1.4.m4.1a"><msup id="S4.SS1.SSS3.p1.4.m4.1.1" xref="S4.SS1.SSS3.p1.4.m4.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS3.p1.4.m4.1.1.2" xref="S4.SS1.SSS3.p1.4.m4.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.SS1.SSS3.p1.4.m4.1.1.3" xref="S4.SS1.SSS3.p1.4.m4.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS3.p1.4.m4.1b"><apply id="S4.SS1.SSS3.p1.4.m4.1.1.cmml" xref="S4.SS1.SSS3.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS3.p1.4.m4.1.1.1.cmml" xref="S4.SS1.SSS3.p1.4.m4.1.1">superscript</csymbol><ci id="S4.SS1.SSS3.p1.4.m4.1.1.2.cmml" xref="S4.SS1.SSS3.p1.4.m4.1.1.2">𝐱</ci><ci id="S4.SS1.SSS3.p1.4.m4.1.1.3.cmml" xref="S4.SS1.SSS3.p1.4.m4.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS3.p1.4.m4.1c">\mathbf{x}^{\prime}</annotation></semantics></math><span id="S4.SS1.SSS3.p1.5.5" class="ltx_text" style="font-size:70%;">. Then, we input both original and rephrased questions to the responding LLM, denoted by </span><math id="S4.SS1.SSS3.p1.5.m5.1" class="ltx_Math" alttext="\text{LLM}_{2}" display="inline"><semantics id="S4.SS1.SSS3.p1.5.m5.1a"><msub id="S4.SS1.SSS3.p1.5.m5.1.1" xref="S4.SS1.SSS3.p1.5.m5.1.1.cmml"><mtext mathsize="70%" id="S4.SS1.SSS3.p1.5.m5.1.1.2" xref="S4.SS1.SSS3.p1.5.m5.1.1.2a.cmml">LLM</mtext><mn mathsize="70%" id="S4.SS1.SSS3.p1.5.m5.1.1.3" xref="S4.SS1.SSS3.p1.5.m5.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS3.p1.5.m5.1b"><apply id="S4.SS1.SSS3.p1.5.m5.1.1.cmml" xref="S4.SS1.SSS3.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS3.p1.5.m5.1.1.1.cmml" xref="S4.SS1.SSS3.p1.5.m5.1.1">subscript</csymbol><ci id="S4.SS1.SSS3.p1.5.m5.1.1.2a.cmml" xref="S4.SS1.SSS3.p1.5.m5.1.1.2"><mtext mathsize="70%" id="S4.SS1.SSS3.p1.5.m5.1.1.2.cmml" xref="S4.SS1.SSS3.p1.5.m5.1.1.2">LLM</mtext></ci><cn type="integer" id="S4.SS1.SSS3.p1.5.m5.1.1.3.cmml" xref="S4.SS1.SSS3.p1.5.m5.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS3.p1.5.m5.1c">\text{LLM}_{2}</annotation></semantics></math><span id="S4.SS1.SSS3.p1.5.6" class="ltx_text" style="font-size:70%;"> to generate the answer.</span></p>
</div>
<div id="S4.SS1.SSS3.p2" class="ltx_para ltx_noindent">
<svg id="S4.SS1.SSS3.p2.pic1" class="ltx_picture" height="74.01" overflow="visible" version="1.1" width="600"><g transform="translate(0,74.01) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#FCB864" fill-opacity="1.0"><path d="M 0 5.91 L 0 68.1 C 0 71.36 2.64 74.01 5.91 74.01 L 594.09 74.01 C 597.36 74.01 600 71.36 600 68.1 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#FFFBF7" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 53.69 L 598.03 53.69 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 59.6)"><foreignObject width="556.69" height="8.5" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S4.SS1.SSS3.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S4.SS1.SSS3.p2.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S4.SS1.SSS3.p2.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Two-step RaR</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="28.1" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S4.SS1.SSS3.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S4.I7" class="ltx_itemize">
<span id="S4.I7.i1" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I7.i1.p1" class="ltx_para">
<span id="S4.I7.i1.p1.2" class="ltx_p"><span id="S4.I7.i1.p1.2.1" class="ltx_text" style="font-size:70%;">(Rephrase Step) Find an instruction </span><math id="S4.I7.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{z}_{r}" display="inline"><semantics id="S4.I7.i1.p1.1.m1.1a"><msub id="S4.I7.i1.p1.1.m1.1.1" xref="S4.I7.i1.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.I7.i1.p1.1.m1.1.1.2" xref="S4.I7.i1.p1.1.m1.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.I7.i1.p1.1.m1.1.1.3" xref="S4.I7.i1.p1.1.m1.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="S4.I7.i1.p1.1.m1.1b"><apply id="S4.I7.i1.p1.1.m1.1.1.cmml" xref="S4.I7.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.I7.i1.p1.1.m1.1.1.1.cmml" xref="S4.I7.i1.p1.1.m1.1.1">subscript</csymbol><ci id="S4.I7.i1.p1.1.m1.1.1.2.cmml" xref="S4.I7.i1.p1.1.m1.1.1.2">𝐳</ci><ci id="S4.I7.i1.p1.1.m1.1.1.3.cmml" xref="S4.I7.i1.p1.1.m1.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I7.i1.p1.1.m1.1c">\mathbf{z}_{r}</annotation></semantics></math><span id="S4.I7.i1.p1.2.2" class="ltx_text" style="font-size:70%;">, generate rephrased question </span><math id="S4.I7.i1.p1.2.m2.1" class="ltx_math_unparsed" alttext="\mathbf{x}^{\prime}\sim p_{\text{LLM}_{1}}(\cdot|\mathbf{x},\mathbf{z}_{r})" display="inline"><semantics id="S4.I7.i1.p1.2.m2.1a"><mrow id="S4.I7.i1.p1.2.m2.1b"><msup id="S4.I7.i1.p1.2.m2.1.1"><mi mathsize="70%" id="S4.I7.i1.p1.2.m2.1.1.2">𝐱</mi><mo mathsize="70%" id="S4.I7.i1.p1.2.m2.1.1.3">′</mo></msup><mo mathsize="70%" id="S4.I7.i1.p1.2.m2.1.2">∼</mo><msub id="S4.I7.i1.p1.2.m2.1.3"><mi mathsize="70%" id="S4.I7.i1.p1.2.m2.1.3.2">p</mi><msub id="S4.I7.i1.p1.2.m2.1.3.3"><mtext mathsize="70%" id="S4.I7.i1.p1.2.m2.1.3.3.2">LLM</mtext><mn mathsize="70%" id="S4.I7.i1.p1.2.m2.1.3.3.3">1</mn></msub></msub><mrow id="S4.I7.i1.p1.2.m2.1.4"><mo maxsize="70%" minsize="70%" id="S4.I7.i1.p1.2.m2.1.4.1">(</mo><mo lspace="0em" mathsize="70%" rspace="0em" id="S4.I7.i1.p1.2.m2.1.4.2">⋅</mo><mo fence="false" maxsize="70%" minsize="70%" rspace="0.167em" id="S4.I7.i1.p1.2.m2.1.4.3">|</mo><mi mathsize="70%" id="S4.I7.i1.p1.2.m2.1.4.4">𝐱</mi><mo mathsize="70%" id="S4.I7.i1.p1.2.m2.1.4.5">,</mo><msub id="S4.I7.i1.p1.2.m2.1.4.6"><mi mathsize="70%" id="S4.I7.i1.p1.2.m2.1.4.6.2">𝐳</mi><mi mathsize="70%" id="S4.I7.i1.p1.2.m2.1.4.6.3">r</mi></msub><mo maxsize="70%" minsize="70%" id="S4.I7.i1.p1.2.m2.1.4.7">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.I7.i1.p1.2.m2.1c">\mathbf{x}^{\prime}\sim p_{\text{LLM}_{1}}(\cdot|\mathbf{x},\mathbf{z}_{r})</annotation></semantics></math><span id="S4.I7.i1.p1.2.3" class="ltx_text" style="font-size:70%;">.</span></span>
</span></span>
<span id="S4.I7.i2" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I7.i2.p1" class="ltx_para">
<span id="S4.I7.i2.p1.2" class="ltx_p"><span id="S4.I7.i2.p1.2.1" class="ltx_text" style="font-size:70%;">(Respond Step) Generate the desired response </span><math id="S4.I7.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.I7.i2.p1.1.m1.1a"><mi mathsize="70%" id="S4.I7.i2.p1.1.m1.1.1" xref="S4.I7.i2.p1.1.m1.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.I7.i2.p1.1.m1.1b"><ci id="S4.I7.i2.p1.1.m1.1.1.cmml" xref="S4.I7.i2.p1.1.m1.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.I7.i2.p1.1.m1.1c">\mathbf{y}</annotation></semantics></math><span id="S4.I7.i2.p1.2.2" class="ltx_text" style="font-size:70%;"> following </span><math id="S4.I7.i2.p1.2.m2.1" class="ltx_math_unparsed" alttext="\mathbf{y}\sim p_{\text{LLM}_{2}}(\cdot|\mathbf{x},\mathbf{x}^{\prime})" display="inline"><semantics id="S4.I7.i2.p1.2.m2.1a"><mrow id="S4.I7.i2.p1.2.m2.1b"><mi mathsize="70%" id="S4.I7.i2.p1.2.m2.1.1">𝐲</mi><mo mathsize="70%" id="S4.I7.i2.p1.2.m2.1.2">∼</mo><msub id="S4.I7.i2.p1.2.m2.1.3"><mi mathsize="70%" id="S4.I7.i2.p1.2.m2.1.3.2">p</mi><msub id="S4.I7.i2.p1.2.m2.1.3.3"><mtext mathsize="70%" id="S4.I7.i2.p1.2.m2.1.3.3.2">LLM</mtext><mn mathsize="70%" id="S4.I7.i2.p1.2.m2.1.3.3.3">2</mn></msub></msub><mrow id="S4.I7.i2.p1.2.m2.1.4"><mo maxsize="70%" minsize="70%" id="S4.I7.i2.p1.2.m2.1.4.1">(</mo><mo lspace="0em" mathsize="70%" rspace="0em" id="S4.I7.i2.p1.2.m2.1.4.2">⋅</mo><mo fence="false" maxsize="70%" minsize="70%" rspace="0.167em" id="S4.I7.i2.p1.2.m2.1.4.3">|</mo><mi mathsize="70%" id="S4.I7.i2.p1.2.m2.1.4.4">𝐱</mi><mo mathsize="70%" id="S4.I7.i2.p1.2.m2.1.4.5">,</mo><msup id="S4.I7.i2.p1.2.m2.1.4.6"><mi mathsize="70%" id="S4.I7.i2.p1.2.m2.1.4.6.2">𝐱</mi><mo mathsize="70%" id="S4.I7.i2.p1.2.m2.1.4.6.3">′</mo></msup><mo maxsize="70%" minsize="70%" id="S4.I7.i2.p1.2.m2.1.4.7">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.I7.i2.p1.2.m2.1c">\mathbf{y}\sim p_{\text{LLM}_{2}}(\cdot|\mathbf{x},\mathbf{x}^{\prime})</annotation></semantics></math><span id="S4.I7.i2.p1.2.3" class="ltx_text" style="font-size:70%;">.</span></span>
</span></span>
</span>
</span></foreignObject></g></g></svg>
</div>
<div id="S4.SS1.SSS3.p3" class="ltx_para">
<p id="S4.SS1.SSS3.p3.1" class="ltx_p"><span id="S4.SS1.SSS3.p3.1.1" class="ltx_text" style="font-size:70%;">We also give an example for Two-step RaR below.</span></p>
</div>
<div id="S4.Thmexmp4" class="ltx_theorem ltx_theorem_exmp">
<h6 class="ltx_title ltx_runin ltx_title_theorem" style="font-size:70%;">
<span class="ltx_tag ltx_tag_theorem"><span id="S4.Thmexmp4.1.1.1" class="ltx_text ltx_font_bold">Example 4.4</span></span><span id="S4.Thmexmp4.2.2" class="ltx_text ltx_font_bold"> </span>(Two-step RaR)<span id="S4.Thmexmp4.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="S4.Thmexmp4.p1" class="ltx_para">
<p id="S4.Thmexmp4.p1.1" class="ltx_p"><span id="S4.Thmexmp4.p1.1.1" class="ltx_text" style="font-size:70%;">Considering the question </span><math id="S4.Thmexmp4.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}=\texttt{``Was Barack Obama born in an even day?''}" display="inline"><semantics id="S4.Thmexmp4.p1.1.m1.1a"><mrow id="S4.Thmexmp4.p1.1.m1.1.1" xref="S4.Thmexmp4.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp4.p1.1.m1.1.1.2" xref="S4.Thmexmp4.p1.1.m1.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.Thmexmp4.p1.1.m1.1.1.1" xref="S4.Thmexmp4.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp4.p1.1.m1.1.1.3" xref="S4.Thmexmp4.p1.1.m1.1.1.3a.cmml">‘‘Was Barack Obama born in an even day?’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp4.p1.1.m1.1b"><apply id="S4.Thmexmp4.p1.1.m1.1.1.cmml" xref="S4.Thmexmp4.p1.1.m1.1.1"><eq id="S4.Thmexmp4.p1.1.m1.1.1.1.cmml" xref="S4.Thmexmp4.p1.1.m1.1.1.1"></eq><ci id="S4.Thmexmp4.p1.1.m1.1.1.2.cmml" xref="S4.Thmexmp4.p1.1.m1.1.1.2">𝐱</ci><ci id="S4.Thmexmp4.p1.1.m1.1.1.3a.cmml" xref="S4.Thmexmp4.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp4.p1.1.m1.1.1.3.cmml" xref="S4.Thmexmp4.p1.1.m1.1.1.3">‘‘Was Barack Obama born in an even day?’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp4.p1.1.m1.1c">\mathbf{x}=\texttt{``Was Barack Obama born in an even day?''}</annotation></semantics></math><span id="S4.Thmexmp4.p1.1.2" class="ltx_text" style="font-size:70%;">, for Two-step RaR, we have</span></p>
<table id="A2.EGx7" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.Ex2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.Ex2.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{z}_{r}=" display="inline"><semantics id="S4.Ex2.m1.1a"><mrow id="S4.Ex2.m1.1.1" xref="S4.Ex2.m1.1.1.cmml"><msub id="S4.Ex2.m1.1.1.2" xref="S4.Ex2.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.Ex2.m1.1.1.2.2" xref="S4.Ex2.m1.1.1.2.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.Ex2.m1.1.1.2.3" xref="S4.Ex2.m1.1.1.2.3.cmml">r</mi></msub><mo mathsize="70%" id="S4.Ex2.m1.1.1.1" xref="S4.Ex2.m1.1.1.1.cmml">=</mo><mi id="S4.Ex2.m1.1.1.3" xref="S4.Ex2.m1.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="S4.Ex2.m1.1b"><apply id="S4.Ex2.m1.1.1.cmml" xref="S4.Ex2.m1.1.1"><eq id="S4.Ex2.m1.1.1.1.cmml" xref="S4.Ex2.m1.1.1.1"></eq><apply id="S4.Ex2.m1.1.1.2.cmml" xref="S4.Ex2.m1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex2.m1.1.1.2.1.cmml" xref="S4.Ex2.m1.1.1.2">subscript</csymbol><ci id="S4.Ex2.m1.1.1.2.2.cmml" xref="S4.Ex2.m1.1.1.2.2">𝐳</ci><ci id="S4.Ex2.m1.1.1.2.3.cmml" xref="S4.Ex2.m1.1.1.2.3">𝑟</ci></apply><csymbol cd="latexml" id="S4.Ex2.m1.1.1.3.cmml" xref="S4.Ex2.m1.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Ex2.m1.1c">\displaystyle\mathbf{z}_{r}=</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S4.Ex2.3.2.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter" style="font-size:70%;">‘‘Given the above question, rephrase and expand it to help you</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S4.Ex3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S4.Ex3.4.3.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter" style="font-size:70%;">do better answering. Maintain all information in the original question.’’</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S4.Thmexmp4.p2" class="ltx_para">
<p id="S4.Thmexmp4.p2.3" class="ltx_p"><span id="S4.Thmexmp4.p2.3.1" class="ltx_text" style="font-size:70%;">as the instruction and generate</span></p>
<table id="A2.EGx8" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.Ex4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.Ex4.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{x}^{\prime}=\texttt{``What is the birth year of Barack Obama and is it an even number?''}" display="inline"><semantics id="S4.Ex4.m1.1a"><mrow id="S4.Ex4.m1.1.1" xref="S4.Ex4.m1.1.1.cmml"><msup id="S4.Ex4.m1.1.1.2" xref="S4.Ex4.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.Ex4.m1.1.1.2.2" xref="S4.Ex4.m1.1.1.2.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.Ex4.m1.1.1.2.3" xref="S4.Ex4.m1.1.1.2.3.cmml">′</mo></msup><mo mathsize="70%" id="S4.Ex4.m1.1.1.1" xref="S4.Ex4.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Ex4.m1.1.1.3" xref="S4.Ex4.m1.1.1.3a.cmml">‘‘What is the birth year of Barack Obama and is it an even number?’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.Ex4.m1.1b"><apply id="S4.Ex4.m1.1.1.cmml" xref="S4.Ex4.m1.1.1"><eq id="S4.Ex4.m1.1.1.1.cmml" xref="S4.Ex4.m1.1.1.1"></eq><apply id="S4.Ex4.m1.1.1.2.cmml" xref="S4.Ex4.m1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex4.m1.1.1.2.1.cmml" xref="S4.Ex4.m1.1.1.2">superscript</csymbol><ci id="S4.Ex4.m1.1.1.2.2.cmml" xref="S4.Ex4.m1.1.1.2.2">𝐱</ci><ci id="S4.Ex4.m1.1.1.2.3.cmml" xref="S4.Ex4.m1.1.1.2.3">′</ci></apply><ci id="S4.Ex4.m1.1.1.3a.cmml" xref="S4.Ex4.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Ex4.m1.1.1.3.cmml" xref="S4.Ex4.m1.1.1.3">‘‘What is the birth year of Barack Obama and is it an even number?’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Ex4.m1.1c">\displaystyle\mathbf{x}^{\prime}=\texttt{``What is the birth year of Barack Obama and is it an even number?''}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S4.Thmexmp4.p2.2" class="ltx_p"><span id="S4.Thmexmp4.p2.2.1" class="ltx_text" style="font-size:70%;">Then we feed </span><math id="S4.Thmexmp4.p2.1.m1.2" class="ltx_Math" alttext="[\mathbf{x},\mathbf{x}^{\prime}]" display="inline"><semantics id="S4.Thmexmp4.p2.1.m1.2a"><mrow id="S4.Thmexmp4.p2.1.m1.2.2.1" xref="S4.Thmexmp4.p2.1.m1.2.2.2.cmml"><mo maxsize="70%" minsize="70%" id="S4.Thmexmp4.p2.1.m1.2.2.1.2" xref="S4.Thmexmp4.p2.1.m1.2.2.2.cmml">[</mo><mi mathsize="70%" id="S4.Thmexmp4.p2.1.m1.1.1" xref="S4.Thmexmp4.p2.1.m1.1.1.cmml">𝐱</mi><mo mathsize="70%" id="S4.Thmexmp4.p2.1.m1.2.2.1.3" xref="S4.Thmexmp4.p2.1.m1.2.2.2.cmml">,</mo><msup id="S4.Thmexmp4.p2.1.m1.2.2.1.1" xref="S4.Thmexmp4.p2.1.m1.2.2.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp4.p2.1.m1.2.2.1.1.2" xref="S4.Thmexmp4.p2.1.m1.2.2.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.Thmexmp4.p2.1.m1.2.2.1.1.3" xref="S4.Thmexmp4.p2.1.m1.2.2.1.1.3.cmml">′</mo></msup><mo maxsize="70%" minsize="70%" id="S4.Thmexmp4.p2.1.m1.2.2.1.4" xref="S4.Thmexmp4.p2.1.m1.2.2.2.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp4.p2.1.m1.2b"><interval closure="closed" id="S4.Thmexmp4.p2.1.m1.2.2.2.cmml" xref="S4.Thmexmp4.p2.1.m1.2.2.1"><ci id="S4.Thmexmp4.p2.1.m1.1.1.cmml" xref="S4.Thmexmp4.p2.1.m1.1.1">𝐱</ci><apply id="S4.Thmexmp4.p2.1.m1.2.2.1.1.cmml" xref="S4.Thmexmp4.p2.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.Thmexmp4.p2.1.m1.2.2.1.1.1.cmml" xref="S4.Thmexmp4.p2.1.m1.2.2.1.1">superscript</csymbol><ci id="S4.Thmexmp4.p2.1.m1.2.2.1.1.2.cmml" xref="S4.Thmexmp4.p2.1.m1.2.2.1.1.2">𝐱</ci><ci id="S4.Thmexmp4.p2.1.m1.2.2.1.1.3.cmml" xref="S4.Thmexmp4.p2.1.m1.2.2.1.1.3">′</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp4.p2.1.m1.2c">[\mathbf{x},\mathbf{x}^{\prime}]</annotation></semantics></math><span id="S4.Thmexmp4.p2.2.2" class="ltx_text" style="font-size:70%;"> together into the LLM and get </span><math id="S4.Thmexmp4.p2.2.m2.1" class="ltx_Math" alttext="\mathbf{y}=\texttt{``No''}" display="inline"><semantics id="S4.Thmexmp4.p2.2.m2.1a"><mrow id="S4.Thmexmp4.p2.2.m2.1.1" xref="S4.Thmexmp4.p2.2.m2.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp4.p2.2.m2.1.1.2" xref="S4.Thmexmp4.p2.2.m2.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.Thmexmp4.p2.2.m2.1.1.1" xref="S4.Thmexmp4.p2.2.m2.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp4.p2.2.m2.1.1.3" xref="S4.Thmexmp4.p2.2.m2.1.1.3a.cmml">‘‘No’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp4.p2.2.m2.1b"><apply id="S4.Thmexmp4.p2.2.m2.1.1.cmml" xref="S4.Thmexmp4.p2.2.m2.1.1"><eq id="S4.Thmexmp4.p2.2.m2.1.1.1.cmml" xref="S4.Thmexmp4.p2.2.m2.1.1.1"></eq><ci id="S4.Thmexmp4.p2.2.m2.1.1.2.cmml" xref="S4.Thmexmp4.p2.2.m2.1.1.2">𝐲</ci><ci id="S4.Thmexmp4.p2.2.m2.1.1.3a.cmml" xref="S4.Thmexmp4.p2.2.m2.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.Thmexmp4.p2.2.m2.1.1.3.cmml" xref="S4.Thmexmp4.p2.2.m2.1.1.3">‘‘No’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp4.p2.2.m2.1c">\mathbf{y}=\texttt{``No''}</annotation></semantics></math><span id="S4.Thmexmp4.p2.2.3" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</div>
<div id="S4.SS1.SSS3.p4" class="ltx_para">
<p id="S4.SS1.SSS3.p4.1" class="ltx_p"><span id="S4.SS1.SSS3.p4.1.1" class="ltx_text" style="font-size:70%;">In our experiments, we find that Two-step RaR can consistently achieve better performance. The rephrased question can also be used by another LLM, which makes our RaR method more flexible.</span></p>
</div>
</section>
<section id="S4.SS1.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsubsection">4.1.4 </span>Combining RaR and CoT</h4>

<div id="S4.SS1.SSS4.p1" class="ltx_para">
<p id="S4.SS1.SSS4.p1.2" class="ltx_p"><span id="S4.SS1.SSS4.p1.2.1" class="ltx_text" style="font-size:70%;">In addition, our method is complementary to CoT and can be naturally combined with CoT. For zero-shot CoT, we can simply concatenate the two instructions to obtain </span><math id="S4.SS1.SSS4.p1.1.m1.2" class="ltx_Math" alttext="\mathbf{z}_{r}^{\prime}=[\mathbf{z}_{r},\mathbf{z}_{c}]" display="inline"><semantics id="S4.SS1.SSS4.p1.1.m1.2a"><mrow id="S4.SS1.SSS4.p1.1.m1.2.2" xref="S4.SS1.SSS4.p1.1.m1.2.2.cmml"><msubsup id="S4.SS1.SSS4.p1.1.m1.2.2.4" xref="S4.SS1.SSS4.p1.1.m1.2.2.4.cmml"><mi mathsize="70%" id="S4.SS1.SSS4.p1.1.m1.2.2.4.2.2" xref="S4.SS1.SSS4.p1.1.m1.2.2.4.2.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.SS1.SSS4.p1.1.m1.2.2.4.2.3" xref="S4.SS1.SSS4.p1.1.m1.2.2.4.2.3.cmml">r</mi><mo mathsize="70%" id="S4.SS1.SSS4.p1.1.m1.2.2.4.3" xref="S4.SS1.SSS4.p1.1.m1.2.2.4.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.SS1.SSS4.p1.1.m1.2.2.3" xref="S4.SS1.SSS4.p1.1.m1.2.2.3.cmml">=</mo><mrow id="S4.SS1.SSS4.p1.1.m1.2.2.2.2" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.3.cmml"><mo maxsize="70%" minsize="70%" id="S4.SS1.SSS4.p1.1.m1.2.2.2.2.3" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.3.cmml">[</mo><msub id="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1" xref="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.2" xref="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.3" xref="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.3.cmml">r</mi></msub><mo mathsize="70%" id="S4.SS1.SSS4.p1.1.m1.2.2.2.2.4" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.3.cmml">,</mo><msub id="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.cmml"><mi mathsize="70%" id="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.2" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.3" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.3.cmml">c</mi></msub><mo maxsize="70%" minsize="70%" id="S4.SS1.SSS4.p1.1.m1.2.2.2.2.5" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS4.p1.1.m1.2b"><apply id="S4.SS1.SSS4.p1.1.m1.2.2.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2"><eq id="S4.SS1.SSS4.p1.1.m1.2.2.3.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.3"></eq><apply id="S4.SS1.SSS4.p1.1.m1.2.2.4.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.4"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.1.m1.2.2.4.1.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.4">superscript</csymbol><apply id="S4.SS1.SSS4.p1.1.m1.2.2.4.2.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.4"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.1.m1.2.2.4.2.1.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.4">subscript</csymbol><ci id="S4.SS1.SSS4.p1.1.m1.2.2.4.2.2.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.4.2.2">𝐳</ci><ci id="S4.SS1.SSS4.p1.1.m1.2.2.4.2.3.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.4.2.3">𝑟</ci></apply><ci id="S4.SS1.SSS4.p1.1.m1.2.2.4.3.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.4.3">′</ci></apply><interval closure="closed" id="S4.SS1.SSS4.p1.1.m1.2.2.2.3.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.2"><apply id="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.cmml" xref="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.1.cmml" xref="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1">subscript</csymbol><ci id="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.2.cmml" xref="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.2">𝐳</ci><ci id="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.3.cmml" xref="S4.SS1.SSS4.p1.1.m1.1.1.1.1.1.3">𝑟</ci></apply><apply id="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.1.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2">subscript</csymbol><ci id="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.2.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.2">𝐳</ci><ci id="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.3.cmml" xref="S4.SS1.SSS4.p1.1.m1.2.2.2.2.2.3">𝑐</ci></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS4.p1.1.m1.2c">\mathbf{z}_{r}^{\prime}=[\mathbf{z}_{r},\mathbf{z}_{c}]</annotation></semantics></math><span id="S4.SS1.SSS4.p1.2.2" class="ltx_text" style="font-size:70%;">, such as “Given the above question, rephrase and expand it to help you
do better answering. Lastly, let’s think step by step to answer”.
For few-shot CoT, where the instruction/context is </span><math id="S4.SS1.SSS4.p1.2.m2.7" class="ltx_Math" alttext="\mathbf{z}_{c}=[\mathbf{x}_{1},\mathbf{y}_{1}^{\prime},\mathbf{x}_{2},\mathbf{y}_{2}^{\prime},\ldots,\mathbf{x}_{k},\mathbf{y}_{k}^{\prime}]" display="inline"><semantics id="S4.SS1.SSS4.p1.2.m2.7a"><mrow id="S4.SS1.SSS4.p1.2.m2.7.7" xref="S4.SS1.SSS4.p1.2.m2.7.7.cmml"><msub id="S4.SS1.SSS4.p1.2.m2.7.7.8" xref="S4.SS1.SSS4.p1.2.m2.7.7.8.cmml"><mi mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.8.2" xref="S4.SS1.SSS4.p1.2.m2.7.7.8.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.8.3" xref="S4.SS1.SSS4.p1.2.m2.7.7.8.3.cmml">c</mi></msub><mo mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.7" xref="S4.SS1.SSS4.p1.2.m2.7.7.7.cmml">=</mo><mrow id="S4.SS1.SSS4.p1.2.m2.7.7.6.6" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.7.cmml"><mo maxsize="70%" minsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.7" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.7.cmml">[</mo><msub id="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1" xref="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.cmml"><mi mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.2" xref="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.3" xref="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.8" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.7.cmml">,</mo><msubsup id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.cmml"><mi mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.2.2" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.2.3" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.2.3.cmml">1</mn><mo mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.3" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.9" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.7.cmml">,</mo><msub id="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3" xref="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.cmml"><mi mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.2" xref="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.3" xref="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.3.cmml">2</mn></msub><mo mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.10" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.7.cmml">,</mo><msubsup id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.cmml"><mi mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.2.2" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.2.3" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.2.3.cmml">2</mn><mo mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.3" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.11" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.7.cmml">,</mo><mi mathsize="70%" mathvariant="normal" id="S4.SS1.SSS4.p1.2.m2.1.1" xref="S4.SS1.SSS4.p1.2.m2.1.1.cmml">…</mi><mo mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.12" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.7.cmml">,</mo><msub id="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5" xref="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.cmml"><mi mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.2" xref="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.2.cmml">𝐱</mi><mi mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.3" xref="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.3.cmml">k</mi></msub><mo mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.13" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.7.cmml">,</mo><msubsup id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.cmml"><mi mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.2.2" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.2.2.cmml">𝐲</mi><mi mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.2.3" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.2.3.cmml">k</mi><mo mathsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.3" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.3.cmml">′</mo></msubsup><mo maxsize="70%" minsize="70%" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.14" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.7.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS4.p1.2.m2.7b"><apply id="S4.SS1.SSS4.p1.2.m2.7.7.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7"><eq id="S4.SS1.SSS4.p1.2.m2.7.7.7.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.7"></eq><apply id="S4.SS1.SSS4.p1.2.m2.7.7.8.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.8"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.2.m2.7.7.8.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.8">subscript</csymbol><ci id="S4.SS1.SSS4.p1.2.m2.7.7.8.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.8.2">𝐳</ci><ci id="S4.SS1.SSS4.p1.2.m2.7.7.8.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.8.3">𝑐</ci></apply><list id="S4.SS1.SSS4.p1.2.m2.7.7.6.7.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6"><apply id="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1">subscript</csymbol><ci id="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.2">𝐱</ci><cn type="integer" id="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.2.2.1.1.1.3">1</cn></apply><apply id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2">superscript</csymbol><apply id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.2.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2">subscript</csymbol><ci id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.2.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.2.2">𝐲</ci><cn type="integer" id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.2.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.2.3">1</cn></apply><ci id="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.3.3.2.2.2.3">′</ci></apply><apply id="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3">subscript</csymbol><ci id="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.2">𝐱</ci><cn type="integer" id="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.4.4.3.3.3.3">2</cn></apply><apply id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.cmml" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4">superscript</csymbol><apply id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.2.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4">subscript</csymbol><ci id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.2.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.2.2">𝐲</ci><cn type="integer" id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.2.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.2.3">2</cn></apply><ci id="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.5.5.4.4.4.3">′</ci></apply><ci id="S4.SS1.SSS4.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.1.1">…</ci><apply id="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.cmml" xref="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5">subscript</csymbol><ci id="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.2">𝐱</ci><ci id="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.6.6.5.5.5.3">𝑘</ci></apply><apply id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6">superscript</csymbol><apply id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6"><csymbol cd="ambiguous" id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.2.1.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6">subscript</csymbol><ci id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.2.2.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.2.2">𝐲</ci><ci id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.2.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.2.3">𝑘</ci></apply><ci id="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.3.cmml" xref="S4.SS1.SSS4.p1.2.m2.7.7.6.6.6.3">′</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS4.p1.2.m2.7c">\mathbf{z}_{c}=[\mathbf{x}_{1},\mathbf{y}_{1}^{\prime},\mathbf{x}_{2},\mathbf{y}_{2}^{\prime},\ldots,\mathbf{x}_{k},\mathbf{y}_{k}^{\prime}]</annotation></semantics></math><span id="S4.SS1.SSS4.p1.2.3" class="ltx_text" style="font-size:70%;">, we can use Two-step RaR to improve its few-shot examples by the following procedure.</span></p>
</div>
<div id="S4.SS1.SSS4.p2" class="ltx_para ltx_noindent">
<svg id="S4.SS1.SSS4.p2.pic1" class="ltx_picture" height="71.42" overflow="visible" version="1.1" width="600"><g transform="translate(0,71.42) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#FCB864" fill-opacity="1.0"><path d="M 0 5.91 L 0 65.52 C 0 68.78 2.64 71.42 5.91 71.42 L 594.09 71.42 C 597.36 71.42 600 68.78 600 65.52 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#FFFBF7" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 52.19 L 598.03 52.19 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 58.09)"><foreignObject width="556.69" height="7.43" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#FFFFFF">
<span id="S4.SS1.SSS4.p2.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S4.SS1.SSS4.p2.pic1.1.1.1.1.1.1" class="ltx_p"><span id="S4.SS1.SSS4.p2.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">RaR + CoT</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignObject width="556.69" height="26.6" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="S4.SS1.SSS4.p2.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:402.3pt;">
<span id="S4.I8" class="ltx_itemize">
<span id="S4.I8.i1" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I8.i1.p1" class="ltx_para">
<span id="S4.I8.i1.p1.2" class="ltx_p"><span id="S4.I8.i1.p1.2.1" class="ltx_text" style="font-size:70%;">Use RaR instruction </span><math id="S4.I8.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{z}_{r}" display="inline"><semantics id="S4.I8.i1.p1.1.m1.1a"><msub id="S4.I8.i1.p1.1.m1.1.1" xref="S4.I8.i1.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.I8.i1.p1.1.m1.1.1.2" xref="S4.I8.i1.p1.1.m1.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.I8.i1.p1.1.m1.1.1.3" xref="S4.I8.i1.p1.1.m1.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="S4.I8.i1.p1.1.m1.1b"><apply id="S4.I8.i1.p1.1.m1.1.1.cmml" xref="S4.I8.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.I8.i1.p1.1.m1.1.1.1.cmml" xref="S4.I8.i1.p1.1.m1.1.1">subscript</csymbol><ci id="S4.I8.i1.p1.1.m1.1.1.2.cmml" xref="S4.I8.i1.p1.1.m1.1.1.2">𝐳</ci><ci id="S4.I8.i1.p1.1.m1.1.1.3.cmml" xref="S4.I8.i1.p1.1.m1.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I8.i1.p1.1.m1.1c">\mathbf{z}_{r}</annotation></semantics></math><span id="S4.I8.i1.p1.2.2" class="ltx_text" style="font-size:70%;">, generate </span><math id="S4.I8.i1.p1.2.m2.1" class="ltx_math_unparsed" alttext="\mathbf{z}_{c}^{\prime}\sim p_{\text{LLM}}(\cdot|\mathbf{z}_{c},\mathbf{z}_{r})" display="inline"><semantics id="S4.I8.i1.p1.2.m2.1a"><mrow id="S4.I8.i1.p1.2.m2.1b"><msubsup id="S4.I8.i1.p1.2.m2.1.1"><mi mathsize="70%" id="S4.I8.i1.p1.2.m2.1.1.2.2">𝐳</mi><mi mathsize="70%" id="S4.I8.i1.p1.2.m2.1.1.2.3">c</mi><mo mathsize="70%" id="S4.I8.i1.p1.2.m2.1.1.3">′</mo></msubsup><mo mathsize="70%" id="S4.I8.i1.p1.2.m2.1.2">∼</mo><msub id="S4.I8.i1.p1.2.m2.1.3"><mi mathsize="70%" id="S4.I8.i1.p1.2.m2.1.3.2">p</mi><mtext mathsize="70%" id="S4.I8.i1.p1.2.m2.1.3.3">LLM</mtext></msub><mrow id="S4.I8.i1.p1.2.m2.1.4"><mo maxsize="70%" minsize="70%" id="S4.I8.i1.p1.2.m2.1.4.1">(</mo><mo lspace="0em" mathsize="70%" rspace="0em" id="S4.I8.i1.p1.2.m2.1.4.2">⋅</mo><mo fence="false" maxsize="70%" minsize="70%" rspace="0.167em" id="S4.I8.i1.p1.2.m2.1.4.3">|</mo><msub id="S4.I8.i1.p1.2.m2.1.4.4"><mi mathsize="70%" id="S4.I8.i1.p1.2.m2.1.4.4.2">𝐳</mi><mi mathsize="70%" id="S4.I8.i1.p1.2.m2.1.4.4.3">c</mi></msub><mo mathsize="70%" id="S4.I8.i1.p1.2.m2.1.4.5">,</mo><msub id="S4.I8.i1.p1.2.m2.1.4.6"><mi mathsize="70%" id="S4.I8.i1.p1.2.m2.1.4.6.2">𝐳</mi><mi mathsize="70%" id="S4.I8.i1.p1.2.m2.1.4.6.3">r</mi></msub><mo maxsize="70%" minsize="70%" id="S4.I8.i1.p1.2.m2.1.4.7">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.I8.i1.p1.2.m2.1c">\mathbf{z}_{c}^{\prime}\sim p_{\text{LLM}}(\cdot|\mathbf{z}_{c},\mathbf{z}_{r})</annotation></semantics></math><span id="S4.I8.i1.p1.2.3" class="ltx_text" style="font-size:70%;"> in the Rephrase step.</span></span>
</span></span>
<span id="S4.I8.i2" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I8.i2.p1" class="ltx_para">
<span id="S4.I8.i2.p1.2" class="ltx_p"><span id="S4.I8.i2.p1.2.1" class="ltx_text" style="font-size:70%;">Apply </span><math id="S4.I8.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{z}_{c}^{\prime}" display="inline"><semantics id="S4.I8.i2.p1.1.m1.1a"><msubsup id="S4.I8.i2.p1.1.m1.1.1" xref="S4.I8.i2.p1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.I8.i2.p1.1.m1.1.1.2.2" xref="S4.I8.i2.p1.1.m1.1.1.2.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.I8.i2.p1.1.m1.1.1.2.3" xref="S4.I8.i2.p1.1.m1.1.1.2.3.cmml">c</mi><mo mathsize="70%" id="S4.I8.i2.p1.1.m1.1.1.3" xref="S4.I8.i2.p1.1.m1.1.1.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="S4.I8.i2.p1.1.m1.1b"><apply id="S4.I8.i2.p1.1.m1.1.1.cmml" xref="S4.I8.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.I8.i2.p1.1.m1.1.1.1.cmml" xref="S4.I8.i2.p1.1.m1.1.1">superscript</csymbol><apply id="S4.I8.i2.p1.1.m1.1.1.2.cmml" xref="S4.I8.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.I8.i2.p1.1.m1.1.1.2.1.cmml" xref="S4.I8.i2.p1.1.m1.1.1">subscript</csymbol><ci id="S4.I8.i2.p1.1.m1.1.1.2.2.cmml" xref="S4.I8.i2.p1.1.m1.1.1.2.2">𝐳</ci><ci id="S4.I8.i2.p1.1.m1.1.1.2.3.cmml" xref="S4.I8.i2.p1.1.m1.1.1.2.3">𝑐</ci></apply><ci id="S4.I8.i2.p1.1.m1.1.1.3.cmml" xref="S4.I8.i2.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I8.i2.p1.1.m1.1c">\mathbf{z}_{c}^{\prime}</annotation></semantics></math><span id="S4.I8.i2.p1.2.2" class="ltx_text" style="font-size:70%;"> to get </span><math id="S4.I8.i2.p1.2.m2.1" class="ltx_math_unparsed" alttext="\mathbf{y}^{\prime}\sim p_{\text{LLM}}(\cdot|\mathbf{z}_{c}^{\prime},\mathbf{x})" display="inline"><semantics id="S4.I8.i2.p1.2.m2.1a"><mrow id="S4.I8.i2.p1.2.m2.1b"><msup id="S4.I8.i2.p1.2.m2.1.1"><mi mathsize="70%" id="S4.I8.i2.p1.2.m2.1.1.2">𝐲</mi><mo mathsize="70%" id="S4.I8.i2.p1.2.m2.1.1.3">′</mo></msup><mo mathsize="70%" id="S4.I8.i2.p1.2.m2.1.2">∼</mo><msub id="S4.I8.i2.p1.2.m2.1.3"><mi mathsize="70%" id="S4.I8.i2.p1.2.m2.1.3.2">p</mi><mtext mathsize="70%" id="S4.I8.i2.p1.2.m2.1.3.3">LLM</mtext></msub><mrow id="S4.I8.i2.p1.2.m2.1.4"><mo maxsize="70%" minsize="70%" id="S4.I8.i2.p1.2.m2.1.4.1">(</mo><mo lspace="0em" mathsize="70%" rspace="0em" id="S4.I8.i2.p1.2.m2.1.4.2">⋅</mo><mo fence="false" maxsize="70%" minsize="70%" rspace="0.167em" id="S4.I8.i2.p1.2.m2.1.4.3">|</mo><msubsup id="S4.I8.i2.p1.2.m2.1.4.4"><mi mathsize="70%" id="S4.I8.i2.p1.2.m2.1.4.4.2.2">𝐳</mi><mi mathsize="70%" id="S4.I8.i2.p1.2.m2.1.4.4.2.3">c</mi><mo mathsize="70%" id="S4.I8.i2.p1.2.m2.1.4.4.3">′</mo></msubsup><mo mathsize="70%" id="S4.I8.i2.p1.2.m2.1.4.5">,</mo><mi mathsize="70%" id="S4.I8.i2.p1.2.m2.1.4.6">𝐱</mi><mo maxsize="70%" minsize="70%" id="S4.I8.i2.p1.2.m2.1.4.7">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.I8.i2.p1.2.m2.1c">\mathbf{y}^{\prime}\sim p_{\text{LLM}}(\cdot|\mathbf{z}_{c}^{\prime},\mathbf{x})</annotation></semantics></math><span id="S4.I8.i2.p1.2.3" class="ltx_text" style="font-size:70%;">.</span></span>
</span></span>
<span id="S4.I8.i3" class="ltx_item" style="list-style-type:none;"><span class="ltx_tag ltx_tag_item">•</span> 
<span id="S4.I8.i3.p1" class="ltx_para">
<span id="S4.I8.i3.p1.2" class="ltx_p"><span id="S4.I8.i3.p1.2.1" class="ltx_text" style="font-size:70%;">Extract the response </span><math id="S4.I8.i3.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.I8.i3.p1.1.m1.1a"><mi mathsize="70%" id="S4.I8.i3.p1.1.m1.1.1" xref="S4.I8.i3.p1.1.m1.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.I8.i3.p1.1.m1.1b"><ci id="S4.I8.i3.p1.1.m1.1.1.cmml" xref="S4.I8.i3.p1.1.m1.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.I8.i3.p1.1.m1.1c">\mathbf{y}</annotation></semantics></math><span id="S4.I8.i3.p1.2.2" class="ltx_text" style="font-size:70%;"> from the sequence </span><math id="S4.I8.i3.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{y}^{\prime}" display="inline"><semantics id="S4.I8.i3.p1.2.m2.1a"><msup id="S4.I8.i3.p1.2.m2.1.1" xref="S4.I8.i3.p1.2.m2.1.1.cmml"><mi mathsize="70%" id="S4.I8.i3.p1.2.m2.1.1.2" xref="S4.I8.i3.p1.2.m2.1.1.2.cmml">𝐲</mi><mo mathsize="70%" id="S4.I8.i3.p1.2.m2.1.1.3" xref="S4.I8.i3.p1.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.I8.i3.p1.2.m2.1b"><apply id="S4.I8.i3.p1.2.m2.1.1.cmml" xref="S4.I8.i3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.I8.i3.p1.2.m2.1.1.1.cmml" xref="S4.I8.i3.p1.2.m2.1.1">superscript</csymbol><ci id="S4.I8.i3.p1.2.m2.1.1.2.cmml" xref="S4.I8.i3.p1.2.m2.1.1.2">𝐲</ci><ci id="S4.I8.i3.p1.2.m2.1.1.3.cmml" xref="S4.I8.i3.p1.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I8.i3.p1.2.m2.1c">\mathbf{y}^{\prime}</annotation></semantics></math><span id="S4.I8.i3.p1.2.3" class="ltx_text" style="font-size:70%;"> by eliminating intermediate steps.</span></span>
</span></span>
</span>
</span></foreignObject></g></g></svg>
</div>
<div id="S4.Thmtheorem1" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem" style="font-size:70%;">
<span class="ltx_tag ltx_tag_theorem"><span id="S4.Thmtheorem1.1.1.1" class="ltx_text ltx_font_bold">Remark 4.1</span></span><span id="S4.Thmtheorem1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="S4.Thmtheorem1.p1" class="ltx_para">
<p id="S4.Thmtheorem1.p1.4" class="ltx_p"><span id="S4.Thmtheorem1.p1.4.1" class="ltx_text" style="font-size:70%;">Compared with Two-step RaR, which uses both the original question </span><math id="S4.Thmtheorem1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}" display="inline"><semantics id="S4.Thmtheorem1.p1.1.m1.1a"><mi mathsize="70%" id="S4.Thmtheorem1.p1.1.m1.1.1" xref="S4.Thmtheorem1.p1.1.m1.1.1.cmml">𝐱</mi><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.1.m1.1b"><ci id="S4.Thmtheorem1.p1.1.m1.1.1.cmml" xref="S4.Thmtheorem1.p1.1.m1.1.1">𝐱</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.1.m1.1c">\mathbf{x}</annotation></semantics></math><span id="S4.Thmtheorem1.p1.4.2" class="ltx_text" style="font-size:70%;"> and the rephrased question </span><math id="S4.Thmtheorem1.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{x}^{\prime}" display="inline"><semantics id="S4.Thmtheorem1.p1.2.m2.1a"><msup id="S4.Thmtheorem1.p1.2.m2.1.1" xref="S4.Thmtheorem1.p1.2.m2.1.1.cmml"><mi mathsize="70%" id="S4.Thmtheorem1.p1.2.m2.1.1.2" xref="S4.Thmtheorem1.p1.2.m2.1.1.2.cmml">𝐱</mi><mo mathsize="70%" id="S4.Thmtheorem1.p1.2.m2.1.1.3" xref="S4.Thmtheorem1.p1.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.2.m2.1b"><apply id="S4.Thmtheorem1.p1.2.m2.1.1.cmml" xref="S4.Thmtheorem1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.Thmtheorem1.p1.2.m2.1.1.1.cmml" xref="S4.Thmtheorem1.p1.2.m2.1.1">superscript</csymbol><ci id="S4.Thmtheorem1.p1.2.m2.1.1.2.cmml" xref="S4.Thmtheorem1.p1.2.m2.1.1.2">𝐱</ci><ci id="S4.Thmtheorem1.p1.2.m2.1.1.3.cmml" xref="S4.Thmtheorem1.p1.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.2.m2.1c">\mathbf{x}^{\prime}</annotation></semantics></math><span id="S4.Thmtheorem1.p1.4.3" class="ltx_text" style="font-size:70%;"> to prompt the LLM to generate the response, RaR+CoT only uses the rephrased few-shot examples </span><math id="S4.Thmtheorem1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{z}_{c}^{\prime}" display="inline"><semantics id="S4.Thmtheorem1.p1.3.m3.1a"><msubsup id="S4.Thmtheorem1.p1.3.m3.1.1" xref="S4.Thmtheorem1.p1.3.m3.1.1.cmml"><mi mathsize="70%" id="S4.Thmtheorem1.p1.3.m3.1.1.2.2" xref="S4.Thmtheorem1.p1.3.m3.1.1.2.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.Thmtheorem1.p1.3.m3.1.1.2.3" xref="S4.Thmtheorem1.p1.3.m3.1.1.2.3.cmml">c</mi><mo mathsize="70%" id="S4.Thmtheorem1.p1.3.m3.1.1.3" xref="S4.Thmtheorem1.p1.3.m3.1.1.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.3.m3.1b"><apply id="S4.Thmtheorem1.p1.3.m3.1.1.cmml" xref="S4.Thmtheorem1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S4.Thmtheorem1.p1.3.m3.1.1.1.cmml" xref="S4.Thmtheorem1.p1.3.m3.1.1">superscript</csymbol><apply id="S4.Thmtheorem1.p1.3.m3.1.1.2.cmml" xref="S4.Thmtheorem1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S4.Thmtheorem1.p1.3.m3.1.1.2.1.cmml" xref="S4.Thmtheorem1.p1.3.m3.1.1">subscript</csymbol><ci id="S4.Thmtheorem1.p1.3.m3.1.1.2.2.cmml" xref="S4.Thmtheorem1.p1.3.m3.1.1.2.2">𝐳</ci><ci id="S4.Thmtheorem1.p1.3.m3.1.1.2.3.cmml" xref="S4.Thmtheorem1.p1.3.m3.1.1.2.3">𝑐</ci></apply><ci id="S4.Thmtheorem1.p1.3.m3.1.1.3.cmml" xref="S4.Thmtheorem1.p1.3.m3.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.3.m3.1c">\mathbf{z}_{c}^{\prime}</annotation></semantics></math><span id="S4.Thmtheorem1.p1.4.4" class="ltx_text" style="font-size:70%;"> instead of combining with the original few-shot examples </span><math id="S4.Thmtheorem1.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{z}_{c}" display="inline"><semantics id="S4.Thmtheorem1.p1.4.m4.1a"><msub id="S4.Thmtheorem1.p1.4.m4.1.1" xref="S4.Thmtheorem1.p1.4.m4.1.1.cmml"><mi mathsize="70%" id="S4.Thmtheorem1.p1.4.m4.1.1.2" xref="S4.Thmtheorem1.p1.4.m4.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.Thmtheorem1.p1.4.m4.1.1.3" xref="S4.Thmtheorem1.p1.4.m4.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.4.m4.1b"><apply id="S4.Thmtheorem1.p1.4.m4.1.1.cmml" xref="S4.Thmtheorem1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.Thmtheorem1.p1.4.m4.1.1.1.cmml" xref="S4.Thmtheorem1.p1.4.m4.1.1">subscript</csymbol><ci id="S4.Thmtheorem1.p1.4.m4.1.1.2.cmml" xref="S4.Thmtheorem1.p1.4.m4.1.1.2">𝐳</ci><ci id="S4.Thmtheorem1.p1.4.m4.1.1.3.cmml" xref="S4.Thmtheorem1.p1.4.m4.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.4.m4.1c">\mathbf{z}_{c}</annotation></semantics></math><span id="S4.Thmtheorem1.p1.4.5" class="ltx_text" style="font-size:70%;">. This will save the token usage and prevent an increase in the number of in-context examples while maintaining a similar performance.</span></p>
</div>
</div>
<div id="S4.SS1.SSS4.p3" class="ltx_para">
<p id="S4.SS1.SSS4.p3.1" class="ltx_p"><span id="S4.SS1.SSS4.p3.1.1" class="ltx_text" style="font-size:70%;">The following example showcases how to combine RaR with CoT.</span></p>
</div>
<div id="S4.Thmexmp5" class="ltx_theorem ltx_theorem_exmp">
<h6 class="ltx_title ltx_runin ltx_title_theorem" style="font-size:70%;">
<span class="ltx_tag ltx_tag_theorem"><span id="S4.Thmexmp5.1.1.1" class="ltx_text ltx_font_bold">Example 4.5</span></span><span id="S4.Thmexmp5.2.2" class="ltx_text ltx_font_bold"> </span>(RaR + Few-shot CoT)<span id="S4.Thmexmp5.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="S4.Thmexmp5.p1" class="ltx_para">
<p id="S4.Thmexmp5.p1.1" class="ltx_p"><span id="S4.Thmexmp5.p1.1.1" class="ltx_text" style="font-size:70%;">Consider two-shot CoT instruction </span><math id="S4.Thmexmp5.p1.1.m1.4" class="ltx_Math" alttext="\mathbf{z}_{c}=[\mathbf{x}_{1},\mathbf{y}_{1}^{\prime},\mathbf{x}_{2},\mathbf{y}_{2}^{\prime}]" display="inline"><semantics id="S4.Thmexmp5.p1.1.m1.4a"><mrow id="S4.Thmexmp5.p1.1.m1.4.4" xref="S4.Thmexmp5.p1.1.m1.4.4.cmml"><msub id="S4.Thmexmp5.p1.1.m1.4.4.6" xref="S4.Thmexmp5.p1.1.m1.4.4.6.cmml"><mi mathsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.6.2" xref="S4.Thmexmp5.p1.1.m1.4.4.6.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.6.3" xref="S4.Thmexmp5.p1.1.m1.4.4.6.3.cmml">c</mi></msub><mo mathsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.5" xref="S4.Thmexmp5.p1.1.m1.4.4.5.cmml">=</mo><mrow id="S4.Thmexmp5.p1.1.m1.4.4.4.4" xref="S4.Thmexmp5.p1.1.m1.4.4.4.5.cmml"><mo maxsize="70%" minsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.5" xref="S4.Thmexmp5.p1.1.m1.4.4.4.5.cmml">[</mo><msub id="S4.Thmexmp5.p1.1.m1.1.1.1.1.1" xref="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.2" xref="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.3" xref="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.6" xref="S4.Thmexmp5.p1.1.m1.4.4.4.5.cmml">,</mo><msubsup id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.cmml"><mi mathsize="70%" id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.2.2" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.2.3" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.2.3.cmml">1</mn><mo mathsize="70%" id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.3" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.7" xref="S4.Thmexmp5.p1.1.m1.4.4.4.5.cmml">,</mo><msub id="S4.Thmexmp5.p1.1.m1.3.3.3.3.3" xref="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.cmml"><mi mathsize="70%" id="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.2" xref="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.3" xref="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.3.cmml">2</mn></msub><mo mathsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.8" xref="S4.Thmexmp5.p1.1.m1.4.4.4.5.cmml">,</mo><msubsup id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.cmml"><mi mathsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.2.2" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.2.3" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.2.3.cmml">2</mn><mo mathsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.3" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.3.cmml">′</mo></msubsup><mo maxsize="70%" minsize="70%" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.9" xref="S4.Thmexmp5.p1.1.m1.4.4.4.5.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmexmp5.p1.1.m1.4b"><apply id="S4.Thmexmp5.p1.1.m1.4.4.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4"><eq id="S4.Thmexmp5.p1.1.m1.4.4.5.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.5"></eq><apply id="S4.Thmexmp5.p1.1.m1.4.4.6.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.6"><csymbol cd="ambiguous" id="S4.Thmexmp5.p1.1.m1.4.4.6.1.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.6">subscript</csymbol><ci id="S4.Thmexmp5.p1.1.m1.4.4.6.2.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.6.2">𝐳</ci><ci id="S4.Thmexmp5.p1.1.m1.4.4.6.3.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.6.3">𝑐</ci></apply><list id="S4.Thmexmp5.p1.1.m1.4.4.4.5.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4"><apply id="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.cmml" xref="S4.Thmexmp5.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.1.cmml" xref="S4.Thmexmp5.p1.1.m1.1.1.1.1.1">subscript</csymbol><ci id="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.2.cmml" xref="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.2">𝐱</ci><cn type="integer" id="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.3.cmml" xref="S4.Thmexmp5.p1.1.m1.1.1.1.1.1.3">1</cn></apply><apply id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.cmml" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.1.cmml" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2">superscript</csymbol><apply id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.2.cmml" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.2.1.cmml" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2">subscript</csymbol><ci id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.2.2.cmml" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.2.2">𝐲</ci><cn type="integer" id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.2.3.cmml" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.2.3">1</cn></apply><ci id="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.3.cmml" xref="S4.Thmexmp5.p1.1.m1.2.2.2.2.2.3">′</ci></apply><apply id="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.cmml" xref="S4.Thmexmp5.p1.1.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.1.cmml" xref="S4.Thmexmp5.p1.1.m1.3.3.3.3.3">subscript</csymbol><ci id="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.2.cmml" xref="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.2">𝐱</ci><cn type="integer" id="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.3.cmml" xref="S4.Thmexmp5.p1.1.m1.3.3.3.3.3.3">2</cn></apply><apply id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4"><csymbol cd="ambiguous" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.1.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4">superscript</csymbol><apply id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.2.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4"><csymbol cd="ambiguous" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.2.1.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4">subscript</csymbol><ci id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.2.2.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.2.2">𝐲</ci><cn type="integer" id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.2.3.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.2.3">2</cn></apply><ci id="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.3.cmml" xref="S4.Thmexmp5.p1.1.m1.4.4.4.4.4.3">′</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp5.p1.1.m1.4c">\mathbf{z}_{c}=[\mathbf{x}_{1},\mathbf{y}_{1}^{\prime},\mathbf{x}_{2},\mathbf{y}_{2}^{\prime}]</annotation></semantics></math><span id="S4.Thmexmp5.p1.1.2" class="ltx_text" style="font-size:70%;">, where</span></p>
<ul id="S4.I9" class="ltx_itemize">
<li id="S4.I9.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I9.i1.p1" class="ltx_para">
<p id="S4.I9.i1.p1.1" class="ltx_p"><math id="S4.I9.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}_{1}=\texttt{``Q: Take the last letters of the words in &quot;Elon Musk&quot; and concatenate them.''}" display="inline"><semantics id="S4.I9.i1.p1.1.m1.1a"><mrow id="S4.I9.i1.p1.1.m1.1.1" xref="S4.I9.i1.p1.1.m1.1.1.cmml"><msub id="S4.I9.i1.p1.1.m1.1.1.2" xref="S4.I9.i1.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I9.i1.p1.1.m1.1.1.2.2" xref="S4.I9.i1.p1.1.m1.1.1.2.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.I9.i1.p1.1.m1.1.1.2.3" xref="S4.I9.i1.p1.1.m1.1.1.2.3.cmml">1</mn></msub><mo mathsize="70%" id="S4.I9.i1.p1.1.m1.1.1.1" xref="S4.I9.i1.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I9.i1.p1.1.m1.1.1.3" xref="S4.I9.i1.p1.1.m1.1.1.3a.cmml">‘‘Q: Take the last letters of the words in "Elon Musk" and concatenate them.’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I9.i1.p1.1.m1.1b"><apply id="S4.I9.i1.p1.1.m1.1.1.cmml" xref="S4.I9.i1.p1.1.m1.1.1"><eq id="S4.I9.i1.p1.1.m1.1.1.1.cmml" xref="S4.I9.i1.p1.1.m1.1.1.1"></eq><apply id="S4.I9.i1.p1.1.m1.1.1.2.cmml" xref="S4.I9.i1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I9.i1.p1.1.m1.1.1.2.1.cmml" xref="S4.I9.i1.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I9.i1.p1.1.m1.1.1.2.2.cmml" xref="S4.I9.i1.p1.1.m1.1.1.2.2">𝐱</ci><cn type="integer" id="S4.I9.i1.p1.1.m1.1.1.2.3.cmml" xref="S4.I9.i1.p1.1.m1.1.1.2.3">1</cn></apply><ci id="S4.I9.i1.p1.1.m1.1.1.3a.cmml" xref="S4.I9.i1.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I9.i1.p1.1.m1.1.1.3.cmml" xref="S4.I9.i1.p1.1.m1.1.1.3">‘‘Q: Take the last letters of the words in "Elon Musk" and concatenate them.’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I9.i1.p1.1.m1.1c">\mathbf{x}_{1}=\texttt{``Q: Take the last letters of the words in "Elon Musk" and concatenate them.''}</annotation></semantics></math><span id="S4.I9.i1.p1.1.1" class="ltx_text" style="font-size:70%;"></span></p>
</div>
</li>
<li id="S4.I9.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I9.i2.p1" class="ltx_para">
<p id="S4.I9.i2.p1.1" class="ltx_p"><math id="S4.I9.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}_{1}^{\prime}=\texttt{``A: The first letter of &quot;Elon&quot; is &quot;E&quot;. \ldots The answer is nk.''}" display="inline"><semantics id="S4.I9.i2.p1.1.m1.1a"><mrow id="S4.I9.i2.p1.1.m1.1.1" xref="S4.I9.i2.p1.1.m1.1.1.cmml"><msubsup id="S4.I9.i2.p1.1.m1.1.1.2" xref="S4.I9.i2.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I9.i2.p1.1.m1.1.1.2.2.2" xref="S4.I9.i2.p1.1.m1.1.1.2.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.I9.i2.p1.1.m1.1.1.2.2.3" xref="S4.I9.i2.p1.1.m1.1.1.2.2.3.cmml">1</mn><mo mathsize="70%" id="S4.I9.i2.p1.1.m1.1.1.2.3" xref="S4.I9.i2.p1.1.m1.1.1.2.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.I9.i2.p1.1.m1.1.1.1" xref="S4.I9.i2.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I9.i2.p1.1.m1.1.1.3" xref="S4.I9.i2.p1.1.m1.1.1.3a.cmml">‘‘A: The first letter of "Elon" is "E". …The answer is nk.’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I9.i2.p1.1.m1.1b"><apply id="S4.I9.i2.p1.1.m1.1.1.cmml" xref="S4.I9.i2.p1.1.m1.1.1"><eq id="S4.I9.i2.p1.1.m1.1.1.1.cmml" xref="S4.I9.i2.p1.1.m1.1.1.1"></eq><apply id="S4.I9.i2.p1.1.m1.1.1.2.cmml" xref="S4.I9.i2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I9.i2.p1.1.m1.1.1.2.1.cmml" xref="S4.I9.i2.p1.1.m1.1.1.2">superscript</csymbol><apply id="S4.I9.i2.p1.1.m1.1.1.2.2.cmml" xref="S4.I9.i2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I9.i2.p1.1.m1.1.1.2.2.1.cmml" xref="S4.I9.i2.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I9.i2.p1.1.m1.1.1.2.2.2.cmml" xref="S4.I9.i2.p1.1.m1.1.1.2.2.2">𝐲</ci><cn type="integer" id="S4.I9.i2.p1.1.m1.1.1.2.2.3.cmml" xref="S4.I9.i2.p1.1.m1.1.1.2.2.3">1</cn></apply><ci id="S4.I9.i2.p1.1.m1.1.1.2.3.cmml" xref="S4.I9.i2.p1.1.m1.1.1.2.3">′</ci></apply><ci id="S4.I9.i2.p1.1.m1.1.1.3a.cmml" xref="S4.I9.i2.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I9.i2.p1.1.m1.1.1.3.cmml" xref="S4.I9.i2.p1.1.m1.1.1.3">‘‘A: The first letter of "Elon" is "E". …The answer is nk.’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I9.i2.p1.1.m1.1c">\mathbf{y}_{1}^{\prime}=\texttt{``A: The first letter of "Elon" is "E". \ldots The answer is nk.''}</annotation></semantics></math><span id="S4.I9.i2.p1.1.1" class="ltx_text" style="font-size:70%;"></span></p>
</div>
</li>
<li id="S4.I9.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I9.i3.p1" class="ltx_para">
<p id="S4.I9.i3.p1.1" class="ltx_p"><math id="S4.I9.i3.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}_{2}=\texttt{``Q: Take the last letters of the words in &quot;Larry Page&quot; and concatenate them.''}" display="inline"><semantics id="S4.I9.i3.p1.1.m1.1a"><mrow id="S4.I9.i3.p1.1.m1.1.1" xref="S4.I9.i3.p1.1.m1.1.1.cmml"><msub id="S4.I9.i3.p1.1.m1.1.1.2" xref="S4.I9.i3.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I9.i3.p1.1.m1.1.1.2.2" xref="S4.I9.i3.p1.1.m1.1.1.2.2.cmml">𝐱</mi><mn mathsize="70%" id="S4.I9.i3.p1.1.m1.1.1.2.3" xref="S4.I9.i3.p1.1.m1.1.1.2.3.cmml">2</mn></msub><mo mathsize="70%" id="S4.I9.i3.p1.1.m1.1.1.1" xref="S4.I9.i3.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I9.i3.p1.1.m1.1.1.3" xref="S4.I9.i3.p1.1.m1.1.1.3a.cmml">‘‘Q: Take the last letters of the words in "Larry Page" and concatenate them.’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I9.i3.p1.1.m1.1b"><apply id="S4.I9.i3.p1.1.m1.1.1.cmml" xref="S4.I9.i3.p1.1.m1.1.1"><eq id="S4.I9.i3.p1.1.m1.1.1.1.cmml" xref="S4.I9.i3.p1.1.m1.1.1.1"></eq><apply id="S4.I9.i3.p1.1.m1.1.1.2.cmml" xref="S4.I9.i3.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I9.i3.p1.1.m1.1.1.2.1.cmml" xref="S4.I9.i3.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I9.i3.p1.1.m1.1.1.2.2.cmml" xref="S4.I9.i3.p1.1.m1.1.1.2.2">𝐱</ci><cn type="integer" id="S4.I9.i3.p1.1.m1.1.1.2.3.cmml" xref="S4.I9.i3.p1.1.m1.1.1.2.3">2</cn></apply><ci id="S4.I9.i3.p1.1.m1.1.1.3a.cmml" xref="S4.I9.i3.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I9.i3.p1.1.m1.1.1.3.cmml" xref="S4.I9.i3.p1.1.m1.1.1.3">‘‘Q: Take the last letters of the words in "Larry Page" and concatenate them.’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I9.i3.p1.1.m1.1c">\mathbf{x}_{2}=\texttt{``Q: Take the last letters of the words in "Larry Page" and concatenate them.''}</annotation></semantics></math><span id="S4.I9.i3.p1.1.1" class="ltx_text" style="font-size:70%;"></span></p>
</div>
</li>
<li id="S4.I9.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I9.i4.p1" class="ltx_para">
<p id="S4.I9.i4.p1.1" class="ltx_p"><math id="S4.I9.i4.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{y}_{2}^{\prime}=\texttt{``A: The first letter of &quot;Larry&quot; is &quot;L&quot;. \ldots The answer is ye.''}" display="inline"><semantics id="S4.I9.i4.p1.1.m1.1a"><mrow id="S4.I9.i4.p1.1.m1.1.1" xref="S4.I9.i4.p1.1.m1.1.1.cmml"><msubsup id="S4.I9.i4.p1.1.m1.1.1.2" xref="S4.I9.i4.p1.1.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.I9.i4.p1.1.m1.1.1.2.2.2" xref="S4.I9.i4.p1.1.m1.1.1.2.2.2.cmml">𝐲</mi><mn mathsize="70%" id="S4.I9.i4.p1.1.m1.1.1.2.2.3" xref="S4.I9.i4.p1.1.m1.1.1.2.2.3.cmml">2</mn><mo mathsize="70%" id="S4.I9.i4.p1.1.m1.1.1.2.3" xref="S4.I9.i4.p1.1.m1.1.1.2.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.I9.i4.p1.1.m1.1.1.1" xref="S4.I9.i4.p1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I9.i4.p1.1.m1.1.1.3" xref="S4.I9.i4.p1.1.m1.1.1.3a.cmml">‘‘A: The first letter of "Larry" is "L". …The answer is ye.’’</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.I9.i4.p1.1.m1.1b"><apply id="S4.I9.i4.p1.1.m1.1.1.cmml" xref="S4.I9.i4.p1.1.m1.1.1"><eq id="S4.I9.i4.p1.1.m1.1.1.1.cmml" xref="S4.I9.i4.p1.1.m1.1.1.1"></eq><apply id="S4.I9.i4.p1.1.m1.1.1.2.cmml" xref="S4.I9.i4.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I9.i4.p1.1.m1.1.1.2.1.cmml" xref="S4.I9.i4.p1.1.m1.1.1.2">superscript</csymbol><apply id="S4.I9.i4.p1.1.m1.1.1.2.2.cmml" xref="S4.I9.i4.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.I9.i4.p1.1.m1.1.1.2.2.1.cmml" xref="S4.I9.i4.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.I9.i4.p1.1.m1.1.1.2.2.2.cmml" xref="S4.I9.i4.p1.1.m1.1.1.2.2.2">𝐲</ci><cn type="integer" id="S4.I9.i4.p1.1.m1.1.1.2.2.3.cmml" xref="S4.I9.i4.p1.1.m1.1.1.2.2.3">2</cn></apply><ci id="S4.I9.i4.p1.1.m1.1.1.2.3.cmml" xref="S4.I9.i4.p1.1.m1.1.1.2.3">′</ci></apply><ci id="S4.I9.i4.p1.1.m1.1.1.3a.cmml" xref="S4.I9.i4.p1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.I9.i4.p1.1.m1.1.1.3.cmml" xref="S4.I9.i4.p1.1.m1.1.1.3">‘‘A: The first letter of "Larry" is "L". …The answer is ye.’’</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I9.i4.p1.1.m1.1c">\mathbf{y}_{2}^{\prime}=\texttt{``A: The first letter of "Larry" is "L". \ldots The answer is ye.''}</annotation></semantics></math><span id="S4.I9.i4.p1.1.1" class="ltx_text" style="font-size:70%;"></span></p>
</div>
</li>
</ul>
<p id="S4.Thmexmp5.p1.2" class="ltx_p"><span id="S4.Thmexmp5.p1.2.1" class="ltx_text" style="font-size:70%;">Applying </span><math id="S4.Thmexmp5.p1.2.m1.1" class="ltx_Math" alttext="\mathbf{z}_{r}" display="inline"><semantics id="S4.Thmexmp5.p1.2.m1.1a"><msub id="S4.Thmexmp5.p1.2.m1.1.1" xref="S4.Thmexmp5.p1.2.m1.1.1.cmml"><mi mathsize="70%" id="S4.Thmexmp5.p1.2.m1.1.1.2" xref="S4.Thmexmp5.p1.2.m1.1.1.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.Thmexmp5.p1.2.m1.1.1.3" xref="S4.Thmexmp5.p1.2.m1.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="S4.Thmexmp5.p1.2.m1.1b"><apply id="S4.Thmexmp5.p1.2.m1.1.1.cmml" xref="S4.Thmexmp5.p1.2.m1.1.1"><csymbol cd="ambiguous" id="S4.Thmexmp5.p1.2.m1.1.1.1.cmml" xref="S4.Thmexmp5.p1.2.m1.1.1">subscript</csymbol><ci id="S4.Thmexmp5.p1.2.m1.1.1.2.cmml" xref="S4.Thmexmp5.p1.2.m1.1.1.2">𝐳</ci><ci id="S4.Thmexmp5.p1.2.m1.1.1.3.cmml" xref="S4.Thmexmp5.p1.2.m1.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmexmp5.p1.2.m1.1c">\mathbf{z}_{r}</annotation></semantics></math><span id="S4.Thmexmp5.p1.2.2" class="ltx_text" style="font-size:70%;"> as in Figure </span><a href="#S4.F9" title="Figure 9 ‣ Results. ‣ 4.3 Empirical Improvement on Few-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">9</span></a><span id="S4.Thmexmp5.p1.2.3" class="ltx_text" style="font-size:70%;">, we get</span></p>
<table id="A2.EGx9" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.Ex5"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.Ex5.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{z}_{c}^{\prime}=" display="inline"><semantics id="S4.Ex5.m1.1a"><mrow id="S4.Ex5.m1.1.1" xref="S4.Ex5.m1.1.1.cmml"><msubsup id="S4.Ex5.m1.1.1.2" xref="S4.Ex5.m1.1.1.2.cmml"><mi mathsize="70%" id="S4.Ex5.m1.1.1.2.2.2" xref="S4.Ex5.m1.1.1.2.2.2.cmml">𝐳</mi><mi mathsize="70%" id="S4.Ex5.m1.1.1.2.2.3" xref="S4.Ex5.m1.1.1.2.2.3.cmml">c</mi><mo mathsize="70%" id="S4.Ex5.m1.1.1.2.3" xref="S4.Ex5.m1.1.1.2.3.cmml">′</mo></msubsup><mo mathsize="70%" id="S4.Ex5.m1.1.1.1" xref="S4.Ex5.m1.1.1.1.cmml">=</mo><mi id="S4.Ex5.m1.1.1.3" xref="S4.Ex5.m1.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="S4.Ex5.m1.1b"><apply id="S4.Ex5.m1.1.1.cmml" xref="S4.Ex5.m1.1.1"><eq id="S4.Ex5.m1.1.1.1.cmml" xref="S4.Ex5.m1.1.1.1"></eq><apply id="S4.Ex5.m1.1.1.2.cmml" xref="S4.Ex5.m1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex5.m1.1.1.2.1.cmml" xref="S4.Ex5.m1.1.1.2">superscript</csymbol><apply id="S4.Ex5.m1.1.1.2.2.cmml" xref="S4.Ex5.m1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex5.m1.1.1.2.2.1.cmml" xref="S4.Ex5.m1.1.1.2">subscript</csymbol><ci id="S4.Ex5.m1.1.1.2.2.2.cmml" xref="S4.Ex5.m1.1.1.2.2.2">𝐳</ci><ci id="S4.Ex5.m1.1.1.2.2.3.cmml" xref="S4.Ex5.m1.1.1.2.2.3">𝑐</ci></apply><ci id="S4.Ex5.m1.1.1.2.3.cmml" xref="S4.Ex5.m1.1.1.2.3">′</ci></apply><csymbol cd="latexml" id="S4.Ex5.m1.1.1.3.cmml" xref="S4.Ex5.m1.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Ex5.m1.1c">\displaystyle\mathbf{z}_{c}^{\prime}=</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S4.Ex5.3.2.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter" style="font-size:70%;">‘‘Q: Identify the last letters of each word in the name "Elon Musk", …together.</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S4.Ex6"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S4.Ex6.4.3.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter" style="font-size:70%;"> A: The last letter of "Elon" is "n". …, the result is nk.’’</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S4.Ex7"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S4.Ex7.4.3.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter" style="font-size:70%;"> Q: Identify the last letters of each word in the name "Larry Page", …together.</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S4.Ex8"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S4.Ex8.4.3.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter" style="font-size:70%;"> A: The last letter of "Larry" is "y". …The result is ye.</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</div>
<div id="S4.SS1.SSS4.p4" class="ltx_para">
<p id="S4.SS1.SSS4.p4.1" class="ltx_p"><span id="S4.SS1.SSS4.p4.1.1" class="ltx_text" style="font-size:70%;">Our experiment demonstrates that the integration of RaR with few-shot CoT significantly improves the performance of CoT. A comprehensive discussion of the results can be found in Section </span><a href="#S4.SS3" title="4.3 Empirical Improvement on Few-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">4.3</span></a><span id="S4.SS1.SSS4.p4.1.2" class="ltx_text" style="font-size:70%;">.
Lastly, we present illustrations of our mathematical formulations for CoT and RaR in Figure </span><a href="#S4.F7" title="Figure 7 ‣ 4.1.4 Combining RaR and CoT ‣ 4.1 Mathematical Formulation ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">7</span></a><span id="S4.SS1.SSS4.p4.1.3" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
<figure id="S4.F7" class="ltx_figure"><img src="/html/2311.04205/assets/x7.png" id="S4.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="438" height="147" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>Demonstration of our mathematical formulations of RaR and CoT methods with question <math id="S4.F7.16.m1.1" class="ltx_Math" alttext="\mathbf{x}" display="inline"><semantics id="S4.F7.16.m1.1b"><mi id="S4.F7.16.m1.1.1" xref="S4.F7.16.m1.1.1.cmml">𝐱</mi><annotation-xml encoding="MathML-Content" id="S4.F7.16.m1.1c"><ci id="S4.F7.16.m1.1.1.cmml" xref="S4.F7.16.m1.1.1">𝐱</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.16.m1.1d">\mathbf{x}</annotation></semantics></math>. In particular, <span id="S4.F7.62.1" class="ltx_text ltx_font_bold">A. One-step RaR</span> can be viewed as using a single prompt <math id="S4.F7.17.m2.1" class="ltx_Math" alttext="\mathbf{z}_{r}" display="inline"><semantics id="S4.F7.17.m2.1b"><msub id="S4.F7.17.m2.1.1" xref="S4.F7.17.m2.1.1.cmml"><mi id="S4.F7.17.m2.1.1.2" xref="S4.F7.17.m2.1.1.2.cmml">𝐳</mi><mi id="S4.F7.17.m2.1.1.3" xref="S4.F7.17.m2.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="S4.F7.17.m2.1c"><apply id="S4.F7.17.m2.1.1.cmml" xref="S4.F7.17.m2.1.1"><csymbol cd="ambiguous" id="S4.F7.17.m2.1.1.1.cmml" xref="S4.F7.17.m2.1.1">subscript</csymbol><ci id="S4.F7.17.m2.1.1.2.cmml" xref="S4.F7.17.m2.1.1.2">𝐳</ci><ci id="S4.F7.17.m2.1.1.3.cmml" xref="S4.F7.17.m2.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.17.m2.1d">\mathbf{z}_{r}</annotation></semantics></math> to generate the rephrased <math id="S4.F7.18.m3.1" class="ltx_Math" alttext="\mathbf{x}^{\prime}" display="inline"><semantics id="S4.F7.18.m3.1b"><msup id="S4.F7.18.m3.1.1" xref="S4.F7.18.m3.1.1.cmml"><mi id="S4.F7.18.m3.1.1.2" xref="S4.F7.18.m3.1.1.2.cmml">𝐱</mi><mo id="S4.F7.18.m3.1.1.3" xref="S4.F7.18.m3.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.F7.18.m3.1c"><apply id="S4.F7.18.m3.1.1.cmml" xref="S4.F7.18.m3.1.1"><csymbol cd="ambiguous" id="S4.F7.18.m3.1.1.1.cmml" xref="S4.F7.18.m3.1.1">superscript</csymbol><ci id="S4.F7.18.m3.1.1.2.cmml" xref="S4.F7.18.m3.1.1.2">𝐱</ci><ci id="S4.F7.18.m3.1.1.3.cmml" xref="S4.F7.18.m3.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.18.m3.1d">\mathbf{x}^{\prime}</annotation></semantics></math> and answer <math id="S4.F7.19.m4.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.F7.19.m4.1b"><mi id="S4.F7.19.m4.1.1" xref="S4.F7.19.m4.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.F7.19.m4.1c"><ci id="S4.F7.19.m4.1.1.cmml" xref="S4.F7.19.m4.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.19.m4.1d">\mathbf{y}</annotation></semantics></math>. <span id="S4.F7.63.2" class="ltx_text ltx_font_bold">B. Two-step RaR</span> can be viewed as first rephrasing the question <math id="S4.F7.20.m5.1" class="ltx_Math" alttext="\mathbf{x}" display="inline"><semantics id="S4.F7.20.m5.1b"><mi id="S4.F7.20.m5.1.1" xref="S4.F7.20.m5.1.1.cmml">𝐱</mi><annotation-xml encoding="MathML-Content" id="S4.F7.20.m5.1c"><ci id="S4.F7.20.m5.1.1.cmml" xref="S4.F7.20.m5.1.1">𝐱</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.20.m5.1d">\mathbf{x}</annotation></semantics></math> using prompt <math id="S4.F7.21.m6.1" class="ltx_Math" alttext="\mathbf{z}_{r}" display="inline"><semantics id="S4.F7.21.m6.1b"><msub id="S4.F7.21.m6.1.1" xref="S4.F7.21.m6.1.1.cmml"><mi id="S4.F7.21.m6.1.1.2" xref="S4.F7.21.m6.1.1.2.cmml">𝐳</mi><mi id="S4.F7.21.m6.1.1.3" xref="S4.F7.21.m6.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="S4.F7.21.m6.1c"><apply id="S4.F7.21.m6.1.1.cmml" xref="S4.F7.21.m6.1.1"><csymbol cd="ambiguous" id="S4.F7.21.m6.1.1.1.cmml" xref="S4.F7.21.m6.1.1">subscript</csymbol><ci id="S4.F7.21.m6.1.1.2.cmml" xref="S4.F7.21.m6.1.1.2">𝐳</ci><ci id="S4.F7.21.m6.1.1.3.cmml" xref="S4.F7.21.m6.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.21.m6.1d">\mathbf{z}_{r}</annotation></semantics></math>, then combining the original question <math id="S4.F7.22.m7.1" class="ltx_Math" alttext="\mathbf{x}" display="inline"><semantics id="S4.F7.22.m7.1b"><mi id="S4.F7.22.m7.1.1" xref="S4.F7.22.m7.1.1.cmml">𝐱</mi><annotation-xml encoding="MathML-Content" id="S4.F7.22.m7.1c"><ci id="S4.F7.22.m7.1.1.cmml" xref="S4.F7.22.m7.1.1">𝐱</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.22.m7.1d">\mathbf{x}</annotation></semantics></math> and the rephrased question <math id="S4.F7.23.m8.1" class="ltx_Math" alttext="\mathbf{x}^{\prime}" display="inline"><semantics id="S4.F7.23.m8.1b"><msup id="S4.F7.23.m8.1.1" xref="S4.F7.23.m8.1.1.cmml"><mi id="S4.F7.23.m8.1.1.2" xref="S4.F7.23.m8.1.1.2.cmml">𝐱</mi><mo id="S4.F7.23.m8.1.1.3" xref="S4.F7.23.m8.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.F7.23.m8.1c"><apply id="S4.F7.23.m8.1.1.cmml" xref="S4.F7.23.m8.1.1"><csymbol cd="ambiguous" id="S4.F7.23.m8.1.1.1.cmml" xref="S4.F7.23.m8.1.1">superscript</csymbol><ci id="S4.F7.23.m8.1.1.2.cmml" xref="S4.F7.23.m8.1.1.2">𝐱</ci><ci id="S4.F7.23.m8.1.1.3.cmml" xref="S4.F7.23.m8.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.23.m8.1d">\mathbf{x}^{\prime}</annotation></semantics></math> to answer.
<span id="S4.F7.64.3" class="ltx_text ltx_font_bold">C. Zero-shot CoT</span> is a prompt <math id="S4.F7.24.m9.1" class="ltx_Math" alttext="\mathbf{z}_{c}" display="inline"><semantics id="S4.F7.24.m9.1b"><msub id="S4.F7.24.m9.1.1" xref="S4.F7.24.m9.1.1.cmml"><mi id="S4.F7.24.m9.1.1.2" xref="S4.F7.24.m9.1.1.2.cmml">𝐳</mi><mi id="S4.F7.24.m9.1.1.3" xref="S4.F7.24.m9.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S4.F7.24.m9.1c"><apply id="S4.F7.24.m9.1.1.cmml" xref="S4.F7.24.m9.1.1"><csymbol cd="ambiguous" id="S4.F7.24.m9.1.1.1.cmml" xref="S4.F7.24.m9.1.1">subscript</csymbol><ci id="S4.F7.24.m9.1.1.2.cmml" xref="S4.F7.24.m9.1.1.2">𝐳</ci><ci id="S4.F7.24.m9.1.1.3.cmml" xref="S4.F7.24.m9.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.24.m9.1d">\mathbf{z}_{c}</annotation></semantics></math> to enable LLM to generate the CoT steps <math id="S4.F7.25.m10.3" class="ltx_Math" alttext="\mathbf{s}_{1},\cdots,\mathbf{s}_{m}" display="inline"><semantics id="S4.F7.25.m10.3b"><mrow id="S4.F7.25.m10.3.3.2" xref="S4.F7.25.m10.3.3.3.cmml"><msub id="S4.F7.25.m10.2.2.1.1" xref="S4.F7.25.m10.2.2.1.1.cmml"><mi id="S4.F7.25.m10.2.2.1.1.2" xref="S4.F7.25.m10.2.2.1.1.2.cmml">𝐬</mi><mn id="S4.F7.25.m10.2.2.1.1.3" xref="S4.F7.25.m10.2.2.1.1.3.cmml">1</mn></msub><mo id="S4.F7.25.m10.3.3.2.3" xref="S4.F7.25.m10.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.F7.25.m10.1.1" xref="S4.F7.25.m10.1.1.cmml">⋯</mi><mo id="S4.F7.25.m10.3.3.2.4" xref="S4.F7.25.m10.3.3.3.cmml">,</mo><msub id="S4.F7.25.m10.3.3.2.2" xref="S4.F7.25.m10.3.3.2.2.cmml"><mi id="S4.F7.25.m10.3.3.2.2.2" xref="S4.F7.25.m10.3.3.2.2.2.cmml">𝐬</mi><mi id="S4.F7.25.m10.3.3.2.2.3" xref="S4.F7.25.m10.3.3.2.2.3.cmml">m</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.F7.25.m10.3c"><list id="S4.F7.25.m10.3.3.3.cmml" xref="S4.F7.25.m10.3.3.2"><apply id="S4.F7.25.m10.2.2.1.1.cmml" xref="S4.F7.25.m10.2.2.1.1"><csymbol cd="ambiguous" id="S4.F7.25.m10.2.2.1.1.1.cmml" xref="S4.F7.25.m10.2.2.1.1">subscript</csymbol><ci id="S4.F7.25.m10.2.2.1.1.2.cmml" xref="S4.F7.25.m10.2.2.1.1.2">𝐬</ci><cn type="integer" id="S4.F7.25.m10.2.2.1.1.3.cmml" xref="S4.F7.25.m10.2.2.1.1.3">1</cn></apply><ci id="S4.F7.25.m10.1.1.cmml" xref="S4.F7.25.m10.1.1">⋯</ci><apply id="S4.F7.25.m10.3.3.2.2.cmml" xref="S4.F7.25.m10.3.3.2.2"><csymbol cd="ambiguous" id="S4.F7.25.m10.3.3.2.2.1.cmml" xref="S4.F7.25.m10.3.3.2.2">subscript</csymbol><ci id="S4.F7.25.m10.3.3.2.2.2.cmml" xref="S4.F7.25.m10.3.3.2.2.2">𝐬</ci><ci id="S4.F7.25.m10.3.3.2.2.3.cmml" xref="S4.F7.25.m10.3.3.2.2.3">𝑚</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.25.m10.3d">\mathbf{s}_{1},\cdots,\mathbf{s}_{m}</annotation></semantics></math> then lead to the answer <math id="S4.F7.26.m11.1" class="ltx_Math" alttext="\mathbf{y}" display="inline"><semantics id="S4.F7.26.m11.1b"><mi id="S4.F7.26.m11.1.1" xref="S4.F7.26.m11.1.1.cmml">𝐲</mi><annotation-xml encoding="MathML-Content" id="S4.F7.26.m11.1c"><ci id="S4.F7.26.m11.1.1.cmml" xref="S4.F7.26.m11.1.1">𝐲</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.26.m11.1d">\mathbf{y}</annotation></semantics></math>. <span id="S4.F7.65.4" class="ltx_text ltx_font_bold">D. RaR + Zero-shot CoT</span> is instructing the LLM to first rephrase the question <math id="S4.F7.27.m12.1" class="ltx_Math" alttext="\mathbf{x}^{\prime}" display="inline"><semantics id="S4.F7.27.m12.1b"><msup id="S4.F7.27.m12.1.1" xref="S4.F7.27.m12.1.1.cmml"><mi id="S4.F7.27.m12.1.1.2" xref="S4.F7.27.m12.1.1.2.cmml">𝐱</mi><mo id="S4.F7.27.m12.1.1.3" xref="S4.F7.27.m12.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.F7.27.m12.1c"><apply id="S4.F7.27.m12.1.1.cmml" xref="S4.F7.27.m12.1.1"><csymbol cd="ambiguous" id="S4.F7.27.m12.1.1.1.cmml" xref="S4.F7.27.m12.1.1">superscript</csymbol><ci id="S4.F7.27.m12.1.1.2.cmml" xref="S4.F7.27.m12.1.1.2">𝐱</ci><ci id="S4.F7.27.m12.1.1.3.cmml" xref="S4.F7.27.m12.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.27.m12.1d">\mathbf{x}^{\prime}</annotation></semantics></math> and then provide the CoT steps before answering. <span id="S4.F7.66.5" class="ltx_text ltx_font_bold">E. Few-shot CoT</span> provides several human-crafted CoT steps <math id="S4.F7.28.m13.2" class="ltx_Math" alttext="\mathbf{z}_{c,\text{few-shot}}" display="inline"><semantics id="S4.F7.28.m13.2b"><msub id="S4.F7.28.m13.2.3" xref="S4.F7.28.m13.2.3.cmml"><mi id="S4.F7.28.m13.2.3.2" xref="S4.F7.28.m13.2.3.2.cmml">𝐳</mi><mrow id="S4.F7.28.m13.2.2.2.4" xref="S4.F7.28.m13.2.2.2.3.cmml"><mi id="S4.F7.28.m13.1.1.1.1" xref="S4.F7.28.m13.1.1.1.1.cmml">c</mi><mo id="S4.F7.28.m13.2.2.2.4.1" xref="S4.F7.28.m13.2.2.2.3.cmml">,</mo><mtext id="S4.F7.28.m13.2.2.2.2" xref="S4.F7.28.m13.2.2.2.2a.cmml">few-shot</mtext></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.F7.28.m13.2c"><apply id="S4.F7.28.m13.2.3.cmml" xref="S4.F7.28.m13.2.3"><csymbol cd="ambiguous" id="S4.F7.28.m13.2.3.1.cmml" xref="S4.F7.28.m13.2.3">subscript</csymbol><ci id="S4.F7.28.m13.2.3.2.cmml" xref="S4.F7.28.m13.2.3.2">𝐳</ci><list id="S4.F7.28.m13.2.2.2.3.cmml" xref="S4.F7.28.m13.2.2.2.4"><ci id="S4.F7.28.m13.1.1.1.1.cmml" xref="S4.F7.28.m13.1.1.1.1">𝑐</ci><ci id="S4.F7.28.m13.2.2.2.2a.cmml" xref="S4.F7.28.m13.2.2.2.2"><mtext id="S4.F7.28.m13.2.2.2.2.cmml" xref="S4.F7.28.m13.2.2.2.2">few-shot</mtext></ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.28.m13.2d">\mathbf{z}_{c,\text{few-shot}}</annotation></semantics></math> for the model to generate the CoT steps for the question and then the answer. <span id="S4.F7.67.6" class="ltx_text ltx_font_bold">F. Two-step RaR + Few-shot CoT</span> instructs the model to first rephrase the human-crafted CoT steps to <math id="S4.F7.29.m14.2" class="ltx_Math" alttext="\mathbf{z}_{c,\text{few-shot}}^{\prime}" display="inline"><semantics id="S4.F7.29.m14.2b"><msubsup id="S4.F7.29.m14.2.3" xref="S4.F7.29.m14.2.3.cmml"><mi id="S4.F7.29.m14.2.3.2.2" xref="S4.F7.29.m14.2.3.2.2.cmml">𝐳</mi><mrow id="S4.F7.29.m14.2.2.2.4" xref="S4.F7.29.m14.2.2.2.3.cmml"><mi id="S4.F7.29.m14.1.1.1.1" xref="S4.F7.29.m14.1.1.1.1.cmml">c</mi><mo id="S4.F7.29.m14.2.2.2.4.1" xref="S4.F7.29.m14.2.2.2.3.cmml">,</mo><mtext id="S4.F7.29.m14.2.2.2.2" xref="S4.F7.29.m14.2.2.2.2a.cmml">few-shot</mtext></mrow><mo id="S4.F7.29.m14.2.3.3" xref="S4.F7.29.m14.2.3.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="S4.F7.29.m14.2c"><apply id="S4.F7.29.m14.2.3.cmml" xref="S4.F7.29.m14.2.3"><csymbol cd="ambiguous" id="S4.F7.29.m14.2.3.1.cmml" xref="S4.F7.29.m14.2.3">superscript</csymbol><apply id="S4.F7.29.m14.2.3.2.cmml" xref="S4.F7.29.m14.2.3"><csymbol cd="ambiguous" id="S4.F7.29.m14.2.3.2.1.cmml" xref="S4.F7.29.m14.2.3">subscript</csymbol><ci id="S4.F7.29.m14.2.3.2.2.cmml" xref="S4.F7.29.m14.2.3.2.2">𝐳</ci><list id="S4.F7.29.m14.2.2.2.3.cmml" xref="S4.F7.29.m14.2.2.2.4"><ci id="S4.F7.29.m14.1.1.1.1.cmml" xref="S4.F7.29.m14.1.1.1.1">𝑐</ci><ci id="S4.F7.29.m14.2.2.2.2a.cmml" xref="S4.F7.29.m14.2.2.2.2"><mtext id="S4.F7.29.m14.2.2.2.2.cmml" xref="S4.F7.29.m14.2.2.2.2">few-shot</mtext></ci></list></apply><ci id="S4.F7.29.m14.2.3.3.cmml" xref="S4.F7.29.m14.2.3.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F7.29.m14.2d">\mathbf{z}_{c,\text{few-shot}}^{\prime}</annotation></semantics></math> before using them as examples to generate the CoT steps and the answer. We denote the first prompt used in Two-step RaR (i.e., Prompt (<a href="#S2.E2" title="Equation 2.2 ‣ 2.2 Two-step RaR: Rephrase the Question and Respond to the Rephrased Question ‣ 2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a>)) as Two-step<sup id="S4.F7.68.7" class="ltx_sup">1</sup> in this figure.</figcaption>
</figure>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Empirical Comparison with Zero-Shot CoT</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p"><span id="S4.SS2.p1.1.1" class="ltx_text" style="font-size:70%;">It is widely known that zero-shot CoT, by appending the instruction “Let’s think step by step.” to queries, can effectively improve the performance of LLMs on reasoning tasks.
However, we highlight some examples where zero-shot CoT fails to deliver improvements, sometimes even leading to diminished performance. In contrast, RaR consistently demonstrates effectiveness.
We also emphasize the importance of question quality with an example, demonstrating that it should be prioritized before enhancing the model’s reasoning capabilities.
Lastly, we note that our method is complementary to zero-shot CoT and can be combined together by simply adding “let’s think step by step” to (</span><a href="#S2.E3" title="Equation 2.3 ‣ 2.2 Two-step RaR: Rephrase the Question and Respond to the Rephrased Question ‣ 2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">2.3</span></a><span id="S4.SS2.p1.1.2" class="ltx_text" style="font-size:70%;">) or (</span><a href="#S2.E1" title="Equation 2.1 ‣ 2.1 One-step RaR: Rephrase and Respond in a Single Prompt ‣ 2 Rephrase and Respond ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">2.1</span></a><span id="S4.SS2.p1.1.3" class="ltx_text" style="font-size:70%;">).</span></p>
</div>
<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph" style="font-size:70%;">Tasks.</h5>

<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px1.p1.1" class="ltx_p"><span id="S4.SS2.SSS0.Px1.p1.1.1" class="ltx_text" style="font-size:70%;">We examine the </span><span id="S4.SS2.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">Chinese Idiom</span><span id="S4.SS2.SSS0.Px1.p1.1.3" class="ltx_text" style="font-size:70%;"> task as introduced in </span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Allen-Zhu and Li</span> <span id="S4.SS2.SSS0.Px1.p1.1.4.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S4.SS2.SSS0.Px1.p1.1.5.2.2.1" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S4.SS2.SSS0.Px1.p1.1.6" class="ltx_text" style="font-size:70%;">, specifically the most difficult task of inferring the first letter. This task involves taking widely recognized four-character Chinese idioms and masking one character at each respective position.
The task is to let LLM correctly infer the masked character.
It has been discovered that GPT models suffer from inferring the masked character, particularly when it is located in the first position.
Furthermore, we also use the </span><span id="S4.SS2.SSS0.Px1.p1.1.7" class="ltx_text ltx_font_bold" style="font-size:70%;">StereoSet</span><span id="S4.SS2.SSS0.Px1.p1.1.8" class="ltx_text" style="font-size:70%;"> task </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S4.SS2.SSS0.Px1.p1.1.9.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Nadeem et al.</span><span id="S4.SS2.SSS0.Px1.p1.1.10.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2021</span></a><span id="S4.SS2.SSS0.Px1.p1.1.11.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S4.SS2.SSS0.Px1.p1.1.12" class="ltx_text" style="font-size:70%;">, which assesses the stereotypical biases present in LLMs with respect to gender, race, profession, and religion.
From the inter-sentence data, we sample </span><math id="S4.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="220" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.1.m1.1a"><mn mathsize="70%" id="S4.SS2.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.cmml">220</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.1.m1.1b"><cn type="integer" id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1">220</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.1.m1.1c">220</annotation></semantics></math><span id="S4.SS2.SSS0.Px1.p1.1.13" class="ltx_text" style="font-size:70%;"> examples, each comprising a context sentence and three choices: one stereotypical, one anti-stereotypical, and one unrelated.
We adopt the prompt format used by </span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Shaikh et al.</span> <span id="S4.SS2.SSS0.Px1.p1.1.14.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib32" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S4.SS2.SSS0.Px1.p1.1.15.2.2.1" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S4.SS2.SSS0.Px1.p1.1.16" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
</section>
<section id="S4.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph" style="font-size:70%;">Evaluation.</h5>

<div id="S4.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px2.p1.1" class="ltx_p"><span id="S4.SS2.SSS0.Px2.p1.1.1" class="ltx_text" style="font-size:70%;">For the Chinese Idiom task, we evaluate the zero-shot accuracy of GPT-4’s responses, with automated accuracy estimation and further manual checking. For StereoSet, as suggested by </span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Nadeem et al.</span> <span id="S4.SS2.SSS0.Px2.p1.1.2.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2021</span></a><span id="S4.SS2.SSS0.Px2.p1.1.3.2.2.1" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S4.SS2.SSS0.Px2.p1.1.4" class="ltx_text" style="font-size:70%;">, two crucial evaluation metrics should be considered: the </span><span id="S4.SS2.SSS0.Px2.p1.1.5" class="ltx_text ltx_font_bold" style="font-size:70%;">Language Modeling Score</span><span id="S4.SS2.SSS0.Px2.p1.1.6" class="ltx_text" style="font-size:70%;">, which assesses whether the LLM selects related options over unrelated ones, and the </span><span id="S4.SS2.SSS0.Px2.p1.1.7" class="ltx_text ltx_font_bold" style="font-size:70%;">Stereotype Score</span><span id="S4.SS2.SSS0.Px2.p1.1.8" class="ltx_text" style="font-size:70%;">, which quantifies the percentage of data that a model favors stereotypical choices over anti-stereotypical ones.
As identified by the authors, an ideal model would display no bias toward either stereotypical or anti-stereotypical associations, yielding an optimal score of </span><math id="S4.SS2.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="0.5" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.1.m1.1a"><mn mathsize="70%" id="S4.SS2.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.cmml">0.5</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.1.m1.1b"><cn type="float" id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1">0.5</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.1.m1.1c">0.5</annotation></semantics></math><span id="S4.SS2.SSS0.Px2.p1.1.9" class="ltx_text" style="font-size:70%;"> for the Stereotype Score.
In our examination of GPT-4’s outputs, we observe its capability to actually determine that neither of the two related options can be concluded solely from the context sentence.
Consequently, we categorize such outputs as fair responses and introduce a </span><span id="S4.SS2.SSS0.Px2.p1.1.10" class="ltx_text ltx_font_bold" style="font-size:70%;">Fair Score</span><span id="S4.SS2.SSS0.Px2.p1.1.11" class="ltx_text" style="font-size:70%;">, determined by the proportion of these responses, complementing the </span><span id="S4.SS2.SSS0.Px2.p1.1.12" class="ltx_text ltx_font_bold" style="font-size:70%;">Language Modeling Score</span><span id="S4.SS2.SSS0.Px2.p1.1.13" class="ltx_text" style="font-size:70%;">. We provide an example of such a response below.</span></p>
<table id="A2.EGx10" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.Ex9"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S4.Ex9.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter" style="font-size:70%;">The question does not provide enough information to determine the accuracy of options</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S4.Ex10"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S4.Ex10.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter" style="font-size:70%;">A and C. Option B is irrelevant to the given context. Therefore, none of the provided</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S4.Ex11"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><span id="S4.Ex11.3.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter" style="font-size:70%;">options can be selected as the most sensible answer based on the information available.</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</section>
<section id="S4.SS2.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph" style="font-size:70%;">Results.</h5>

<div id="S4.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px3.p1.2" class="ltx_p"><span id="S4.SS2.SSS0.Px3.p1.2.1" class="ltx_text" style="font-size:70%;">As illustrated in Table </span><a href="#S4.T5" title="Table 5 ‣ Results. ‣ 4.2 Empirical Comparison with Zero-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">5</span></a><span id="S4.SS2.SSS0.Px3.p1.2.2" class="ltx_text" style="font-size:70%;">, even though RaR enhances LLM’s performance, accurately inferring the first character of the Chinese Idiom task remains a challenge.
One might then ask: does zero-shot CoT provide consistent improvement to LLM in such tasks as it does on other reasoning tasks?
Our discovery is, in fact, zero-shot CoT may result in worse performances (</span><math id="S4.SS2.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="31.43\%" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.1.m1.1a"><mrow id="S4.SS2.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.cmml"><mn mathcolor="#FF0000" mathsize="70%" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.cmml">31.43</mn><mo mathcolor="#FF0000" mathsize="70%" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.1.m1.1b"><apply id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1"><csymbol cd="latexml" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1">percent</csymbol><cn type="float" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2">31.43</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.1.m1.1c">31.43\%</annotation></semantics></math><span id="S4.SS2.SSS0.Px3.p1.2.3" class="ltx_text" style="font-size:70%;">) for such hard tasks, as the LLM tends to hallucinate during the intermediate steps—a phenomenon similar to hallucination snowballing </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S4.SS2.SSS0.Px3.p1.2.4.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Zhang et al.</span><span id="S4.SS2.SSS0.Px3.p1.2.5.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib50" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023a</span></a><span id="S4.SS2.SSS0.Px3.p1.2.6.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S4.SS2.SSS0.Px3.p1.2.7" class="ltx_text" style="font-size:70%;">.
Furthermore, as </span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Shaikh et al.</span> <span id="S4.SS2.SSS0.Px3.p1.2.8.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib32" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S4.SS2.SSS0.Px3.p1.2.9.2.2.1" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S4.SS2.SSS0.Px3.p1.2.10" class="ltx_text" style="font-size:70%;"> discovered on other language models, zero-shot CoT may result in undesired reasoning towards bias and toxicity.
Also in Table </span><a href="#S4.T5" title="Table 5 ‣ Results. ‣ 4.2 Empirical Comparison with Zero-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">5</span></a><span id="S4.SS2.SSS0.Px3.p1.2.11" class="ltx_text" style="font-size:70%;">, we demonstrate the performance of GPT-4 on StereoSet.
We can observe that, while zero-shot CoT fails to improve the Language Modeling Score, rephrased questions improve it significantly to </span><math id="S4.SS2.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="97.73\%" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.2.m2.1a"><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml"><mn mathsize="70%" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.cmml">97.73</mn><mo mathsize="70%" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.2.m2.1b"><apply id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1"><csymbol cd="latexml" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1">percent</csymbol><cn type="float" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2">97.73</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.2.m2.1c">97.73\%</annotation></semantics></math><span id="S4.SS2.SSS0.Px3.p1.2.12" class="ltx_text" style="font-size:70%;">. This implies that, with RaR, the LLM rarely opts for unrelated choices.
Moreover, while zero-shot CoT improves the percentage of fair responses (choosing neither of them), RaR achieves the best performance.</span></p>
</div>
<figure id="S4.T5" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 5: </span>Performance of GPT-4 using original questions, zero-shot CoT prompt and RaR. We consider two tasks, Chinese Idiom and StereoSet. Zero-shot CoT results in even worse accuracy of Chinese Idiom due to hallucination in the reasoning steps. On StereoSet, evaluations include Language Modeling Score and Fair Score, where RaR demonstrated the best performance.</figcaption>
<table id="S4.T5.3" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T5.3.4" class="ltx_tr">
<td id="S4.T5.3.4.1" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T5.3.4.1.1" class="ltx_text" style="font-size:70%;">Dataset</span></td>
<td id="S4.T5.3.4.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T5.3.4.2.1" class="ltx_text" style="font-size:70%;">Evaluation</span></td>
<td id="S4.T5.3.4.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T5.3.4.3.1" class="ltx_text" style="font-size:70%;">Original</span></td>
<td id="S4.T5.3.4.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T5.3.4.4.1" class="ltx_text" style="font-size:70%;">Zero-shot CoT</span></td>
<td id="S4.T5.3.4.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T5.3.4.5.1" class="ltx_text" style="font-size:70%;">RaR</span></td>
</tr>
<tr id="S4.T5.1.1" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S4.T5.1.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.1.2.1" class="ltx_text" style="font-size:70%;background-color:#E0FFFF;">Chinese Idiom</span></td>
<td id="S4.T5.1.1.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.1.1.1" class="ltx_text" style="font-size:70%;background-color:#E0FFFF;">Accuracy (<math id="S4.T5.1.1.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T5.1.1.1.1.m1.1a"><mo mathbackground="#E0FFFF" id="S4.T5.1.1.1.1.m1.1.1" xref="S4.T5.1.1.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T5.1.1.1.1.m1.1b"><csymbol cd="latexml" id="S4.T5.1.1.1.1.m1.1.1.cmml" xref="S4.T5.1.1.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.1.1.1.1.m1.1c">\%</annotation></semantics></math>)</span></td>
<td id="S4.T5.1.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.1.3.1" class="ltx_text" style="font-size:70%;background-color:#E0FFFF;">32.38</span></td>
<td id="S4.T5.1.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.1.4.1" class="ltx_text" style="font-size:70%;color:#FF0000;background-color:#E0FFFF;">31.43</span></td>
<td id="S4.T5.1.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.1.5.1" class="ltx_text ltx_font_bold" style="font-size:70%;background-color:#E0FFFF;">35.24</span></td>
</tr>
<tr id="S4.T5.2.2" class="ltx_tr">
<td id="S4.T5.2.2.2" class="ltx_td ltx_align_center ltx_border_bb" rowspan="2"><span id="S4.T5.2.2.2.1" class="ltx_text" style="font-size:70%;">StereoSet</span></td>
<td id="S4.T5.2.2.1" class="ltx_td ltx_align_center">
<span id="S4.T5.2.2.1.1" class="ltx_text" style="font-size:70%;">Language Modeling Score (</span><math id="S4.T5.2.2.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T5.2.2.1.m1.1a"><mo mathsize="70%" id="S4.T5.2.2.1.m1.1.1" xref="S4.T5.2.2.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T5.2.2.1.m1.1b"><csymbol cd="latexml" id="S4.T5.2.2.1.m1.1.1.cmml" xref="S4.T5.2.2.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.2.2.1.m1.1c">\%</annotation></semantics></math><span id="S4.T5.2.2.1.2" class="ltx_text" style="font-size:70%;">)</span>
</td>
<td id="S4.T5.2.2.3" class="ltx_td ltx_align_center"><span id="S4.T5.2.2.3.1" class="ltx_text" style="font-size:70%;">84.09</span></td>
<td id="S4.T5.2.2.4" class="ltx_td ltx_align_center"><span id="S4.T5.2.2.4.1" class="ltx_text" style="font-size:70%;">84.09</span></td>
<td id="S4.T5.2.2.5" class="ltx_td ltx_align_center"><span id="S4.T5.2.2.5.1" class="ltx_text ltx_font_bold" style="font-size:70%;">97.73</span></td>
</tr>
<tr id="S4.T5.3.3" class="ltx_tr">
<td id="S4.T5.3.3.1" class="ltx_td ltx_align_center ltx_border_bb">
<span id="S4.T5.3.3.1.1" class="ltx_text" style="font-size:70%;">Fair Score (</span><math id="S4.T5.3.3.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T5.3.3.1.m1.1a"><mo mathsize="70%" id="S4.T5.3.3.1.m1.1.1" xref="S4.T5.3.3.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T5.3.3.1.m1.1b"><csymbol cd="latexml" id="S4.T5.3.3.1.m1.1.1.cmml" xref="S4.T5.3.3.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.3.3.1.m1.1c">\%</annotation></semantics></math><span id="S4.T5.3.3.1.2" class="ltx_text" style="font-size:70%;">)</span>
</td>
<td id="S4.T5.3.3.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T5.3.3.2.1" class="ltx_text" style="font-size:70%;">6.82</span></td>
<td id="S4.T5.3.3.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T5.3.3.3.1" class="ltx_text" style="font-size:70%;">35.00</span></td>
<td id="S4.T5.3.3.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T5.3.3.4.1" class="ltx_text ltx_font_bold" style="font-size:70%;">42.27</span></td>
</tr>
</table>
</figure>
</section>
<section id="S4.SS2.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph" style="font-size:70%;">Question quality comes before reasoning.</h5>

<div id="S4.SS2.SSS0.Px4.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px4.p1.3" class="ltx_p"><span id="S4.SS2.SSS0.Px4.p1.3.1" class="ltx_text" style="font-size:70%;">With the following example, we emphasize that the attention to question quality is more important before considering to improve model reasoning. We examine the original </span><span id="S4.SS2.SSS0.Px4.p1.3.2" class="ltx_text ltx_font_bold" style="font-size:70%;">Coin Flip</span><span id="S4.SS2.SSS0.Px4.p1.3.3" class="ltx_text" style="font-size:70%;"> questions. Specifically, an example question is</span></p>
<p id="S4.SS2.SSS0.Px4.p1.4" class="ltx_p ltx_align_center"><span id="S4.SS2.SSS0.Px4.p1.4.1" class="ltx_text ltx_font_typewriter" style="font-size:70%;">A coin is heads up. aluino flips the coin. arthor flips the coin. Is the coin still heads up?</span><span id="S4.SS2.SSS0.Px4.p1.4.2" class="ltx_text" style="font-size:70%;"></span></p>
<p id="S4.SS2.SSS0.Px4.p1.2" class="ltx_p"><span id="S4.SS2.SSS0.Px4.p1.2.1" class="ltx_text" style="font-size:70%;">This question, as originally crafted by Humans, appears clear to human interpreters that “flipping” the coin here means reversing the coin.
However, LLMs like GPT-4 might perceive the flipping as a random toss.
As shown in Figure </span><a href="#S4.F8" title="Figure 8 ‣ Question quality comes before reasoning. ‣ 4.2 Empirical Comparison with Zero-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">8</span></a><span id="S4.SS2.SSS0.Px4.p1.2.2" class="ltx_text" style="font-size:70%;">, such misconception persists even when prompting the LLM to think step by step, which therefore results in an incorrect answer. Once we add a clarifying sentence stating that flip means reverse, GPT-4 can finally start answering the question as we desire. Since the clarification is also created by humans, LLM still exhibit an unsatisfactory performance of </span><math id="S4.SS2.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="74.55\%" display="inline"><semantics id="S4.SS2.SSS0.Px4.p1.1.m1.1a"><mrow id="S4.SS2.SSS0.Px4.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px4.p1.1.m1.1.1.cmml"><mn mathsize="70%" id="S4.SS2.SSS0.Px4.p1.1.m1.1.1.2" xref="S4.SS2.SSS0.Px4.p1.1.m1.1.1.2.cmml">74.55</mn><mo mathsize="70%" id="S4.SS2.SSS0.Px4.p1.1.m1.1.1.1" xref="S4.SS2.SSS0.Px4.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px4.p1.1.m1.1b"><apply id="S4.SS2.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px4.p1.1.m1.1.1"><csymbol cd="latexml" id="S4.SS2.SSS0.Px4.p1.1.m1.1.1.1.cmml" xref="S4.SS2.SSS0.Px4.p1.1.m1.1.1.1">percent</csymbol><cn type="float" id="S4.SS2.SSS0.Px4.p1.1.m1.1.1.2.cmml" xref="S4.SS2.SSS0.Px4.p1.1.m1.1.1.2">74.55</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px4.p1.1.m1.1c">74.55\%</annotation></semantics></math><span id="S4.SS2.SSS0.Px4.p1.2.3" class="ltx_text" style="font-size:70%;">. With self-rephrased questions, the accuracy can finally be improved to </span><math id="S4.SS2.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="97.73\%" display="inline"><semantics id="S4.SS2.SSS0.Px4.p1.2.m2.1a"><mrow id="S4.SS2.SSS0.Px4.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px4.p1.2.m2.1.1.cmml"><mn mathsize="70%" id="S4.SS2.SSS0.Px4.p1.2.m2.1.1.2" xref="S4.SS2.SSS0.Px4.p1.2.m2.1.1.2.cmml">97.73</mn><mo mathsize="70%" id="S4.SS2.SSS0.Px4.p1.2.m2.1.1.1" xref="S4.SS2.SSS0.Px4.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px4.p1.2.m2.1b"><apply id="S4.SS2.SSS0.Px4.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px4.p1.2.m2.1.1"><csymbol cd="latexml" id="S4.SS2.SSS0.Px4.p1.2.m2.1.1.1.cmml" xref="S4.SS2.SSS0.Px4.p1.2.m2.1.1.1">percent</csymbol><cn type="float" id="S4.SS2.SSS0.Px4.p1.2.m2.1.1.2.cmml" xref="S4.SS2.SSS0.Px4.p1.2.m2.1.1.2">97.73</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px4.p1.2.m2.1c">97.73\%</annotation></semantics></math><span id="S4.SS2.SSS0.Px4.p1.2.4" class="ltx_text" style="font-size:70%;">.
In light of such instances, we advocate for careful examination of human-crafted questions when evaluating LLMs, ensuring the removal of ambiguities for a fair assessment.</span></p>
</div>
<figure id="S4.F8" class="ltx_figure"><img src="/html/2311.04205/assets/x8.png" id="S4.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="151" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>Demonstration of the importance of question quality as compared to reasoning in the coin flip question. We highlight the incorrect responses in <span id="S4.F8.5.1" class="ltx_text" style="color:#FF0000;">red</span>. As shown in the left, when the concept of “flip” is not clarified for GPT-4, prompting it to further reasoning will not resolve the problem as the ambiguity persists through the reasoning steps. Only after clarifying that “flip” means the action of reverse will enable the LLM to correctly respond to the question. Lastly, the human-crafted clarification still results in false response. Meanwhile, the application of RaR can effectively improve any human-crafted phrases.</figcaption>
</figure>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Empirical Improvement on Few-Shot CoT</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p"><span id="S4.SS3.p1.1.1" class="ltx_text" style="font-size:70%;">Few-shot CoT </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S4.SS3.p1.1.2.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Wei et al.</span><span id="S4.SS3.p1.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S4.SS3.p1.1.4.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S4.SS3.p1.1.5" class="ltx_text" style="font-size:70%;"> has been the most effective CoT technique. It employs a small set of human-crafted QA examples to facilitate LLMs in addressing similar questions with a congruent structure. LLMs, particularly advanced models such as GPT-4, are adept at extrapolating from the provided examples to improve their performance on new questions.
Providing question-answer pairs effectively communicates the human-desired logical structure to the LLM.
Instead of aligning the question to what the LLM best receives, few-shot CoT guides the LLM to reason using the supplied human logic.
Nonetheless, a concern emerges: How do LLMs respond when the human-crafted examples are flawed or contain errors?
As corroborated by a recent parallel study </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S4.SS3.p1.1.6.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Pawelczyk et al.</span><span id="S4.SS3.p1.1.7.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib24" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S4.SS3.p1.1.8.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S4.SS3.p1.1.9" class="ltx_text" style="font-size:70%;">, we similarly observe that LLMs can be adversely influenced by bad few-shot examples.</span></p>
</div>
<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph" style="font-size:70%;">Tasks.</h5>

<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px1.p1.1" class="ltx_p"><span id="S4.SS3.SSS0.Px1.p1.1.1" class="ltx_text" style="font-size:70%;">We revisit the </span><span id="S4.SS3.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">Last Letter Concatenation</span><span id="S4.SS3.SSS0.Px1.p1.1.3" class="ltx_text" style="font-size:70%;"> task and refer to the few-shot examples provided in </span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Wei et al.</span> <span id="S4.SS3.SSS0.Px1.p1.1.4.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S4.SS3.SSS0.Px1.p1.1.5.2.2.1" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S4.SS3.SSS0.Px1.p1.1.6" class="ltx_text" style="font-size:70%;">. As shown in Figure </span><a href="#S4.F9" title="Figure 9 ‣ Results. ‣ 4.3 Empirical Improvement on Few-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">9</span></a><span id="S4.SS3.SSS0.Px1.p1.1.7" class="ltx_text" style="font-size:70%;">, the examples follow a specific logic: (1) obtain the last letter of the first word; (2) obtain the last letter of the second word; (3) concatenate these letters; resulting in (4) the answer. Such few-shot examples have been demonstrated to most effectively enhance the performance of a language model, achieving an accuracy of </span><math id="S4.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="100\%" display="inline"><semantics id="S4.SS3.SSS0.Px1.p1.1.m1.1a"><mrow id="S4.SS3.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.cmml"><mn mathsize="70%" id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml">100</mn><mo mathsize="70%" id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.1" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p1.1.m1.1b"><apply id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1"><csymbol cd="latexml" id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2">100</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p1.1.m1.1c">100\%</annotation></semantics></math><span id="S4.SS3.SSS0.Px1.p1.1.8" class="ltx_text" style="font-size:70%;"> when concatenating the last letters of two words. Conversely, we explore an example that employs the following logic: (1) obtain the </span><span id="S4.SS3.SSS0.Px1.p1.1.9" class="ltx_text ltx_font_bold" style="font-size:70%;">first</span><span id="S4.SS3.SSS0.Px1.p1.1.10" class="ltx_text" style="font-size:70%;"> letter of the first word; (2) obtain the </span><span id="S4.SS3.SSS0.Px1.p1.1.11" class="ltx_text ltx_font_bold" style="font-size:70%;">first</span><span id="S4.SS3.SSS0.Px1.p1.1.12" class="ltx_text" style="font-size:70%;"> letter of the second word; (3) concatenate these letters; providing (4) the answer for </span><span id="S4.SS3.SSS0.Px1.p1.1.13" class="ltx_text ltx_font_bold" style="font-size:70%;">last</span><span id="S4.SS3.SSS0.Px1.p1.1.14" class="ltx_text" style="font-size:70%;"> letter concatenation.
Our aim is to investigate how this alternative few-shot prompt, despite bearing a logic similar to the original prompt and the correct answer, influences the performance of the GPT-4.</span></p>
</div>
</section>
<section id="S4.SS3.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph" style="font-size:70%;">Results.</h5>

<div id="S4.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px2.p1.1" class="ltx_p"><span id="S4.SS3.SSS0.Px2.p1.1.1" class="ltx_text" style="font-size:70%;">As illustrated in Figure </span><a href="#S4.F9" title="Figure 9 ‣ Results. ‣ 4.3 Empirical Improvement on Few-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">9</span></a><span id="S4.SS3.SSS0.Px2.p1.1.2" class="ltx_text" style="font-size:70%;">, GPT-4 tends to stick to the logic of our modified prompt, resulting in an incorrect answer.
It accurately concatenates all first letters, but concludes with a seemingly arbitrary final answer.
In Table </span><a href="#S4.T6" title="Table 6 ‣ Results. ‣ 4.3 Empirical Improvement on Few-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">6</span></a><span id="S4.SS3.SSS0.Px2.p1.1.3" class="ltx_text" style="font-size:70%;">, we demonstrate the results of one-shot and four-shot CoT using such examples. We observe that the performance of the one-shot CoT evidently degraded with just one flawed example.
As the number of these flawed examples increases, the performance of GPT-4 in a 4-shot setting for last letter concatenation of four words drops to only </span><math id="S4.SS3.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="52.27\%" display="inline"><semantics id="S4.SS3.SSS0.Px2.p1.1.m1.1a"><mrow id="S4.SS3.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.cmml"><mn mathsize="70%" id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.2" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.2.cmml">52.27</mn><mo mathsize="70%" id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.1" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p1.1.m1.1b"><apply id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1"><csymbol cd="latexml" id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.1">percent</csymbol><cn type="float" id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.2">52.27</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p1.1.m1.1c">52.27\%</annotation></semantics></math><span id="S4.SS3.SSS0.Px2.p1.1.4" class="ltx_text" style="font-size:70%;">.
This observation reveals a potential pitfall in employing few-shot CoT: given that these examples are user-crafted, their quality becomes vital.
Meanwhile, we discovered that RaR enables GPT-4 to correct any pitfalls in the logic of the given examples.</span></p>
</div>
<figure id="S4.F9" class="ltx_figure"><img src="/html/2311.04205/assets/x9.png" id="S4.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="179" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Few-shot CoT can effectively instruct the LLM to reason in a similar manner to the provided examples. A badly crafted QA example, as shown in <span id="S4.F9.5.1" class="ltx_text" style="color:#FF0000;">red</span>, result in the LLM following the provided logic but reaching an arbitrary answer.
Slight modification to the few-shot CoT prompt results in notable degradation of LLM’s performance. Meanwhile, our prompt can successfully correct the pitfalls in the few-shot examples and improve the robustness and efficacy of few-shot CoT. </figcaption>
</figure>
<figure id="S4.T6" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 6: </span>Accuracy (%) of GPT-4 on Last Letter Concatenation task. We use one-shot and four-shot examples from <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_text" style="font-size:70%;">Wei et al.</span>, <a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>)</cite>, only changing the intermediate steps to be “first letter” as shown in our example.
Based on our modified examples, the accuracy of GPT-4 are not satisfactory, especially on the task of concatenating four words. Moreover, more of such examples would result in further diminished performance. Meanwhile, such potential risk can be resolved by applying RaR to improve the human-crafted few-shot examples, which increase the accuracy to near <math id="S4.T6.2.m1.1" class="ltx_Math" alttext="100\%" display="inline"><semantics id="S4.T6.2.m1.1b"><mrow id="S4.T6.2.m1.1.1" xref="S4.T6.2.m1.1.1.cmml"><mn id="S4.T6.2.m1.1.1.2" xref="S4.T6.2.m1.1.1.2.cmml">100</mn><mo id="S4.T6.2.m1.1.1.1" xref="S4.T6.2.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.T6.2.m1.1c"><apply id="S4.T6.2.m1.1.1.cmml" xref="S4.T6.2.m1.1.1"><csymbol cd="latexml" id="S4.T6.2.m1.1.1.1.cmml" xref="S4.T6.2.m1.1.1.1">percent</csymbol><cn type="integer" id="S4.T6.2.m1.1.1.2.cmml" xref="S4.T6.2.m1.1.1.2">100</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.2.m1.1d">100\%</annotation></semantics></math>.</figcaption>
<table id="S4.T6.10" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T6.10.1" class="ltx_tr">
<td id="S4.T6.10.1.1" class="ltx_td ltx_border_tt"></td>
<td id="S4.T6.10.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T6.10.1.2.1" class="ltx_text" style="font-size:70%;">Last letter (2)</span></td>
<td id="S4.T6.10.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T6.10.1.3.1" class="ltx_text" style="font-size:70%;">Last letter (4)</span></td>
</tr>
<tr id="S4.T6.10.2" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S4.T6.10.2.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.10.2.1.1" class="ltx_text" style="font-size:70%;background-color:#E0FFFF;">One-shot CoT</span></td>
<td id="S4.T6.10.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.10.2.2.1" class="ltx_text" style="font-size:70%;color:#FF0000;background-color:#E0FFFF;">89.04</span></td>
<td id="S4.T6.10.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.10.2.3.1" class="ltx_text" style="font-size:70%;color:#FF0000;background-color:#E0FFFF;">78.18</span></td>
</tr>
<tr id="S4.T6.10.3" class="ltx_tr">
<td id="S4.T6.10.3.1" class="ltx_td ltx_align_center"><span id="S4.T6.10.3.1.1" class="ltx_text" style="font-size:70%;">RaR+One-shot CoT</span></td>
<td id="S4.T6.10.3.2" class="ltx_td ltx_align_center"><span id="S4.T6.10.3.2.1" class="ltx_text" style="font-size:70%;">100</span></td>
<td id="S4.T6.10.3.3" class="ltx_td ltx_align_center"><span id="S4.T6.10.3.3.1" class="ltx_text" style="font-size:70%;">93.64</span></td>
</tr>
<tr id="S4.T6.10.4" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S4.T6.10.4.1" class="ltx_td ltx_align_center"><span id="S4.T6.10.4.1.1" class="ltx_text" style="font-size:70%;background-color:#E0FFFF;">Four-shot CoT</span></td>
<td id="S4.T6.10.4.2" class="ltx_td ltx_align_center"><span id="S4.T6.10.4.2.1" class="ltx_text" style="font-size:70%;color:#FF0000;background-color:#E0FFFF;">87.21</span></td>
<td id="S4.T6.10.4.3" class="ltx_td ltx_align_center"><span id="S4.T6.10.4.3.1" class="ltx_text" style="font-size:70%;color:#FF0000;background-color:#E0FFFF;">52.27</span></td>
</tr>
<tr id="S4.T6.10.5" class="ltx_tr">
<td id="S4.T6.10.5.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T6.10.5.1.1" class="ltx_text" style="font-size:70%;">RaR+Four-shot CoT</span></td>
<td id="S4.T6.10.5.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T6.10.5.2.1" class="ltx_text" style="font-size:70%;">100</span></td>
<td id="S4.T6.10.5.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T6.10.5.3.1" class="ltx_text" style="font-size:70%;">95.45</span></td>
</tr>
</table>
</figure>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section" style="font-size:70%;">
<span class="ltx_tag ltx_tag_section">5 </span>Related Work</h2>

<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Prompting Methods for LLMs</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p"><span id="S5.SS1.p1.1.1" class="ltx_text" style="font-size:70%;">Since the advent of recent LLMs </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS1.p1.1.2.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">OpenAI</span><span id="S5.SS1.p1.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib22" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Touvron et al.</span><span id="S5.SS1.p1.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib38" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Chiang et al.</span><span id="S5.SS1.p1.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS1.p1.1.4.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p1.1.5" class="ltx_text" style="font-size:70%;">, a growing body of research has focused on prompt engineering for LLMs </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS1.p1.1.6.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Brown et al.</span><span id="S5.SS1.p1.1.7.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib5" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2020</span></a>; <span class="ltx_text" style="font-size:70%;">Schick and Schütze</span><span id="S5.SS1.p1.1.7.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib31" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2021</span></a>; <span class="ltx_text" style="font-size:70%;">Zhou et al.</span><span id="S5.SS1.p1.1.7.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib54" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022b</span></a>; <span class="ltx_text" style="font-size:70%;">White et al.</span><span id="S5.SS1.p1.1.7.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib44" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Wang et al.</span><span id="S5.SS1.p1.1.7.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib40" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS1.p1.1.8.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p1.1.9" class="ltx_text" style="font-size:70%;">.
Manual guidelines have emerged to guide users in designing and revising their prompts </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS1.p1.1.10.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Reynolds and McDonell</span><span id="S5.SS1.p1.1.11.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib29" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2021</span></a>; <span class="ltx_text" style="font-size:70%;">Saravia</span><span id="S5.SS1.p1.1.11.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib30" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S5.SS1.p1.1.12.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p1.1.13" class="ltx_text" style="font-size:70%;">.
Notably, studies have shown that a well-crafted system message, such as ”You are a helpful assistant who always provides explanations,” preceding the main query can encourage an LLM to respond with greater subject expertise </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS1.p1.1.14.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Mukherjee et al.</span><span id="S5.SS1.p1.1.15.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib19" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Ateia and Kruschwitz</span><span id="S5.SS1.p1.1.15.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS1.p1.1.16.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p1.1.17" class="ltx_text" style="font-size:70%;">.
</span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">OpenAI</span> <span id="S5.SS1.p1.1.18.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib21" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S5.SS1.p1.1.19.2.2.1" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p1.1.20" class="ltx_text" style="font-size:70%;"> has also offered general recommendations for crafting queries, emphasizing specificity, detail, and precision. However, individuals often find it challenging to refine their own questions for clarity or to include necessary details for LLMs, as the questions are clear enough for humans themselves.</span></p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p"><span id="S5.SS1.p2.1.1" class="ltx_text" style="font-size:70%;">Subsequent research </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS1.p2.1.2.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Zhou et al.</span><span id="S5.SS1.p2.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib54" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022b</span></a>; <span class="ltx_text" style="font-size:70%;">Sorensen et al.</span><span id="S5.SS1.p2.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib35" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>; <span class="ltx_text" style="font-size:70%;">Pryzant et al.</span><span id="S5.SS1.p2.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS1.p2.1.4.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p2.1.5" class="ltx_text" style="font-size:70%;"> has concentrated on the autonomous refinement of prompts. These methods often employ multiple LLMs to generate candidate prompts, evaluate and score these prompts, and iteratively refine them until a satisfactory prompt is produced.
The evaluation of a prompt typically relies on either the accuracy of an LLM’s response (supervised, </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text" style="font-size:70%;">Zhou et al.</span> <span id="S5.SS1.p2.1.6.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib54" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022b</span></a><span id="S5.SS1.p2.1.7.2.2.1" class="ltx_text" style="font-size:70%;">)</span>; <span class="ltx_text" style="font-size:70%;">Pryzant et al.</span> <span id="S5.SS1.p2.1.6.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS1.p2.1.7.2.2.1" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p2.1.8" class="ltx_text" style="font-size:70%;">) or the mutual information of the question (unsupervised, </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text" style="font-size:70%;">Sorensen et al.</span> <span id="S5.SS1.p2.1.9.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib35" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S5.SS1.p2.1.10.2.2.1" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p2.1.11" class="ltx_text" style="font-size:70%;">). Given the nature of iterative computation and the necessity for qualitative evaluation, such methods are employed for refining single prompt templates; applying them universally to all questions would be expensive. Consequently, these techniques are less frequently adopted in daily user cases.</span></p>
</div>
<div id="S5.SS1.p3" class="ltx_para">
<p id="S5.SS1.p3.1" class="ltx_p"><span id="S5.SS1.p3.1.1" class="ltx_text" style="font-size:70%;">The method most frequently used by users and closely aligned with our approach is the Chain-of-Thought (CoT) prompting, which can be either zero-shot </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS1.p3.1.2.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Kojima et al.</span><span id="S5.SS1.p3.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib14" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S5.SS1.p3.1.4.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p3.1.5" class="ltx_text" style="font-size:70%;"> or few-shot </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS1.p3.1.6.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Wei et al.</span><span id="S5.SS1.p3.1.7.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S5.SS1.p3.1.8.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p3.1.9" class="ltx_text" style="font-size:70%;">. Given that these techniques do not require evaluation and iterative selection, they have gained widespread popularity and inspired a series of subsequent studies </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS1.p3.1.10.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Wang et al.</span><span id="S5.SS1.p3.1.11.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib41" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>; <span class="ltx_text" style="font-size:70%;">Zhou et al.</span><span id="S5.SS1.p3.1.11.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib52" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022a</span></a>; <span class="ltx_text" style="font-size:70%;">Press et al.</span><span id="S5.SS1.p3.1.11.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>; <span class="ltx_text" style="font-size:70%;">Yao et al.</span><span id="S5.SS1.p3.1.11.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib48" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Zhang et al.</span><span id="S5.SS1.p3.1.11.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib51" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023b</span></a>; <span class="ltx_text" style="font-size:70%;">Shao et al.</span><span id="S5.SS1.p3.1.11.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS1.p3.1.12.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p3.1.13" class="ltx_text" style="font-size:70%;">. However, CoT methods are not without their limitations, as observed in our study. Recent investigations have also highlighted challenges with the reliability of both zero-shot CoT </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS1.p3.1.14.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Turpin et al.</span><span id="S5.SS1.p3.1.15.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib39" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS1.p3.1.16.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p3.1.17" class="ltx_text" style="font-size:70%;"> and few-shot CoT </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS1.p3.1.18.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Pawelczyk et al.</span><span id="S5.SS1.p3.1.19.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib24" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS1.p3.1.20.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p3.1.21" class="ltx_text" style="font-size:70%;">. Most recently, </span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text" style="font-size:70%;">Zhou et al.</span> <span id="S5.SS1.p3.1.22.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib53" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS1.p3.1.23.2.2.1" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS1.p3.1.24" class="ltx_text" style="font-size:70%;"> propose Foresee and Reflect similarly as a zero-shot prompting method that targets the proposed task Thinking for Doing (T4D).
Lastly, it is worth noting that our method is complementary to all the prompting techniques mentioned above and can be combined with them.</span></p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Self-correction Methods for LLMs</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p"><span id="S5.SS2.p1.1.1" class="ltx_text" style="font-size:70%;">Another line of work aims at enhancing LLM performance </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS2.p1.1.2.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Madaan et al.</span><span id="S5.SS2.p1.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Welleck et al.</span><span id="S5.SS2.p1.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib43" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>; <span class="ltx_text" style="font-size:70%;">Kim et al.</span><span id="S5.SS2.p1.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib13" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Pan et al.</span><span id="S5.SS2.p1.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib23" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Shinn et al.</span><span id="S5.SS2.p1.1.3.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS2.p1.1.4.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS2.p1.1.5" class="ltx_text" style="font-size:70%;"> by leveraging the LLM to refine its own </span><span id="S5.SS2.p1.1.6" class="ltx_text ltx_font_italic" style="font-size:70%;">responses</span><span id="S5.SS2.p1.1.7" class="ltx_text" style="font-size:70%;">, a concept known as post-hoc prompting. This encompasses terms such as ”self-correction,” ”self-refine,” and ”self-critique,” where LLMs revise their own responses drawing upon various feedback sources or </span><span id="S5.SS2.p1.1.8" class="ltx_text ltx_font_italic" style="font-size:70%;">critic models</span><span id="S5.SS2.p1.1.9" class="ltx_text" style="font-size:70%;">. As classified by </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text" style="font-size:70%;">Pan et al.</span> <span id="S5.SS2.p1.1.10.1.1.1" class="ltx_text" style="font-size:70%;">(</span><a href="#bib.bib23" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS2.p1.1.11.2.2.1" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS2.p1.1.12" class="ltx_text" style="font-size:70%;">, automated critic models generally employ the LLM’s self-feedback </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS2.p1.1.13.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Madaan et al.</span><span id="S5.SS2.p1.1.14.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Shinn et al.</span><span id="S5.SS2.p1.1.14.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Yan et al.</span><span id="S5.SS2.p1.1.14.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib46" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS2.p1.1.15.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS2.p1.1.16" class="ltx_text" style="font-size:70%;">, other trained LLMs </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS2.p1.1.17.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Yang et al.</span><span id="S5.SS2.p1.1.18.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib47" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>; <span class="ltx_text" style="font-size:70%;">Lightman et al.</span><span id="S5.SS2.p1.1.18.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib15" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS2.p1.1.19.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS2.p1.1.20" class="ltx_text" style="font-size:70%;">, or external references </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS2.p1.1.21.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Jung et al.</span><span id="S5.SS2.p1.1.22.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib12" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a>; <span class="ltx_text" style="font-size:70%;">Gao et al.</span><span id="S5.SS2.p1.1.22.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib10" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Yu et al.</span><span id="S5.SS2.p1.1.22.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib49" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Welleck et al.</span><span id="S5.SS2.p1.1.22.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib43" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S5.SS2.p1.1.23.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS2.p1.1.24" class="ltx_text" style="font-size:70%;">. Yet, recent studies </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS2.p1.1.25.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Huang et al.</span><span id="S5.SS2.p1.1.26.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib11" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a>; <span class="ltx_text" style="font-size:70%;">Stechly et al.</span><span id="S5.SS2.p1.1.26.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib36" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2023</span></a><span id="S5.SS2.p1.1.27.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS2.p1.1.28" class="ltx_text" style="font-size:70%;"> examine the self-correction capacities of LLMs and find potential limitations, suggesting that LLMs may not be able to self-correct their reasoning processes. Their findings reveal that self-correction is no better than self-consistency </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS2.p1.1.29.1" class="ltx_text" style="font-size:70%;">(</span><span class="ltx_text" style="font-size:70%;">Wang et al.</span><span id="S5.SS2.p1.1.30.2.1.1" class="ltx_text" style="font-size:70%;">, </span><a href="#bib.bib41" title="" class="ltx_ref"><span class="ltx_text" style="font-size:70%;">2022</span></a><span id="S5.SS2.p1.1.31.3" class="ltx_text" style="font-size:70%;">)</span></cite><span id="S5.SS2.p1.1.32" class="ltx_text" style="font-size:70%;">. Contrary to allowing the LLM to self-refine its responses, our methodology let the LLM instead rephrase questions originally crafted by humans.</span></p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section" style="font-size:70%;">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p"><span id="S6.p1.1.1" class="ltx_text" style="font-size:70%;">In this paper, we have investigated the existing misunderstandings that occur between humans and LLMs and demonstrated that questions that appear clear to humans may still be misinterpreted by LLMs.
Building on this insight, we introduced Rephrase and Respond (RaR), a novel approach that prompts an LLM to first rephrase and clarify the question before answering it.
We also presented Two-step RaR, a variation of RaR that employs a rephrasing LLM to refine questions for subsequent use by any responding LLM.
Our empirical evaluations, conducted across a range of benchmark datasets, confirm the effectiveness of our proposed methods.
Further analysis reveals that while all models gain enhanced performance through question rephrasing, the more sophisticated models exhibit more substantial improvements.
Crucially, we have found that the enhancement in question quality achieved through rephrasing is transferable across models.
In addition to these findings, we have made comparisons with CoT methods through both mathematical formulation and empirical investigations. We also demonstrated that RaR is complementary to CoT, and can be leveraged to achieve additional performance gains.</span></p>
</div>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix" style="font-size:70%;">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Experiment Details</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p"><span id="A1.p1.1.1" class="ltx_text" style="font-size:70%;">Our experiments are done using the publicly available GPT-4 API, as well as the historical version of GPT-4-0613 and GPT-3.5-turbo-0613. We are also considering an open-source LLM model, Vicuna-13B-v1.5. Experiment results for Figure </span><a href="#S3.F5" title="Figure 5 ‣ 3.2 Performance on GPT-4 ‣ 3 RaR Effectively Improves LLM Responses ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">5</span></a><span id="A1.p1.1.2" class="ltx_text" style="font-size:70%;"> are detailed in Table </span><a href="#A1.T7" title="Table 7 ‣ Appendix A Experiment Details ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">7</span></a><span id="A1.p1.1.3" class="ltx_text" style="font-size:70%;">. Moreover, the prompts we used for formatting the answers are detailed in Table </span><a href="#A1.T8" title="Table 8 ‣ Appendix A Experiment Details ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">8</span></a><span id="A1.p1.1.4" class="ltx_text" style="font-size:70%;">. The few-shot examples used in Section </span><a href="#S4.SS3" title="4.3 Empirical Improvement on Few-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">4.3</span></a><span id="A1.p1.1.5" class="ltx_text" style="font-size:70%;"> are demonstrated in Tables </span><a href="#A1.T9" title="Table 9 ‣ Appendix A Experiment Details ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">9</span></a><span id="A1.p1.1.6" class="ltx_text" style="font-size:70%;"> and </span><a href="#A1.T10" title="Table 10 ‣ Appendix A Experiment Details ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref" style="font-size:70%;"><span class="ltx_text ltx_ref_tag">10</span></a><span id="A1.p1.1.7" class="ltx_text" style="font-size:70%;">. In the next appendix section, we provide comprehensive examples of the inputs and outputs of each task for the different methods.</span></p>
</div>
<figure id="A1.T7" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 7: </span>Zero-shot test accuracy of GPT-4 on the original questions of each task, as compared to one-step RaR and two-step RaR.</figcaption>
<div id="A1.T7.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:325.2pt;height:164.1pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(19.9pt,-10.0pt) scale(1.13926110944872,1.13926110944872) ;">
<table id="A1.T7.3.1" class="ltx_tabular ltx_align_middle">
<tr id="A1.T7.3.1.1" class="ltx_tr">
<td id="A1.T7.3.1.1.1" class="ltx_td ltx_border_tt"></td>
<td id="A1.T7.3.1.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T7.3.1.1.2.1" class="ltx_text" style="font-size:70%;">Even day</span></td>
<td id="A1.T7.3.1.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T7.3.1.1.3.1" class="ltx_text" style="font-size:70%;">Even month</span></td>
<td id="A1.T7.3.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T7.3.1.1.4.1" class="ltx_text" style="font-size:70%;">Even year</span></td>
<td id="A1.T7.3.1.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T7.3.1.1.5.1" class="ltx_text" style="font-size:70%;">Compare age</span></td>
<td id="A1.T7.3.1.1.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T7.3.1.1.6.1" class="ltx_text" style="font-size:70%;">CSQA</span></td>
</tr>
<tr id="A1.T7.3.1.2" class="ltx_tr">
<td id="A1.T7.3.1.2.1" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.2.1.1" class="ltx_text" style="font-size:70%;">Original</span></td>
<td id="A1.T7.3.1.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.2.2.1" class="ltx_text" style="font-size:70%;">54.29</span></td>
<td id="A1.T7.3.1.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.2.3.1" class="ltx_text" style="font-size:70%;">58.10</span></td>
<td id="A1.T7.3.1.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.2.4.1" class="ltx_text" style="font-size:70%;">57.14</span></td>
<td id="A1.T7.3.1.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.2.5.1" class="ltx_text" style="font-size:70%;">98.44</span></td>
<td id="A1.T7.3.1.2.6" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.2.6.1" class="ltx_text" style="font-size:70%;">84.09</span></td>
</tr>
<tr id="A1.T7.3.1.3" class="ltx_tr">
<td id="A1.T7.3.1.3.1" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.3.1.1" class="ltx_text" style="font-size:70%;">2-step RaR</span></td>
<td id="A1.T7.3.1.3.2" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.3.2.1" class="ltx_text" style="font-size:70%;">88.18</span></td>
<td id="A1.T7.3.1.3.3" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.3.3.1" class="ltx_text" style="font-size:70%;">87.73</span></td>
<td id="A1.T7.3.1.3.4" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.3.4.1" class="ltx_text" style="font-size:70%;">93.18</span></td>
<td id="A1.T7.3.1.3.5" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.3.5.1" class="ltx_text" style="font-size:70%;">100</span></td>
<td id="A1.T7.3.1.3.6" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.3.6.1" class="ltx_text ltx_font_bold" style="font-size:70%;">85.91</span></td>
</tr>
<tr id="A1.T7.3.1.4" class="ltx_tr">
<td id="A1.T7.3.1.4.1" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.4.1.1" class="ltx_text" style="font-size:70%;">1-step RaR</span></td>
<td id="A1.T7.3.1.4.2" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.4.2.1" class="ltx_text ltx_font_bold" style="font-size:70%;">95.00</span></td>
<td id="A1.T7.3.1.4.3" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.4.3.1" class="ltx_text ltx_font_bold" style="font-size:70%;">90.91</span></td>
<td id="A1.T7.3.1.4.4" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.4.4.1" class="ltx_text ltx_font_bold" style="font-size:70%;">95.00</span></td>
<td id="A1.T7.3.1.4.5" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.4.5.1" class="ltx_text ltx_font_bold" style="font-size:70%;">100</span></td>
<td id="A1.T7.3.1.4.6" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.4.6.1" class="ltx_text" style="font-size:70%;">85.45</span></td>
</tr>
<tr id="A1.T7.3.1.5" class="ltx_tr">
<td id="A1.T7.3.1.5.1" class="ltx_td ltx_border_tt ltx_border_tt"></td>
<td id="A1.T7.3.1.5.2" class="ltx_td ltx_align_center ltx_border_tt ltx_border_tt"><span id="A1.T7.3.1.5.2.1" class="ltx_text" style="font-size:70%;">Dates</span></td>
<td id="A1.T7.3.1.5.3" class="ltx_td ltx_align_center ltx_border_tt ltx_border_tt"><span id="A1.T7.3.1.5.3.1" class="ltx_text" style="font-size:70%;">Sports</span></td>
<td id="A1.T7.3.1.5.4" class="ltx_td ltx_align_center ltx_border_tt ltx_border_tt"><span id="A1.T7.3.1.5.4.1" class="ltx_text" style="font-size:70%;">Last letter (2)</span></td>
<td id="A1.T7.3.1.5.5" class="ltx_td ltx_align_center ltx_border_tt ltx_border_tt"><span id="A1.T7.3.1.5.5.1" class="ltx_text" style="font-size:70%;">Last letter (4)</span></td>
<td id="A1.T7.3.1.5.6" class="ltx_td ltx_align_center ltx_border_tt ltx_border_tt"><span id="A1.T7.3.1.5.6.1" class="ltx_text" style="font-size:70%;">Coin flip</span></td>
</tr>
<tr id="A1.T7.3.1.6" class="ltx_tr">
<td id="A1.T7.3.1.6.1" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.6.1.1" class="ltx_text" style="font-size:70%;">Original</span></td>
<td id="A1.T7.3.1.6.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.6.2.1" class="ltx_text" style="font-size:70%;">69.92</span></td>
<td id="A1.T7.3.1.6.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.6.3.1" class="ltx_text" style="font-size:70%;">79.55</span></td>
<td id="A1.T7.3.1.6.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.6.4.1" class="ltx_text" style="font-size:70%;">52.05</span></td>
<td id="A1.T7.3.1.6.5" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.6.5.1" class="ltx_text" style="font-size:70%;">21.36</span></td>
<td id="A1.T7.3.1.6.6" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T7.3.1.6.6.1" class="ltx_text" style="font-size:70%;">74.55</span></td>
</tr>
<tr id="A1.T7.3.1.7" class="ltx_tr">
<td id="A1.T7.3.1.7.1" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.7.1.1" class="ltx_text" style="font-size:70%;">2-step RaR</span></td>
<td id="A1.T7.3.1.7.2" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.7.2.1" class="ltx_text" style="font-size:70%;">80.45</span></td>
<td id="A1.T7.3.1.7.3" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.7.3.1" class="ltx_text ltx_font_bold" style="font-size:70%;">76.42</span></td>
<td id="A1.T7.3.1.7.4" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.7.4.1" class="ltx_text ltx_font_bold" style="font-size:70%;">99.09</span></td>
<td id="A1.T7.3.1.7.5" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.7.5.1" class="ltx_text ltx_font_bold" style="font-size:70%;">86.82</span></td>
<td id="A1.T7.3.1.7.6" class="ltx_td ltx_align_center"><span id="A1.T7.3.1.7.6.1" class="ltx_text" style="font-size:70%;">98.18</span></td>
</tr>
<tr id="A1.T7.3.1.8" class="ltx_tr">
<td id="A1.T7.3.1.8.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T7.3.1.8.1.1" class="ltx_text" style="font-size:70%;">1-step RaR</span></td>
<td id="A1.T7.3.1.8.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T7.3.1.8.2.1" class="ltx_text ltx_font_bold" style="font-size:70%;">84.54</span></td>
<td id="A1.T7.3.1.8.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T7.3.1.8.3.1" class="ltx_text" style="font-size:70%;">75.88</span></td>
<td id="A1.T7.3.1.8.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T7.3.1.8.4.1" class="ltx_text" style="font-size:70%;">96.36</span></td>
<td id="A1.T7.3.1.8.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T7.3.1.8.5.1" class="ltx_text" style="font-size:70%;">68.64</span></td>
<td id="A1.T7.3.1.8.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T7.3.1.8.6.1" class="ltx_text ltx_font_bold" style="font-size:70%;">99.09</span></td>
</tr>
</table>
</span></div>
</figure>
<figure id="A1.T8" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 8: </span>Formatting prompts that are appended to both the original questions and RaR prompts.</figcaption>
<div id="A1.T8.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:238.5pt;height:127.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(18.2pt,-9.7pt) scale(1.17977798045654,1.17977798045654) ;">
<table id="A1.T8.3.1" class="ltx_tabular ltx_align_middle">
<tr id="A1.T8.3.1.1" class="ltx_tr">
<td id="A1.T8.3.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="A1.T8.3.1.1.1.1" class="ltx_text" style="font-size:70%;">Task</span></td>
<td id="A1.T8.3.1.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T8.3.1.1.2.1" class="ltx_text" style="font-size:70%;">Format prompt</span></td>
</tr>
<tr id="A1.T8.3.1.2" class="ltx_tr">
<td id="A1.T8.3.1.2.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="A1.T8.3.1.2.1.1" class="ltx_text" style="font-size:70%;">Date</span></td>
<td id="A1.T8.3.1.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T8.3.1.2.2.1" class="ltx_text" style="font-size:70%;">Final answer format should be MM/DD/YYYY.</span></td>
</tr>
<tr id="A1.T8.3.1.3" class="ltx_tr">
<td id="A1.T8.3.1.3.1" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T8.3.1.3.1.1" class="ltx_text" style="font-size:70%;">CSQA</span></td>
<td id="A1.T8.3.1.3.2" class="ltx_td ltx_align_center"><span id="A1.T8.3.1.3.2.1" class="ltx_text" style="font-size:70%;">Select the *single* most sensible answer.</span></td>
</tr>
<tr id="A1.T8.3.1.4" class="ltx_tr">
<td id="A1.T8.3.1.4.1" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T8.3.1.4.1.1" class="ltx_text" style="font-size:70%;">StereoSet</span></td>
<td id="A1.T8.3.1.4.2" class="ltx_td ltx_align_center"><span id="A1.T8.3.1.4.2.1" class="ltx_text" style="font-size:70%;">Select the *single* most sensible answer.</span></td>
</tr>
<tr id="A1.T8.3.1.5" class="ltx_tr">
<td id="A1.T8.3.1.5.1" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T8.3.1.5.1.1" class="ltx_text" style="font-size:70%;">Sports</span></td>
<td id="A1.T8.3.1.5.2" class="ltx_td ltx_align_center"><span id="A1.T8.3.1.5.2.1" class="ltx_text" style="font-size:70%;">Answer the Yes or No question.</span></td>
</tr>
<tr id="A1.T8.3.1.6" class="ltx_tr">
<td id="A1.T8.3.1.6.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="A1.T8.3.1.6.1.1" class="ltx_text" style="font-size:70%;">Coin flip</span></td>
<td id="A1.T8.3.1.6.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T8.3.1.6.2.1" class="ltx_text" style="font-size:70%;">Answer the Yes or No question.</span></td>
</tr>
</table>
</span></div>
</figure>
<figure id="A1.T9" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 9: </span>Few-shot examples improved by RaR used in Sec <a href="#S4.SS3" title="4.3 Empirical Improvement on Few-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a> for Last Letter Concatenation.</figcaption>
<table id="A1.T9.4" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A1.T9.4.1" class="ltx_tr">
<td id="A1.T9.4.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A1.T9.4.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.4.1.1.1.1" class="ltx_p"><span id="A1.T9.4.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Q</span><span id="A1.T9.4.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: Take the last letters of the words in “Elon Musk” and concatenate them.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.4.2" class="ltx_tr">
<td id="A1.T9.4.2.1" class="ltx_td ltx_align_justify">
<span id="A1.T9.4.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.4.2.1.1.1" class="ltx_p"><span id="A1.T9.4.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">A</span><span id="A1.T9.4.2.1.1.1.2" class="ltx_text" style="font-size:70%;">: The first letter of “Elon” is “E”. The first letter of “Musk” is “M”. Concatenating them is “EM”. The answer is nk.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.4.3" class="ltx_tr">
<td id="A1.T9.4.3.1" class="ltx_td ltx_align_justify">
<span id="A1.T9.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.4.3.1.1.1" class="ltx_p"><span id="A1.T9.4.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Q</span><span id="A1.T9.4.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: Take the last letters of the words in “Larry Page” and concatenate them.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.4.4" class="ltx_tr">
<td id="A1.T9.4.4.1" class="ltx_td ltx_align_justify">
<span id="A1.T9.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.4.4.1.1.1" class="ltx_p"><span id="A1.T9.4.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">A</span><span id="A1.T9.4.4.1.1.1.2" class="ltx_text" style="font-size:70%;">: The first letter of “Larry” is “L”. The first letter of “Page” is “P”. Concatenating them is “LP”. The answer is ye.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.4.5" class="ltx_tr">
<td id="A1.T9.4.5.1" class="ltx_td ltx_align_justify">
<span id="A1.T9.4.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.4.5.1.1.1" class="ltx_p"><span id="A1.T9.4.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Q</span><span id="A1.T9.4.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: Take the last letters of the words in “Sergey Brin” and concatenate them.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.4.6" class="ltx_tr">
<td id="A1.T9.4.6.1" class="ltx_td ltx_align_justify">
<span id="A1.T9.4.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.4.6.1.1.1" class="ltx_p"><span id="A1.T9.4.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">A</span><span id="A1.T9.4.6.1.1.1.2" class="ltx_text" style="font-size:70%;">: The first letter of “Sergey” is “S”. The first letter of “Brin” is “B”. Concatenating them is “SB”. The answer is yn.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.4.7" class="ltx_tr">
<td id="A1.T9.4.7.1" class="ltx_td ltx_align_justify">
<span id="A1.T9.4.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.4.7.1.1.1" class="ltx_p"><span id="A1.T9.4.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Q</span><span id="A1.T9.4.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: Take the last letters of the words in “Bill Gates” and concatenate them.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.4.8" class="ltx_tr">
<td id="A1.T9.4.8.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A1.T9.4.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.4.8.1.1.1" class="ltx_p"><span id="A1.T9.4.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">A</span><span id="A1.T9.4.8.1.1.1.2" class="ltx_text" style="font-size:70%;">: The first letter of “Bill” is “B”. The first letter of “Gates” is “G”. Concatenating them is “BG”. The answer is ls.</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A1.T10" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 10: </span>Few-shot examples improved by RaR used in Sec <a href="#S4.SS3" title="4.3 Empirical Improvement on Few-Shot CoT ‣ 4 Comparison with Chain-of-Thought ‣ Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a> for Last Letter Concatenation.</figcaption>
<table id="A1.T10.4" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A1.T10.4.1" class="ltx_tr">
<td id="A1.T10.4.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A1.T10.4.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.4.1.1.1.1" class="ltx_p"><span id="A1.T10.4.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Q</span><span id="A1.T10.4.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: Identify the last letters of each word in the name “Elon Musk”, then put those letters together.</span></span>
</span>
</td>
</tr>
<tr id="A1.T10.4.2" class="ltx_tr">
<td id="A1.T10.4.2.1" class="ltx_td ltx_align_justify">
<span id="A1.T10.4.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.4.2.1.1.1" class="ltx_p"><span id="A1.T10.4.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">A</span><span id="A1.T10.4.2.1.1.1.2" class="ltx_text" style="font-size:70%;">: The last letter of ‘Elon’ is ‘n’. The last letter of ‘Musk’ is ‘k’. When you put these letters together, the result is ‘nk’</span></span>
</span>
</td>
</tr>
<tr id="A1.T10.4.3" class="ltx_tr">
<td id="A1.T10.4.3.1" class="ltx_td ltx_align_justify">
<span id="A1.T10.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.4.3.1.1.1" class="ltx_p"><span id="A1.T10.4.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Q</span><span id="A1.T10.4.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: Identify the last letters of each word in the name “Larry Page”, then put those letters together.</span></span>
</span>
</td>
</tr>
<tr id="A1.T10.4.4" class="ltx_tr">
<td id="A1.T10.4.4.1" class="ltx_td ltx_align_justify">
<span id="A1.T10.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.4.4.1.1.1" class="ltx_p"><span id="A1.T10.4.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">A</span><span id="A1.T10.4.4.1.1.1.2" class="ltx_text" style="font-size:70%;">: The last letter of ‘Larry’ is ‘y’. The last letter of ‘Page’ is ‘e’. When you put these letters together, the result is ‘ye’</span></span>
</span>
</td>
</tr>
<tr id="A1.T10.4.5" class="ltx_tr">
<td id="A1.T10.4.5.1" class="ltx_td ltx_align_justify">
<span id="A1.T10.4.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.4.5.1.1.1" class="ltx_p"><span id="A1.T10.4.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Q</span><span id="A1.T10.4.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: Identify the last letters of each word in the name “Sergey Brin”, then put those letters together.</span></span>
</span>
</td>
</tr>
<tr id="A1.T10.4.6" class="ltx_tr">
<td id="A1.T10.4.6.1" class="ltx_td ltx_align_justify">
<span id="A1.T10.4.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.4.6.1.1.1" class="ltx_p"><span id="A1.T10.4.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">A</span><span id="A1.T10.4.6.1.1.1.2" class="ltx_text" style="font-size:70%;">: The last letter of ‘Sergey’ is ‘y’. The last letter of ‘Brin’ is ‘n’. When you put these letters together, the result is ‘yn’</span></span>
</span>
</td>
</tr>
<tr id="A1.T10.4.7" class="ltx_tr">
<td id="A1.T10.4.7.1" class="ltx_td ltx_align_justify">
<span id="A1.T10.4.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.4.7.1.1.1" class="ltx_p"><span id="A1.T10.4.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Q</span><span id="A1.T10.4.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: Identify the last letters of each word in the name “Bill Gates”, then put those letters together.</span></span>
</span>
</td>
</tr>
<tr id="A1.T10.4.8" class="ltx_tr">
<td id="A1.T10.4.8.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A1.T10.4.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.4.8.1.1.1" class="ltx_p"><span id="A1.T10.4.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">A</span><span id="A1.T10.4.8.1.1.1.2" class="ltx_text" style="font-size:70%;">: The last letter of ‘Bill’ is ‘l’. The last letter of ‘Gates’ is ‘s’. When you put these letters together, the result is ‘ls’</span></span>
</span>
</td>
</tr>
</table>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix" style="font-size:70%;">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Input/Output Examples</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p"><span id="A2.p1.1.1" class="ltx_text" style="font-size:70%;">In this section, we provide specific input and output examples of GPT-4 on each of task we considered, using either original questions or RaR.</span></p>
</div>
<figure id="A2.T11" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 11: </span>Examples of correct and incorrect responses by GPT-4 to the original questions of <span id="A2.T11.5.1" class="ltx_text ltx_font_bold">Even day</span>.</figcaption>
<table id="A2.T11.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T11.6.1" class="ltx_tr">
<td id="A2.T11.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T11.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.1.1.1.1" class="ltx_p"><span id="A2.T11.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T11.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: Was Barack Obama born in an even day?</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.2" class="ltx_tr">
<td id="A2.T11.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T11.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.2.1.1.1" class="ltx_p"><span id="A2.T11.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T11.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T11.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T11.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: No, Barack Obama was born on August 4, 1961, which is an even date but an odd day.</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.3" class="ltx_tr">
<td id="A2.T11.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T11.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.3.1.1.1" class="ltx_p"><span id="A2.T11.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T11.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: Was Kanye West born in an even day?</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.4" class="ltx_tr">
<td id="A2.T11.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T11.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.4.1.1.1" class="ltx_p"><span id="A2.T11.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T11.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T11.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T11.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: No, Kanye West was born on June 8, 1977, which is an even day.</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.5" class="ltx_tr">
<td id="A2.T11.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T11.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.5.1.1.1" class="ltx_p"><span id="A2.T11.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T11.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: Was Stephen Hawking born in an even day?</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.6" class="ltx_tr">
<td id="A2.T11.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T11.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.6.1.1.1" class="ltx_p"><span id="A2.T11.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T11.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T11.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T11.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: No, Stephen Hawking was born on January 8, 1942. Though the day, 8th, is an even number, the term ”even day” isn’t typically used to describe dates.</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.7" class="ltx_tr">
<td id="A2.T11.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T11.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.7.1.1.1" class="ltx_p"><span id="A2.T11.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T11.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: Was Frida Kahlo born in an even day?</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.8" class="ltx_tr">
<td id="A2.T11.6.8.1" class="ltx_td ltx_align_justify">
<span id="A2.T11.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.8.1.1.1" class="ltx_p"><span id="A2.T11.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T11.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T11.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T11.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: No, Frida Kahlo was born on July 6, 1907. Therefore, she was born on an odd day.</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.9" class="ltx_tr">
<td id="A2.T11.6.9.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T11.6.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.9.1.1.1" class="ltx_p"><span id="A2.T11.6.9.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T11.6.9.1.1.1.2" class="ltx_text" style="font-size:70%;">: Was Leonardo da Vinci born in an even day?</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.10" class="ltx_tr">
<td id="A2.T11.6.10.1" class="ltx_td ltx_align_justify">
<span id="A2.T11.6.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.10.1.1.1" class="ltx_p"><span id="A2.T11.6.10.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T11.6.10.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T11.6.10.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T11.6.10.1.1.1.4" class="ltx_text" style="font-size:70%;">: No, Leonardo da Vinci was born on April 15, 1452, which is an odd day.</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.11" class="ltx_tr">
<td id="A2.T11.6.11.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T11.6.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.11.1.1.1" class="ltx_p"><span id="A2.T11.6.11.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T11.6.11.1.1.1.2" class="ltx_text" style="font-size:70%;">: Was Ludwig van Beethoven born in an even day?</span></span>
</span>
</td>
</tr>
<tr id="A2.T11.6.12" class="ltx_tr">
<td id="A2.T11.6.12.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T11.6.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T11.6.12.1.1.1" class="ltx_p"><span id="A2.T11.6.12.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T11.6.12.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T11.6.12.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T11.6.12.1.1.1.4" class="ltx_text" style="font-size:70%;">: No, Ludwig van Beethoven was born on December 17, 1770, which is an odd day.</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T12" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 12: </span>Examples of correct and incorrect responses by GPT-4 to 1-step RaR on <span id="A2.T12.5.1" class="ltx_text ltx_font_bold">Even day</span>. We omit ““n” and leave it as a blank space for better readability.</figcaption>
<table id="A2.T12.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T12.6.1" class="ltx_tr">
<td id="A2.T12.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T12.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.1.1.1.1" class="ltx_p"><span id="A2.T12.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T12.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Was Barack Obama born in an even day?” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.2" class="ltx_tr">
<td id="A2.T12.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T12.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.2.1.1.1" class="ltx_p"><span id="A2.T12.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T12.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T12.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T12.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: “Did the former United States President, Barack Obama, have his birthday fall on an even numbered day of a month?” Barack Obama was born on August 4, 1961. So </span><span id="A2.T12.6.2.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">yes</span><span id="A2.T12.6.2.1.1.1.6" class="ltx_text" style="font-size:70%;">, he was born on an even numbered day.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.3" class="ltx_tr">
<td id="A2.T12.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T12.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.3.1.1.1" class="ltx_p"><span id="A2.T12.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T12.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Was Kanye West born in an even day?” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.4" class="ltx_tr">
<td id="A2.T12.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T12.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.4.1.1.1" class="ltx_p"><span id="A2.T12.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T12.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T12.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T12.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: The question asks, “Was the day of Kanye West’s birth an even-numbered day?” Kanye West was born on June 8, 1977. Thus, </span><span id="A2.T12.6.4.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">yes</span><span id="A2.T12.6.4.1.1.1.6" class="ltx_text" style="font-size:70%;">, he was born on an even day, the 8th of June.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.5" class="ltx_tr">
<td id="A2.T12.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T12.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.5.1.1.1" class="ltx_p"><span id="A2.T12.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T12.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Was Stephen Hawking born in an even day?” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.6" class="ltx_tr">
<td id="A2.T12.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T12.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.6.1.1.1" class="ltx_p"><span id="A2.T12.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T12.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T12.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T12.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: The expanded version of the question could be phrased as, “Was renowned physicist and author Stephen Hawking born on a date that is classified as an even day?”. Stephen Hawking was born on January 8, 1942, so the answer is </span><span id="A2.T12.6.6.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">yes</span><span id="A2.T12.6.6.1.1.1.6" class="ltx_text" style="font-size:70%;">; he was born on an even day.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.7" class="ltx_tr">
<td id="A2.T12.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T12.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.7.1.1.1" class="ltx_p"><span id="A2.T12.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T12.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Was Frida Kahlo born in an even day?” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.8" class="ltx_tr">
<td id="A2.T12.6.8.1" class="ltx_td ltx_align_justify">
<span id="A2.T12.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.8.1.1.1" class="ltx_p"><span id="A2.T12.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T12.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T12.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T12.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: “Did the famous Mexican artist Frida Kahlo have her birth date fall on an even numbered day of the month?” </span><span id="A2.T12.6.8.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">Yes</span><span id="A2.T12.6.8.1.1.1.6" class="ltx_text" style="font-size:70%;">, she was. Frida Kahlo was born on July 6th, 1907, which is an even day of the month.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.9" class="ltx_tr">
<td id="A2.T12.6.9.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T12.6.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.9.1.1.1" class="ltx_p"><span id="A2.T12.6.9.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T12.6.9.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Was Leonardo da Vinci born in an even day?” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.10" class="ltx_tr">
<td id="A2.T12.6.10.1" class="ltx_td ltx_align_justify">
<span id="A2.T12.6.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.10.1.1.1" class="ltx_p"><span id="A2.T12.6.10.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T12.6.10.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T12.6.10.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T12.6.10.1.1.1.4" class="ltx_text" style="font-size:70%;">: “Did the day of birth for the renowned artist and inventor, Leonardo da Vinci, fall on an even-numbered day?” Yes, Leonardo da Vinci was born on April 15, 1452, which is an odd-numbered day.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.11" class="ltx_tr">
<td id="A2.T12.6.11.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T12.6.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.11.1.1.1" class="ltx_p"><span id="A2.T12.6.11.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T12.6.11.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Was Ludwig van Beethoven born in an even day?” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T12.6.12" class="ltx_tr">
<td id="A2.T12.6.12.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T12.6.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T12.6.12.1.1.1" class="ltx_p"><span id="A2.T12.6.12.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T12.6.12.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T12.6.12.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T12.6.12.1.1.1.4" class="ltx_text" style="font-size:70%;">: “Was the celebrated composer Ludwig van Beethoven born on a day that falls under an even-numbered date?” Ludwig van Beethoven was born on December 17, 1770. Since 17 is an odd number, the answer is </span><span id="A2.T12.6.12.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">no</span><span id="A2.T12.6.12.1.1.1.6" class="ltx_text" style="font-size:70%;">, Beethoven was not born on an even day.</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T13" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 13: </span>Examples of correct and incorrect responses by GPT-4 to 1-step RaR on <span id="A2.T13.5.1" class="ltx_text ltx_font_bold">Even day</span>. We omit ““n” and leave it as a blank space for better readability.</figcaption>
<table id="A2.T13.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T13.6.1" class="ltx_tr">
<td id="A2.T13.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T13.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.1.1.1.1" class="ltx_p"><span id="A2.T13.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T13.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Was Barack Obama born in an even day? (rephrased) Was the 44th President of the United States, Barack Obama, born on a day of the month that is classified as an even number? Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.2" class="ltx_tr">
<td id="A2.T13.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T13.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.2.1.1.1" class="ltx_p"><span id="A2.T13.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T13.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T13.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T13.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: No, Barack Obama was not born on an even day. He was born on August 4, 1961.</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.3" class="ltx_tr">
<td id="A2.T13.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T13.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.3.1.1.1" class="ltx_p"><span id="A2.T13.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T13.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Was Kanye West born in an even day? (rephrased) Can you tell me if the day of the month on which Kanye West was born falls on an even number? Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.4" class="ltx_tr">
<td id="A2.T13.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T13.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.4.1.1.1" class="ltx_p"><span id="A2.T13.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T13.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T13.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T13.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: Kanye West was born on June 8, 1977. Since 8 is an even number, the answer to both questions is </span><span id="A2.T13.6.4.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">yes</span><span id="A2.T13.6.4.1.1.1.6" class="ltx_text" style="font-size:70%;">.</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.5" class="ltx_tr">
<td id="A2.T13.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T13.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.5.1.1.1" class="ltx_p"><span id="A2.T13.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T13.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Was Stephen Hawking born in an even day? (rephrased) Stephen Hawking, did his birth take place on a day that falls into the category of even numbers? Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.6" class="ltx_tr">
<td id="A2.T13.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T13.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.6.1.1.1" class="ltx_p"><span id="A2.T13.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T13.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T13.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T13.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: Stephen Hawking was born on January 8, 1942. Therefore, the answer to both the original and rephrased questions is </span><span id="A2.T13.6.6.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">“yes.”</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.7" class="ltx_tr">
<td id="A2.T13.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T13.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.7.1.1.1" class="ltx_p"><span id="A2.T13.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T13.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Was Frida Kahlo born in an even day? (rephrased) Was the famous artist Frida Kahlo born on a day of the month that is an even number? Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.8" class="ltx_tr">
<td id="A2.T13.6.8.1" class="ltx_td ltx_align_justify">
<span id="A2.T13.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.8.1.1.1" class="ltx_p"><span id="A2.T13.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T13.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T13.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T13.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: Frida Kahlo was born on July 6, 1907. Thus, she was not born on an even day.</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.9" class="ltx_tr">
<td id="A2.T13.6.9.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T13.6.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.9.1.1.1" class="ltx_p"><span id="A2.T13.6.9.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T13.6.9.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Was Leonardo da Vinci born in an even day? (rephrased) Was the historical figure Leonardo da Vinci, renowned for his artistic and scientific achievements, born on a day that falls under an even number? Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.10" class="ltx_tr">
<td id="A2.T13.6.10.1" class="ltx_td ltx_align_justify">
<span id="A2.T13.6.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.10.1.1.1" class="ltx_p"><span id="A2.T13.6.10.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T13.6.10.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T13.6.10.1.1.1.3" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T13.6.10.1.1.1.4" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T13.6.10.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T13.6.10.1.1.1.6" class="ltx_text" style="font-size:70%;">: Leonardo da Vinci was born on April 15, 1452. 15 is an odd number, so he was </span><span id="A2.T13.6.10.1.1.1.7" class="ltx_text" style="font-size:70%;color:#00FF00;">not</span><span id="A2.T13.6.10.1.1.1.8" class="ltx_text" style="font-size:70%;"> born on an even day.</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.11" class="ltx_tr">
<td id="A2.T13.6.11.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T13.6.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.11.1.1.1" class="ltx_p"><span id="A2.T13.6.11.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T13.6.11.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Was the celebrated composer Ludwig van Beethoven born on a day that falls under an even-numbered date? (rephrased) Did the birth of the famous composer Ludwig van Beethoven fall on a day of the month that is considered an even number? Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T13.6.12" class="ltx_tr">
<td id="A2.T13.6.12.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T13.6.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T13.6.12.1.1.1" class="ltx_p"><span id="A2.T13.6.12.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T13.6.12.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T13.6.12.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T13.6.12.1.1.1.4" class="ltx_text" style="font-size:70%;">: Ludwig van Beethoven was born on December 17, 1770. 17 is an odd number, not an even one. Therefore, the answer to both the original and rephrased question is </span><span id="A2.T13.6.12.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">no</span><span id="A2.T13.6.12.1.1.1.6" class="ltx_text" style="font-size:70%;">, Ludwig van Beethoven was not born on an even day.</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T14" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 14: </span>Examples of correct and incorrect responses by GPT-4 to the original questions of <span id="A2.T14.5.1" class="ltx_text ltx_font_bold">Coin flip</span>.</figcaption>
<table id="A2.T14.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T14.6.1" class="ltx_tr">
<td id="A2.T14.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T14.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T14.6.1.1.1.1" class="ltx_p"><span id="A2.T14.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T14.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: A coin is heads up. aluino flips the coin. arthor flips the coin. Is the coin still heads up? Flip means reverse. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T14.6.2" class="ltx_tr">
<td id="A2.T14.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T14.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T14.6.2.1.1.1" class="ltx_p"><span id="A2.T14.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T14.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T14.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T14.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: No</span></span>
</span>
</td>
</tr>
<tr id="A2.T14.6.3" class="ltx_tr">
<td id="A2.T14.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T14.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T14.6.3.1.1.1" class="ltx_p"><span id="A2.T14.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T14.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: A coin is heads up. jumaimah flips the coin. buity flips the coin. Is the coin still heads up? Flip means reverse. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T14.6.4" class="ltx_tr">
<td id="A2.T14.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T14.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T14.6.4.1.1.1" class="ltx_p"><span id="A2.T14.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T14.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T14.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T14.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: No</span></span>
</span>
</td>
</tr>
<tr id="A2.T14.6.5" class="ltx_tr">
<td id="A2.T14.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T14.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T14.6.5.1.1.1" class="ltx_p"><span id="A2.T14.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T14.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: A coin is heads up. kinta does not flip the coin. renit does not flip the coin. Is the coin still heads up? Flip means reverse. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T14.6.6" class="ltx_tr">
<td id="A2.T14.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T14.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T14.6.6.1.1.1" class="ltx_p"><span id="A2.T14.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T14.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T14.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T14.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: Yes</span></span>
</span>
</td>
</tr>
<tr id="A2.T14.6.7" class="ltx_tr">
<td id="A2.T14.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T14.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T14.6.7.1.1.1" class="ltx_p"><span id="A2.T14.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T14.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: A coin is heads up. brandarius does not flip the coin. gitta flips the coin. Is the coin still heads up? Flip means reverse. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T14.6.8" class="ltx_tr">
<td id="A2.T14.6.8.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T14.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T14.6.8.1.1.1" class="ltx_p"><span id="A2.T14.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T14.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T14.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T14.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: No.</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T15" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 15: </span>Examples of correct and incorrect responses by GPT-4 using 1-step RaR on <span id="A2.T15.5.1" class="ltx_text ltx_font_bold">Coin flip</span>.</figcaption>
<table id="A2.T15.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T15.6.1" class="ltx_tr">
<td id="A2.T15.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T15.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T15.6.1.1.1.1" class="ltx_p"><span id="A2.T15.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T15.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: “A coin is heads up. aluino flips the coin. arthor flips the coin. Is the coin still heads up? Flip means reverse.” Rephrase and expand the question, and respond. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T15.6.2" class="ltx_tr">
<td id="A2.T15.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T15.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T15.6.2.1.1.1" class="ltx_p"><span id="A2.T15.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T15.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T15.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T15.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: The coin started out being heads up. Then, Aluino flipped the coin, reversing its side. After that, Arthor also flipped the coin, reversing its side again. Is the coin facing heads up now? Considering the action of flipping the coin means to reverse its side, after two flips, the coin would indeed be back in its original position. So, </span><span id="A2.T15.6.2.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">yes</span><span id="A2.T15.6.2.1.1.1.6" class="ltx_text" style="font-size:70%;">, the coin is still heads up.</span></span>
</span>
</td>
</tr>
<tr id="A2.T15.6.3" class="ltx_tr">
<td id="A2.T15.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T15.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T15.6.3.1.1.1" class="ltx_p"><span id="A2.T15.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T15.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: “A coin is heads up. jumaimah flips the coin. buity flips the coin. Is the coin still heads up? Flip means reverse.” Rephrase and expand the question, and respond. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T15.6.4" class="ltx_tr">
<td id="A2.T15.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T15.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T15.6.4.1.1.1" class="ltx_p"><span id="A2.T15.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T15.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T15.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T15.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: The coin is initially facing heads up. Then, it is flipped once by Jumaimah and again by Buity. After these two flips, is the coin still facing heads up? Taking into consideration that flipping the coin means to turn it over to its opposite side, the answer is </span><span id="A2.T15.6.4.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">Yes</span><span id="A2.T15.6.4.1.1.1.6" class="ltx_text" style="font-size:70%;">, the coin is still heads up after two flips.</span></span>
</span>
</td>
</tr>
<tr id="A2.T15.6.5" class="ltx_tr">
<td id="A2.T15.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T15.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T15.6.5.1.1.1" class="ltx_p"><span id="A2.T15.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T15.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: “A coin is heads up. kinta does not flip the coin. renit does not flip the coin. Is the coin still heads up? Flip means reverse.” Rephrase and expand the question, and respond. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T15.6.6" class="ltx_tr">
<td id="A2.T15.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T15.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T15.6.6.1.1.1" class="ltx_p"><span id="A2.T15.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T15.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T15.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T15.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: Has the position of the coin, which was initially heads up, changed considering that neither Kinta nor Renit have flipped, implying reversed, the coin? The answer is </span><span id="A2.T15.6.6.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">yes</span><span id="A2.T15.6.6.1.1.1.6" class="ltx_text" style="font-size:70%;">, the coin is still heads up.</span></span>
</span>
</td>
</tr>
<tr id="A2.T15.6.7" class="ltx_tr">
<td id="A2.T15.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T15.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T15.6.7.1.1.1" class="ltx_p"><span id="A2.T15.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T15.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: “A coin is heads up. brandarius does not flip the coin. gitta flips the coin. Is the coin still heads up? Flip means reverse.” Rephrase and expand the question, and respond. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T15.6.8" class="ltx_tr">
<td id="A2.T15.6.8.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T15.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T15.6.8.1.1.1" class="ltx_p"><span id="A2.T15.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T15.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T15.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T15.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: Rephrased and expanded question: Initially, the coin was facing heads up. The scenario suggests that brandarius doesn’t flip or reverse the coin but gitta does. Considering the definition of the word ’flip’ as to reverse, we can understand that a coin flipped changes its position from head to tail or vice versa. So, would the coin maintain its initial position of being heads up after gitta flipped it?Answer: </span><span id="A2.T15.6.8.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">No.</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T16" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 16: </span>Examples of correct and incorrect responses by GPT-4 using 2-step RaR on <span id="A2.T16.5.1" class="ltx_text ltx_font_bold">Coin flip</span>.</figcaption>
<table id="A2.T16.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T16.6.1" class="ltx_tr">
<td id="A2.T16.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T16.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T16.6.1.1.1.1" class="ltx_p"><span id="A2.T16.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T16.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) A coin is heads up. aluino flips the coin. arthor flips the coin. Is the coin still heads up? Flip means reverse. (rephrased) The initial state of a coin is that it is face up with heads. Then, Aluino performs an action of flipping the coin, which implies reversing its current face from heads to tails. Following this, Arthor takes his turn and also flips the coin. Now, can it be determined whether the coin reverts to its original state of being heads up after these sequential flips? Remember, in this context, flipping pertains to reversing the side that the coin is currently showing. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T16.6.2" class="ltx_tr">
<td id="A2.T16.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T16.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T16.6.2.1.1.1" class="ltx_p"><span id="A2.T16.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T16.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T16.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T16.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: Yes.</span></span>
</span>
</td>
</tr>
<tr id="A2.T16.6.3" class="ltx_tr">
<td id="A2.T16.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T16.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T16.6.3.1.1.1" class="ltx_p"><span id="A2.T16.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T16.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) A coin is heads up. jumaimah flips the coin. buity flips the coin. Is the coin still heads up? Flip means reverse. (rephrased) The initial position of the coin is heads up. Jumaimah proceeds to perform an action on the coin known as a flip, which changes the coin from its current position to its opposite position. Following this, Buity also conducts the same action on the coin as Jumaimah did, which is a flip. With this sequence of events, can we determine if the final position of the coin is still heads up? Remember, the term ’flip’ is defined as reversing the current position of the coin. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T16.6.4" class="ltx_tr">
<td id="A2.T16.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T16.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T16.6.4.1.1.1" class="ltx_p"><span id="A2.T16.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T16.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T16.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T16.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: Yes.</span></span>
</span>
</td>
</tr>
<tr id="A2.T16.6.5" class="ltx_tr">
<td id="A2.T16.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T16.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T16.6.5.1.1.1" class="ltx_p"><span id="A2.T16.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T16.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) A coin is heads up. kinta does not flip the coin. renit does not flip the coin. Is the coin still heads up? Flip means reverse. (rephrased) The original question gives us a scenario wherein a coin is initially placed in a heads up position. It also states that two individuals, Kinta and Renit, have not interacted or manipulated the coin, specifically, they have not flipped it. To flip, as described, is to reverse the coin’s present state. Considering these points, is the coin’s position unchanged and is it still facing heads up? Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T16.6.6" class="ltx_tr">
<td id="A2.T16.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T16.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T16.6.6.1.1.1" class="ltx_p"><span id="A2.T16.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T16.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T16.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T16.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: Yes.</span></span>
</span>
</td>
</tr>
<tr id="A2.T16.6.7" class="ltx_tr">
<td id="A2.T16.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T16.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T16.6.7.1.1.1" class="ltx_p"><span id="A2.T16.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T16.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) A coin is heads up. brandarius does not flip the coin. gitta flips the coin. Is the coin still heads up? Flip means reverse. (rephrased) The current state of a coin is that it is facing heads up. An individual named brandarius does not alter this state by flipping the coin. However, another individual named gitta does change the state of the coin by flipping it. A flip in this context refers to reversing the side of the coin that is facing up. Hence, is it possible that following gitta’s action on the coin, it remains heads up? Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T16.6.8" class="ltx_tr">
<td id="A2.T16.6.8.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T16.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T16.6.8.1.1.1" class="ltx_p"><span id="A2.T16.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T16.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T16.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T16.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: No.</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T17" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 17: </span>Examples of correct and incorrect responses by GPT-4 to the original questions of <span id="A2.T17.5.1" class="ltx_text ltx_font_bold">Last letter concatenation (2)</span>.</figcaption>
<table id="A2.T17.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T17.6.1" class="ltx_tr">
<td id="A2.T17.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T17.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.1.1.1.1" class="ltx_p"><span id="A2.T17.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T17.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: Take the last letters of the words in “Beth Joel” and concatenate them.</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.2" class="ltx_tr">
<td id="A2.T17.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T17.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.2.1.1.1" class="ltx_p"><span id="A2.T17.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T17.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T17.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T17.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: “hel”</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.3" class="ltx_tr">
<td id="A2.T17.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T17.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.3.1.1.1" class="ltx_p"><span id="A2.T17.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T17.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: Take the last letters of the words in “Patrick Lorena” and concatenate them.</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.4" class="ltx_tr">
<td id="A2.T17.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T17.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.4.1.1.1" class="ltx_p"><span id="A2.T17.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T17.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T17.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T17.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: kkna</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.5" class="ltx_tr">
<td id="A2.T17.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T17.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.5.1.1.1" class="ltx_p"><span id="A2.T17.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T17.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: Take the last letters of the words in “Edgar Bob” and concatenate them.</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.6" class="ltx_tr">
<td id="A2.T17.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T17.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.6.1.1.1" class="ltx_p"><span id="A2.T17.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T17.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T17.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T17.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: “rb”</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.7" class="ltx_tr">
<td id="A2.T17.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T17.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.7.1.1.1" class="ltx_p"><span id="A2.T17.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T17.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: Take the last letters of the words in “Alexandra Betty” and concatenate them.</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.8" class="ltx_tr">
<td id="A2.T17.6.8.1" class="ltx_td ltx_align_justify">
<span id="A2.T17.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.8.1.1.1" class="ltx_p"><span id="A2.T17.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T17.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T17.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T17.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: ara</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.9" class="ltx_tr">
<td id="A2.T17.6.9.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T17.6.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.9.1.1.1" class="ltx_p"><span id="A2.T17.6.9.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T17.6.9.1.1.1.2" class="ltx_text" style="font-size:70%;">: Take the last letters of the words in “Paul Liz” and concatenate them.</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.10" class="ltx_tr">
<td id="A2.T17.6.10.1" class="ltx_td ltx_align_justify">
<span id="A2.T17.6.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.10.1.1.1" class="ltx_p"><span id="A2.T17.6.10.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T17.6.10.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T17.6.10.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T17.6.10.1.1.1.4" class="ltx_text" style="font-size:70%;">: uliz</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.11" class="ltx_tr">
<td id="A2.T17.6.11.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T17.6.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.11.1.1.1" class="ltx_p"><span id="A2.T17.6.11.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T17.6.11.1.1.1.2" class="ltx_text" style="font-size:70%;">: Take the last letters of the words in “Sheila Alberto” and concatenate them.</span></span>
</span>
</td>
</tr>
<tr id="A2.T17.6.12" class="ltx_tr">
<td id="A2.T17.6.12.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T17.6.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T17.6.12.1.1.1" class="ltx_p"><span id="A2.T17.6.12.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T17.6.12.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T17.6.12.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T17.6.12.1.1.1.4" class="ltx_text" style="font-size:70%;">: “ao”</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T18" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 18: </span>Examples of correct and incorrect responses by GPT-4 using 1-step RaR on <span id="A2.T18.5.1" class="ltx_text ltx_font_bold">Last letter concatenation (2)</span>.</figcaption>
<table id="A2.T18.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T18.6.1" class="ltx_tr">
<td id="A2.T18.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T18.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.1.1.1.1" class="ltx_p"><span id="A2.T18.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T18.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Take the last letters of the words in “Beth Joel” and concatenate them.” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.2" class="ltx_tr">
<td id="A2.T18.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T18.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.2.1.1.1" class="ltx_p"><span id="A2.T18.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T18.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T18.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T18.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: Can you merge the last letters from each of the words in the name “Beth Joel” together? What would the resultant combination look like? The last letters from the words “Beth” and “Joel” are “h” and “l” respectively. When we merge them together, the resultant combination is </span><span id="A2.T18.6.2.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">“hl”</span><span id="A2.T18.6.2.1.1.1.6" class="ltx_text" style="font-size:70%;">.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.3" class="ltx_tr">
<td id="A2.T18.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T18.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.3.1.1.1" class="ltx_p"><span id="A2.T18.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T18.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Take the last letters of the words in “Patrick Lorena” and concatenate them.” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.4" class="ltx_tr">
<td id="A2.T18.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T18.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.4.1.1.1" class="ltx_p"><span id="A2.T18.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T18.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T18.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T18.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: Could you please look at each word in the term “Patrick Lorena,” find the last letter of each word, and then join these letters together? Response: The last letters of “Patrick” and “Lorena” are “k” and “a”, respectively. When we concatenate them, we get </span><span id="A2.T18.6.4.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">“ka”</span><span id="A2.T18.6.4.1.1.1.6" class="ltx_text" style="font-size:70%;">.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.5" class="ltx_tr">
<td id="A2.T18.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T18.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.5.1.1.1" class="ltx_p"><span id="A2.T18.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T18.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Take the last letters of the words in “Edgar Bob” and concatenate them.” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.6" class="ltx_tr">
<td id="A2.T18.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T18.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.6.1.1.1" class="ltx_p"><span id="A2.T18.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T18.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T18.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T18.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: Could you please form a new string or series of characters by joining together the final letters from each word in the phrase “Edgar Bob”? The last letters in the words “Edgar” and “Bob” are “r” and “b”, hence when concatenated, it forms </span><span id="A2.T18.6.6.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">“rb”</span><span id="A2.T18.6.6.1.1.1.6" class="ltx_text" style="font-size:70%;">.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.7" class="ltx_tr">
<td id="A2.T18.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T18.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.7.1.1.1" class="ltx_p"><span id="A2.T18.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T18.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Take the last letters of the words in “Alexandra Betty” and concatenate them.” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.8" class="ltx_tr">
<td id="A2.T18.6.8.1" class="ltx_td ltx_align_justify">
<span id="A2.T18.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.8.1.1.1" class="ltx_p"><span id="A2.T18.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T18.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T18.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T18.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: Could you take the final letters that are from both words in “Alexandra Betty”, combine these letters, and present them together in one word? The response would be </span><span id="A2.T18.6.8.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">“ay”</span><span id="A2.T18.6.8.1.1.1.6" class="ltx_text" style="font-size:70%;">.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.9" class="ltx_tr">
<td id="A2.T18.6.9.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T18.6.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.9.1.1.1" class="ltx_p"><span id="A2.T18.6.9.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T18.6.9.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Take the last letters of the words in “Paul Liz” and concatenate them.” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.10" class="ltx_tr">
<td id="A2.T18.6.10.1" class="ltx_td ltx_align_justify">
<span id="A2.T18.6.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.10.1.1.1" class="ltx_p"><span id="A2.T18.6.10.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T18.6.10.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T18.6.10.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T18.6.10.1.1.1.4" class="ltx_text" style="font-size:70%;">: Can you take the final letters from each word in the phrase “Paul Liz” and put them together sequentially? Yes, by doing so, you get “ulz”.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.11" class="ltx_tr">
<td id="A2.T18.6.11.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T18.6.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.11.1.1.1" class="ltx_p"><span id="A2.T18.6.11.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T18.6.11.1.1.1.2" class="ltx_text" style="font-size:70%;">: “Take the last letters of the words in “Sheila Alberto” and concatenate them.” Rephrase and expand the question, and respond.</span></span>
</span>
</td>
</tr>
<tr id="A2.T18.6.12" class="ltx_tr">
<td id="A2.T18.6.12.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T18.6.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T18.6.12.1.1.1" class="ltx_p"><span id="A2.T18.6.12.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T18.6.12.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T18.6.12.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T18.6.12.1.1.1.4" class="ltx_text" style="font-size:70%;">: Can you identify the last letter from each word in the name “Sheila Alberto” and then join them together? In the name “Sheila Alberto,” the last letters of each word are “a” from Sheila and “o” from Alberto. If you concatenate or combine these letters, you get </span><span id="A2.T18.6.12.1.1.1.5" class="ltx_text" style="font-size:70%;color:#00FF00;">“ao”</span><span id="A2.T18.6.12.1.1.1.6" class="ltx_text" style="font-size:70%;">.</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T19" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 19: </span>Examples of correct and incorrect responses by GPT-4 using 2-step RaR on <span id="A2.T19.5.1" class="ltx_text ltx_font_bold">Last letter concatenation (2)</span>.</figcaption>
<table id="A2.T19.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T19.6.1" class="ltx_tr">
<td id="A2.T19.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T19.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.1.1.1.1" class="ltx_p"><span id="A2.T19.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T19.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Take the last letters of the words in “Beth Joel” and concatenate them. (rephrased) Can you extract the final letters from both “Beth” and “Joel”, and then combine them to form a new arrangement of letters, maintaining the original instruction to concatenate them? Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.2" class="ltx_tr">
<td id="A2.T19.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T19.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.2.1.1.1" class="ltx_p"><span id="A2.T19.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T19.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T19.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T19.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: The last letter of “Beth” is “h” and the last letter of “Joel” is “l”. Concatenate them, the result is “hl”.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.3" class="ltx_tr">
<td id="A2.T19.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T19.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.3.1.1.1" class="ltx_p"><span id="A2.T19.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T19.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Take the last letters of the words in “Patrick Lorena” and concatenate them. (rephrased) Identify the final letters in each of the words in ‘Patrick Lorena’. After you’ve identified those, combine them all together, in the order they appeared, to form a new word or string of letters. Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.4" class="ltx_tr">
<td id="A2.T19.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T19.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.4.1.1.1" class="ltx_p"><span id="A2.T19.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T19.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T19.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T19.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: The final letters of “Patrick” and “Lorena” are “k” and “a” respectively. When combined together, they form “ka”.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.5" class="ltx_tr">
<td id="A2.T19.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T19.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.5.1.1.1" class="ltx_p"><span id="A2.T19.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T19.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Take the last letters of the words in “Edgar Bob” and concatenate them.” Rephrase and expand the question, and respond. (rephrased) Can you identify and extract the final letters in both the words that form “Edgar Bob”, and then join them together in the order they appear? Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.6" class="ltx_tr">
<td id="A2.T19.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T19.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.6.1.1.1" class="ltx_p"><span id="A2.T19.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T19.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T19.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T19.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: The last letters in the words “Edgar Bob” are “r” and “b”. Concatenating them in the order they appear would be “rb”.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.7" class="ltx_tr">
<td id="A2.T19.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T19.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.7.1.1.1" class="ltx_p"><span id="A2.T19.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T19.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Take the last letters of the words in “Alexandra Betty” and concatenate them. (rephrased) To answer the question, you would need to look at the words “Alexandra” and “Betty”. Identify the last letter in each of these words. Once you have these last letters, join them together in the order they appear in the original words. The question is asking you to perform these actions. Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.8" class="ltx_tr">
<td id="A2.T19.6.8.1" class="ltx_td ltx_align_justify">
<span id="A2.T19.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.8.1.1.1" class="ltx_p"><span id="A2.T19.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T19.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T19.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T19.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: The last letter of “Alexandra” is “a” and the last letter of “Betty” is “y”. When these are concatenated, or joined together, the result is “ay”.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.9" class="ltx_tr">
<td id="A2.T19.6.9.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T19.6.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.9.1.1.1" class="ltx_p"><span id="A2.T19.6.9.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T19.6.9.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Take the last letters of the words in “Paul Liz” and concatenate them. (rephrased) Identify the last letters of each word in the phrase “Paul Liz”. After identifying, combine these letters together to form a new word or string. This process is known as concatenation. In other words, you will merge the last characters from ‘Paul’ and ‘Liz’ without any spaces between them. Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.10" class="ltx_tr">
<td id="A2.T19.6.10.1" class="ltx_td ltx_align_justify">
<span id="A2.T19.6.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.10.1.1.1" class="ltx_p"><span id="A2.T19.6.10.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T19.6.10.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T19.6.10.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T19.6.10.1.1.1.4" class="ltx_text" style="font-size:70%;">: The last letters of “Paul” and “Liz” are “l” and “z” respectively. Concatenating these letters would form “lz”.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.11" class="ltx_tr">
<td id="A2.T19.6.11.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T19.6.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.11.1.1.1" class="ltx_p"><span id="A2.T19.6.11.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T19.6.11.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Take the last letters of the words in “Sheila Alberto” and concatenate them. (rephrased) Identify the final letters in each of the words ‘Sheila’ and ‘Alberto’. After recognizing these, join or link them together in the same order they appear. This process is referred to as concatenation. Use your answer for the rephrased question to answer the original question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T19.6.12" class="ltx_tr">
<td id="A2.T19.6.12.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T19.6.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T19.6.12.1.1.1" class="ltx_p"><span id="A2.T19.6.12.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T19.6.12.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T19.6.12.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T19.6.12.1.1.1.4" class="ltx_text" style="font-size:70%;">: The last letter of “Sheila” is “a” and the last letter of “Alberto” is “o”. So, when concatenated, the resulting output is “ao”.</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T20" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 20: </span>Examples of correct and incorrect responses by GPT-4 to the original questions of <span id="A2.T20.5.1" class="ltx_text ltx_font_bold">Sports</span>.</figcaption>
<table id="A2.T20.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T20.6.1" class="ltx_tr">
<td id="A2.T20.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T20.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T20.6.1.1.1.1" class="ltx_p"><span id="A2.T20.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T20.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: Is the following sentence plausible? “Javier Zanetti earned a direct kick” Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T20.6.2" class="ltx_tr">
<td id="A2.T20.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T20.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T20.6.2.1.1.1" class="ltx_p"><span id="A2.T20.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T20.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T20.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T20.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: No</span></span>
</span>
</td>
</tr>
<tr id="A2.T20.6.3" class="ltx_tr">
<td id="A2.T20.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T20.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T20.6.3.1.1.1" class="ltx_p"><span id="A2.T20.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T20.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: Is the following sentence plausible? “Jayson Tatum nutmegged the defender” Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T20.6.4" class="ltx_tr">
<td id="A2.T20.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T20.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T20.6.4.1.1.1" class="ltx_p"><span id="A2.T20.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T20.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T20.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T20.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: Yes</span></span>
</span>
</td>
</tr>
<tr id="A2.T20.6.5" class="ltx_tr">
<td id="A2.T20.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T20.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T20.6.5.1.1.1" class="ltx_p"><span id="A2.T20.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T20.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: Is the following sentence plausible? “Sean Couturier passed the puck in the Stanley Cup” Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T20.6.6" class="ltx_tr">
<td id="A2.T20.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T20.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T20.6.6.1.1.1" class="ltx_p"><span id="A2.T20.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T20.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T20.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T20.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: Yes</span></span>
</span>
</td>
</tr>
<tr id="A2.T20.6.7" class="ltx_tr">
<td id="A2.T20.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T20.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T20.6.7.1.1.1" class="ltx_p"><span id="A2.T20.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T20.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: Is the following sentence plausible? “Tuukka Rask hit a double” Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T20.6.8" class="ltx_tr">
<td id="A2.T20.6.8.1" class="ltx_td ltx_align_justify">
<span id="A2.T20.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T20.6.8.1.1.1" class="ltx_p"><span id="A2.T20.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T20.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T20.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T20.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: No</span></span>
</span>
</td>
</tr>
<tr id="A2.T20.6.9" class="ltx_tr">
<td id="A2.T20.6.9.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T20.6.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T20.6.9.1.1.1" class="ltx_p"><span id="A2.T20.6.9.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T20.6.9.1.1.1.2" class="ltx_text" style="font-size:70%;">: Is the following sentence plausible? “Sam Darnold struck out” Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T20.6.10" class="ltx_tr">
<td id="A2.T20.6.10.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T20.6.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T20.6.10.1.1.1" class="ltx_p"><span id="A2.T20.6.10.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T20.6.10.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T20.6.10.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T20.6.10.1.1.1.4" class="ltx_text" style="font-size:70%;">: Yes</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T21" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 21: </span>Examples of correct and incorrect responses by GPT-4 using One-step RaR on <span id="A2.T21.5.1" class="ltx_text ltx_font_bold">Sports</span>.</figcaption>
<table id="A2.T21.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T21.6.1" class="ltx_tr">
<td id="A2.T21.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T21.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T21.6.1.1.1.1" class="ltx_p"><span id="A2.T21.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T21.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: ‘Is the following sentence plausible? “Javier Zanetti earned a direct kick”’ Rephrase and expand the question, and respond. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T21.6.2" class="ltx_tr">
<td id="A2.T21.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T21.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T21.6.2.1.1.1" class="ltx_p"><span id="A2.T21.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T21.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T21.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T21.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: “Is it plausible or possible that Javier Zanetti received a direct kick during a soccer game?” Yes, it is plausible.</span></span>
</span>
</td>
</tr>
<tr id="A2.T21.6.3" class="ltx_tr">
<td id="A2.T21.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T21.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T21.6.3.1.1.1" class="ltx_p"><span id="A2.T21.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T21.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: ‘Is the following sentence plausible? “Jayson Tatum nutmegged the defender”’ Rephrase and expand the question, and respond. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T21.6.4" class="ltx_tr">
<td id="A2.T21.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T21.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T21.6.4.1.1.1" class="ltx_p"><span id="A2.T21.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T21.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T21.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T21.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: Could it be possible that Jayson Tatum managed to play a nutmeg move on a defender during a basketball game? Yes, it’s plausible as nutmegging is a commonly used trick in basketball and Jayson Tatum is known for his skills and tricks in the game.</span></span>
</span>
</td>
</tr>
<tr id="A2.T21.6.5" class="ltx_tr">
<td id="A2.T21.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T21.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T21.6.5.1.1.1" class="ltx_p"><span id="A2.T21.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T21.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: ‘Is the following sentence plausible? “Sean Couturier passed the puck in the Stanley Cup”’ Rephrase and expand the question, and respond. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T21.6.6" class="ltx_tr">
<td id="A2.T21.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T21.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T21.6.6.1.1.1" class="ltx_p"><span id="A2.T21.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T21.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T21.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T21.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: Is it possible for the statement “Sean Couturier maradona’d the defender” to make sense or be meaningful? In context of sports language, particularly soccer, the term “maradona’d” could be used to describe a soccer player performing a maneuver similar to that of Diego Maradona, a renowned soccer player. Sean Couturier is a professional ice hockey player, and although the sports are different, it is plausible that he could carry out a maneuver in hockey analogous to one performed by Maradona in soccer, assuming “maradona’d” means he skillfully bypassed or tricked the defender in some way. So, yes, the sentence could be plausible.</span></span>
</span>
</td>
</tr>
<tr id="A2.T21.6.7" class="ltx_tr">
<td id="A2.T21.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T21.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T21.6.7.1.1.1" class="ltx_p"><span id="A2.T21.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T21.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: ‘Is the following sentence plausible? “Tuukka Rask hit a double”’ Rephrase and expand the question, and respond. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T21.6.8" class="ltx_tr">
<td id="A2.T21.6.8.1" class="ltx_td ltx_align_justify">
<span id="A2.T21.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T21.6.8.1.1.1" class="ltx_p"><span id="A2.T21.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T21.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T21.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T21.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: Is it possible that Tuukka Rask, a professional ice hockey player, could have hit a double in a game? No, Tuukka Rask is a professional ice hockey goaltender for the Boston Bruins. He would not be involved in a sport where he would hit a double, such as baseball. The phrase “hitting a double” isn’t applicable to ice hockey, making the sentence implausible.</span></span>
</span>
</td>
</tr>
<tr id="A2.T21.6.9" class="ltx_tr">
<td id="A2.T21.6.9.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T21.6.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T21.6.9.1.1.1" class="ltx_p"><span id="A2.T21.6.9.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T21.6.9.1.1.1.2" class="ltx_text" style="font-size:70%;">: ‘Is the following sentence plausible? “Sam Darnold struck out”’ Rephrase and expand the question, and respond. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T21.6.10" class="ltx_tr">
<td id="A2.T21.6.10.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T21.6.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T21.6.10.1.1.1" class="ltx_p"><span id="A2.T21.6.10.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T21.6.10.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T21.6.10.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T21.6.10.1.1.1.4" class="ltx_text" style="font-size:70%;">: Is it reasonable or believable to say “Sam Darnold struck out”? No, it’s not plausible because Sam Darnold is a known professional football player, and the term “struck out” is used in baseball, not football.</span></span>
</span>
</td>
</tr>
</table>
</figure>
<figure id="A2.T22" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 22: </span>Examples of correct and incorrect responses by GPT-4 using Two-step RaR on <span id="A2.T22.5.1" class="ltx_text ltx_font_bold">Sports</span>.</figcaption>
<table id="A2.T22.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T22.6.1" class="ltx_tr">
<td id="A2.T22.6.1.1" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="A2.T22.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T22.6.1.1.1.1" class="ltx_p"><span id="A2.T22.6.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T22.6.1.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Is the following sentence plausible? “Javier Zanetti earned a direct kick” (rephrased) Is it believable or possible that Javier Zanetti secured a direct free kick in a match? Use your answer for the rephrased question to answer the original question. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T22.6.2" class="ltx_tr">
<td id="A2.T22.6.2.1" class="ltx_td ltx_align_justify">
<span id="A2.T22.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T22.6.2.1.1.1" class="ltx_p"><span id="A2.T22.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T22.6.2.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T22.6.2.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T22.6.2.1.1.1.4" class="ltx_text" style="font-size:70%;">: Yes</span></span>
</span>
</td>
</tr>
<tr id="A2.T22.6.3" class="ltx_tr">
<td id="A2.T22.6.3.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T22.6.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T22.6.3.1.1.1" class="ltx_p"><span id="A2.T22.6.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T22.6.3.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Is the following sentence plausible? “Jayson Tatum nutmegged the defender” (rephrased) Is it believable that Jayson Tatum, a professional basketball player, could have performed a move where he passed the ball through a defender’s legs, also known as a ‘nutmeg’? Use your answer for the rephrased question to answer the original question. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T22.6.4" class="ltx_tr">
<td id="A2.T22.6.4.1" class="ltx_td ltx_align_justify">
<span id="A2.T22.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T22.6.4.1.1.1" class="ltx_p"><span id="A2.T22.6.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T22.6.4.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T22.6.4.1.1.1.3" class="ltx_text" style="font-size:70%;color:#0000FF;">(Incorrect)</span><span id="A2.T22.6.4.1.1.1.4" class="ltx_text" style="font-size:70%;">: Yes</span></span>
</span>
</td>
</tr>
<tr id="A2.T22.6.5" class="ltx_tr">
<td id="A2.T22.6.5.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T22.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T22.6.5.1.1.1" class="ltx_p"><span id="A2.T22.6.5.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T22.6.5.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Is the following sentence plausible? “Sean Couturier passed the puck in the Stanley Cup” (rephrased) Could it be possible that Sean Couturier passed the puck during a game in the Stanley Cup tournament? Use your answer for the rephrased question to answer the original question. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T22.6.6" class="ltx_tr">
<td id="A2.T22.6.6.1" class="ltx_td ltx_align_justify">
<span id="A2.T22.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T22.6.6.1.1.1" class="ltx_p"><span id="A2.T22.6.6.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T22.6.6.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T22.6.6.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T22.6.6.1.1.1.4" class="ltx_text" style="font-size:70%;">: Yes</span></span>
</span>
</td>
</tr>
<tr id="A2.T22.6.7" class="ltx_tr">
<td id="A2.T22.6.7.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T22.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T22.6.7.1.1.1" class="ltx_p"><span id="A2.T22.6.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T22.6.7.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Is the following sentence plausible? “Tuukka Rask hit a double” (rephrased) Is it feasible that Tuukka Rask, presumably known for a non-baseball sport, hit a double in a baseball game? Use your answer for the rephrased question to answer the original question. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T22.6.8" class="ltx_tr">
<td id="A2.T22.6.8.1" class="ltx_td ltx_align_justify">
<span id="A2.T22.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T22.6.8.1.1.1" class="ltx_p"><span id="A2.T22.6.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T22.6.8.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T22.6.8.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T22.6.8.1.1.1.4" class="ltx_text" style="font-size:70%;">: No</span></span>
</span>
</td>
</tr>
<tr id="A2.T22.6.9" class="ltx_tr">
<td id="A2.T22.6.9.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A2.T22.6.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T22.6.9.1.1.1" class="ltx_p"><span id="A2.T22.6.9.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Question</span><span id="A2.T22.6.9.1.1.1.2" class="ltx_text" style="font-size:70%;">: (original) Is the following sentence plausible? “Sam Darnold struck out” (rephrased) Is it believable that Sam Darnold has struck out in a game? Please provide a detailed explanation supporting your answer without omitting any information from the initial question. Use your answer for the rephrased question to answer the original question. Answer the Yes or No question.</span></span>
</span>
</td>
</tr>
<tr id="A2.T22.6.10" class="ltx_tr">
<td id="A2.T22.6.10.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A2.T22.6.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T22.6.10.1.1.1" class="ltx_p"><span id="A2.T22.6.10.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">GPT-4</span><span id="A2.T22.6.10.1.1.1.2" class="ltx_text" style="font-size:70%;"> </span><span id="A2.T22.6.10.1.1.1.3" class="ltx_text" style="font-size:70%;color:#00FF00;">(Correct)</span><span id="A2.T22.6.10.1.1.1.4" class="ltx_text" style="font-size:70%;">: No</span></span>
</span>
</td>
</tr>
</table>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography" style="font-size:70%;">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib1.4.4.1" class="ltx_text" style="font-size:70%;">Allen-Zhu and Li (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib1.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib1.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Allen-Zhu, Z.</span><span id="bib.bib1.8.3" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib1.9.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Li, Y.</span><span id="bib.bib1.10.5" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.11.1" class="ltx_text" style="font-size:70%;">Physics of language models: Part 3.2, knowledge manipulation.
</span>
</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib2.4.4.1" class="ltx_text" style="font-size:70%;">Ateia and Kruschwitz (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib2.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib2.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Ateia, S.</span><span id="bib.bib2.8.3" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib2.9.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Kruschwitz, U.</span><span id="bib.bib2.10.5" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib2.11.1" class="ltx_text" style="font-size:70%;">Is chatgpt a biomedical expert?–exploring the zero-shot performance of current gpt models in biomedical tasks.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib2.12.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2306.16108</span><span id="bib.bib2.13.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib3.4.4.1" class="ltx_text" style="font-size:70%;">bench authors (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib3.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib3.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">bench authors, B.</span><span id="bib.bib3.8.3" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.9.1" class="ltx_text" style="font-size:70%;">Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.10.1" class="ltx_text ltx_font_italic" style="font-size:70%;">Transactions on Machine Learning Research</span><span id="bib.bib3.11.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib4.4.4.1" class="ltx_text" style="font-size:70%;">Bozkurt (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib4.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib4.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bozkurt, A.</span><span id="bib.bib4.8.3" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.9.1" class="ltx_text" style="font-size:70%;">Generative artificial intelligence (ai) powered conversational educational agents: The inevitable paradigm shift.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.10.1" class="ltx_text ltx_font_italic" style="font-size:70%;">Asian Journal of Distance Education</span><span id="bib.bib4.11.2" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib4.12.3" class="ltx_text ltx_font_bold" style="font-size:70%;">18</span><span id="bib.bib4.13.4" class="ltx_text" style="font-size:70%;">.
</span>
</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib5.5.5.1" class="ltx_text" style="font-size:70%;">Brown et al. (2020)</span></span>
<span class="ltx_bibblock"><span id="bib.bib5.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib5.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Brown, T.</span><span id="bib.bib5.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib5.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Mann, B.</span><span id="bib.bib5.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib5.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Ryder, N.</span><span id="bib.bib5.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib5.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Subbiah, M.</span><span id="bib.bib5.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib5.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Kaplan, J. D.</span><span id="bib.bib5.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib5.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Dhariwal, P.</span><span id="bib.bib5.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib5.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Neelakantan, A.</span><span id="bib.bib5.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib5.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Shyam, P.</span><span id="bib.bib5.23.17" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib5.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Sastry, G.</span><span id="bib.bib5.25.19" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib5.26.20" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Askell, A.</span><span id="bib.bib5.27.21" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib5.28.22" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">et al.</span><span id="bib.bib5.29.23" class="ltx_text" style="font-size:70%;"> (2020).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib5.30.1" class="ltx_text" style="font-size:70%;">Language models are few-shot learners.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib5.31.1" class="ltx_text ltx_font_italic" style="font-size:70%;">Advances in neural information processing systems</span><span id="bib.bib5.32.2" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib5.33.3" class="ltx_text ltx_font_bold" style="font-size:70%;">33</span><span id="bib.bib5.34.4" class="ltx_text" style="font-size:70%;"> 1877–1901.
</span>
</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib6.5.5.1" class="ltx_text" style="font-size:70%;">Chiang et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib6.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib6.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Chiang, W.-L.</span><span id="bib.bib6.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib6.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Li, Z.</span><span id="bib.bib6.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib6.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lin, Z.</span><span id="bib.bib6.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib6.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Sheng, Y.</span><span id="bib.bib6.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib6.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wu, Z.</span><span id="bib.bib6.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib6.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhang, H.</span><span id="bib.bib6.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib6.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zheng, L.</span><span id="bib.bib6.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib6.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhuang, S.</span><span id="bib.bib6.23.17" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib6.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhuang, Y.</span><span id="bib.bib6.25.19" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib6.26.20" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gonzalez, J. E.</span><span id="bib.bib6.27.21" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib6.28.22" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Stoica, I.</span><span id="bib.bib6.29.23" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib6.30.24" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Xing, E. P.</span><span id="bib.bib6.31.25" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib6.32.1" class="ltx_text" style="font-size:70%;">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
</span>
</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib7.4.4.1" class="ltx_text" style="font-size:70%;">Druckman (2001)</span></span>
<span class="ltx_bibblock"><span id="bib.bib7.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib7.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Druckman, J. N.</span><span id="bib.bib7.8.3" class="ltx_text" style="font-size:70%;"> (2001).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.9.1" class="ltx_text" style="font-size:70%;">The implications of framing effects for citizen competence.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.10.1" class="ltx_text ltx_font_italic" style="font-size:70%;">Political behavior</span><span id="bib.bib7.11.2" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib7.12.3" class="ltx_text ltx_font_bold" style="font-size:70%;">23</span><span id="bib.bib7.13.4" class="ltx_text" style="font-size:70%;"> 225–256.
</span>
</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib8.4.4.1" class="ltx_text" style="font-size:70%;">Erving (1974)</span></span>
<span class="ltx_bibblock"><span id="bib.bib8.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib8.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Erving, G.</span><span id="bib.bib8.8.3" class="ltx_text" style="font-size:70%;"> (1974).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib8.9.1" class="ltx_text" style="font-size:70%;">Frame analysis: An essay on the organization of experience.
</span>
</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib9.4.4.1" class="ltx_text" style="font-size:70%;">Fortes (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib9.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib9.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Fortes, A.</span><span id="bib.bib9.8.3" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib9.9.1" class="ltx_text" style="font-size:70%;">simple-dataset-generation.
</span>
</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib10.5.5.1" class="ltx_text" style="font-size:70%;">Gao et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib10.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib10.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gao, L.</span><span id="bib.bib10.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib10.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Dai, Z.</span><span id="bib.bib10.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib10.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Pasupat, P.</span><span id="bib.bib10.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib10.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Chen, A.</span><span id="bib.bib10.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib10.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Chaganty, A. T.</span><span id="bib.bib10.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib10.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Fan, Y.</span><span id="bib.bib10.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib10.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhao, V.</span><span id="bib.bib10.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib10.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lao, N.</span><span id="bib.bib10.23.17" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib10.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lee, H.</span><span id="bib.bib10.25.19" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib10.26.20" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Juan, D.-C.</span><span id="bib.bib10.27.21" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib10.28.22" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">et al.</span><span id="bib.bib10.29.23" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.30.1" class="ltx_text" style="font-size:70%;">Rarr: Researching and revising what language models say, using language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.31.1" class="ltx_text" style="font-size:70%;">In </span><span id="bib.bib10.32.2" class="ltx_text ltx_font_italic" style="font-size:70%;">Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</span><span id="bib.bib10.33.3" class="ltx_text" style="font-size:70%;">.
</span>
</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib11.5.5.1" class="ltx_text" style="font-size:70%;">Huang et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib11.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib11.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Huang, J.</span><span id="bib.bib11.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib11.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Chen, X.</span><span id="bib.bib11.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib11.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Mishra, S.</span><span id="bib.bib11.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib11.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zheng, H. S.</span><span id="bib.bib11.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib11.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yu, A. W.</span><span id="bib.bib11.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib11.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Song, X.</span><span id="bib.bib11.19.13" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib11.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhou, D.</span><span id="bib.bib11.21.15" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib11.22.1" class="ltx_text" style="font-size:70%;">Large language models cannot self-correct reasoning yet.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib11.23.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2310.01798</span><span id="bib.bib11.24.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib12.5.5.1" class="ltx_text" style="font-size:70%;">Jung et al. (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib12.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib12.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Jung, J.</span><span id="bib.bib12.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib12.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Qin, L.</span><span id="bib.bib12.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib12.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Welleck, S.</span><span id="bib.bib12.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib12.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Brahman, F.</span><span id="bib.bib12.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib12.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bhagavatula, C.</span><span id="bib.bib12.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib12.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Le Bras, R.</span><span id="bib.bib12.19.13" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib12.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Choi, Y.</span><span id="bib.bib12.21.15" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.22.1" class="ltx_text" style="font-size:70%;">Maieutic prompting: Logically consistent reasoning with recursive explanations.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.23.1" class="ltx_text" style="font-size:70%;">In </span><span id="bib.bib12.24.2" class="ltx_text ltx_font_italic" style="font-size:70%;">Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</span><span id="bib.bib12.25.3" class="ltx_text" style="font-size:70%;">. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates.
</span>
</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib13.5.5.1" class="ltx_text" style="font-size:70%;">Kim et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib13.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib13.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Kim, G.</span><span id="bib.bib13.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib13.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Baldi, P.</span><span id="bib.bib13.11.5" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib13.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">McAleer, S.</span><span id="bib.bib13.13.7" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib13.14.1" class="ltx_text" style="font-size:70%;">Language models can solve computer tasks.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib13.15.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2303.17491</span><span id="bib.bib13.16.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib14.5.5.1" class="ltx_text" style="font-size:70%;">Kojima et al. (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib14.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib14.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Kojima, T.</span><span id="bib.bib14.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib14.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gu, S. S.</span><span id="bib.bib14.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib14.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Reid, M.</span><span id="bib.bib14.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib14.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Matsuo, Y.</span><span id="bib.bib14.15.9" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib14.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Iwasawa, Y.</span><span id="bib.bib14.17.11" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib14.18.1" class="ltx_text" style="font-size:70%;">Large language models are zero-shot reasoners.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib14.19.1" class="ltx_text ltx_font_italic" style="font-size:70%;">Advances in neural information processing systems</span><span id="bib.bib14.20.2" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib14.21.3" class="ltx_text ltx_font_bold" style="font-size:70%;">35</span><span id="bib.bib14.22.4" class="ltx_text" style="font-size:70%;"> 22199–22213.
</span>
</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib15.5.5.1" class="ltx_text" style="font-size:70%;">Lightman et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib15.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib15.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lightman, H.</span><span id="bib.bib15.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib15.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Kosaraju, V.</span><span id="bib.bib15.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib15.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Burda, Y.</span><span id="bib.bib15.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib15.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Edwards, H.</span><span id="bib.bib15.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib15.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Baker, B.</span><span id="bib.bib15.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib15.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lee, T.</span><span id="bib.bib15.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib15.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Leike, J.</span><span id="bib.bib15.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib15.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Schulman, J.</span><span id="bib.bib15.23.17" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib15.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Sutskever, I.</span><span id="bib.bib15.25.19" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib15.26.20" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Cobbe, K.</span><span id="bib.bib15.27.21" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.28.1" class="ltx_text" style="font-size:70%;">Let’s verify step by step.
</span>
</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib16.5.5.1" class="ltx_text" style="font-size:70%;">Liu et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib16.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib16.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Liu, A.</span><span id="bib.bib16.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib16.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wu, Z.</span><span id="bib.bib16.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib16.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Michael, J.</span><span id="bib.bib16.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib16.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Suhr, A.</span><span id="bib.bib16.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib16.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">West, P.</span><span id="bib.bib16.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib16.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Koller, A.</span><span id="bib.bib16.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib16.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Swayamdipta, S.</span><span id="bib.bib16.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib16.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Smith, N. A.</span><span id="bib.bib16.23.17" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib16.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Choi, Y.</span><span id="bib.bib16.25.19" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.26.1" class="ltx_text" style="font-size:70%;">We’re afraid language models aren’t modeling ambiguity.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.27.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2304.14399</span><span id="bib.bib16.28.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib17.5.5.1" class="ltx_text" style="font-size:70%;">Lu et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib17.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib17.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lu, P.</span><span id="bib.bib17.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib17.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bansal, H.</span><span id="bib.bib17.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib17.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Xia, T.</span><span id="bib.bib17.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib17.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Liu, J.</span><span id="bib.bib17.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib17.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Li, C.</span><span id="bib.bib17.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib17.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Hajishirzi, H.</span><span id="bib.bib17.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib17.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Cheng, H.</span><span id="bib.bib17.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib17.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Chang, K.-W.</span><span id="bib.bib17.23.17" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib17.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Galley, M.</span><span id="bib.bib17.25.19" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib17.26.20" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gao, J.</span><span id="bib.bib17.27.21" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.28.1" class="ltx_text" style="font-size:70%;">Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.
</span>
</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib18.5.5.1" class="ltx_text" style="font-size:70%;">Madaan et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib18.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib18.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Madaan, A.</span><span id="bib.bib18.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib18.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Tandon, N.</span><span id="bib.bib18.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib18.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gupta, P.</span><span id="bib.bib18.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib18.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Hallinan, S.</span><span id="bib.bib18.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib18.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gao, L.</span><span id="bib.bib18.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib18.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wiegreffe, S.</span><span id="bib.bib18.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib18.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Alon, U.</span><span id="bib.bib18.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib18.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Dziri, N.</span><span id="bib.bib18.23.17" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib18.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Prabhumoye, S.</span><span id="bib.bib18.25.19" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib18.26.20" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yang, Y.</span><span id="bib.bib18.27.21" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib18.28.22" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">et al.</span><span id="bib.bib18.29.23" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib18.30.1" class="ltx_text" style="font-size:70%;">Self-refine: Iterative refinement with self-feedback.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib18.31.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2303.17651</span><span id="bib.bib18.32.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib19.5.5.1" class="ltx_text" style="font-size:70%;">Mukherjee et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib19.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib19.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Mukherjee, S.</span><span id="bib.bib19.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib19.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Mitra, A.</span><span id="bib.bib19.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib19.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Jawahar, G.</span><span id="bib.bib19.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib19.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Agarwal, S.</span><span id="bib.bib19.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib19.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Palangi, H.</span><span id="bib.bib19.17.11" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib19.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Awadallah, A.</span><span id="bib.bib19.19.13" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib19.20.1" class="ltx_text" style="font-size:70%;">Orca: Progressive learning from complex explanation traces of gpt-4.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib19.21.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2306.02707</span><span id="bib.bib19.22.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib20.5.5.1" class="ltx_text" style="font-size:70%;">Nadeem et al. (2021)</span></span>
<span class="ltx_bibblock"><span id="bib.bib20.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib20.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Nadeem, M.</span><span id="bib.bib20.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib20.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bethke, A.</span><span id="bib.bib20.11.5" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib20.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Reddy, S.</span><span id="bib.bib20.13.7" class="ltx_text" style="font-size:70%;"> (2021).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib20.14.1" class="ltx_text" style="font-size:70%;">Stereoset: Measuring stereotypical bias in pretrained language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib20.15.1" class="ltx_text" style="font-size:70%;">In </span><span id="bib.bib20.16.2" class="ltx_text ltx_font_italic" style="font-size:70%;">Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)</span><span id="bib.bib20.17.3" class="ltx_text" style="font-size:70%;">.
</span>
</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib21.4.4.1" class="ltx_text" style="font-size:70%;">OpenAI (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib21.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib21.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">OpenAI</span><span id="bib.bib21.8.3" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib21.9.1" class="ltx_text" style="font-size:70%;">Best practices for prompt engineering with openai api.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib21.10.1" class="ltx_text ltx_font_italic" style="font-size:70%;">https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api</span><span id="bib.bib21.11.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib22.4.4.1" class="ltx_text" style="font-size:70%;">OpenAI (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib22.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib22.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">OpenAI</span><span id="bib.bib22.8.3" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib22.9.1" class="ltx_text" style="font-size:70%;">Gpt-4 technical report.
</span>
</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib23.5.5.1" class="ltx_text" style="font-size:70%;">Pan et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib23.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib23.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Pan, L.</span><span id="bib.bib23.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib23.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Saxon, M.</span><span id="bib.bib23.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib23.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Xu, W.</span><span id="bib.bib23.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib23.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Nathani, D.</span><span id="bib.bib23.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib23.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wang, X.</span><span id="bib.bib23.17.11" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib23.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wang, W. Y.</span><span id="bib.bib23.19.13" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib23.20.1" class="ltx_text" style="font-size:70%;">Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib23.21.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2308.03188</span><span id="bib.bib23.22.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib24.5.5.1" class="ltx_text" style="font-size:70%;">Pawelczyk et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib24.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib24.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Pawelczyk, M.</span><span id="bib.bib24.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib24.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Neel, S.</span><span id="bib.bib24.11.5" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib24.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lakkaraju, H.</span><span id="bib.bib24.13.7" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib24.14.1" class="ltx_text" style="font-size:70%;">In-context unlearning: Language models as few shot unlearners.
</span>
</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib25.5.5.1" class="ltx_text" style="font-size:70%;">Poesia et al. (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib25.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib25.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Poesia, G.</span><span id="bib.bib25.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib25.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Polozov, O.</span><span id="bib.bib25.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib25.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Le, V.</span><span id="bib.bib25.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib25.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Tiwari, A.</span><span id="bib.bib25.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib25.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Soares, G.</span><span id="bib.bib25.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib25.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Meek, C.</span><span id="bib.bib25.19.13" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib25.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gulwani, S.</span><span id="bib.bib25.21.15" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib25.22.1" class="ltx_text" style="font-size:70%;">Synchromesh: Reliable code generation from pre-trained language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib25.23.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2201.11227</span><span id="bib.bib25.24.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib26.5.5.1" class="ltx_text" style="font-size:70%;">Press et al. (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib26.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib26.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Press, O.</span><span id="bib.bib26.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib26.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhang, M.</span><span id="bib.bib26.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib26.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Min, S.</span><span id="bib.bib26.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib26.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Schmidt, L.</span><span id="bib.bib26.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib26.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Smith, N. A.</span><span id="bib.bib26.17.11" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib26.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lewis, M.</span><span id="bib.bib26.19.13" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.20.1" class="ltx_text" style="font-size:70%;">Measuring and narrowing the compositionality gap in language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.21.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2210.03350</span><span id="bib.bib26.22.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib27.5.5.1" class="ltx_text" style="font-size:70%;">Pryzant et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib27.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib27.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Pryzant, R.</span><span id="bib.bib27.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib27.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Iter, D.</span><span id="bib.bib27.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib27.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Li, J.</span><span id="bib.bib27.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib27.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lee, Y. T.</span><span id="bib.bib27.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib27.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhu, C.</span><span id="bib.bib27.17.11" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib27.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zeng, M.</span><span id="bib.bib27.19.13" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.20.1" class="ltx_text" style="font-size:70%;">Automatic prompt optimization with” gradient descent” and beam search.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.21.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2305.03495</span><span id="bib.bib27.22.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib28.5.5.1" class="ltx_text" style="font-size:70%;">Radford et al. (2019)</span></span>
<span class="ltx_bibblock"><span id="bib.bib28.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib28.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Radford, A.</span><span id="bib.bib28.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib28.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wu, J.</span><span id="bib.bib28.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib28.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Child, R.</span><span id="bib.bib28.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib28.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Luan, D.</span><span id="bib.bib28.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib28.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Amodei, D.</span><span id="bib.bib28.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib28.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Sutskever, I.</span><span id="bib.bib28.19.13" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib28.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">et al.</span><span id="bib.bib28.21.15" class="ltx_text" style="font-size:70%;"> (2019).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib28.22.1" class="ltx_text" style="font-size:70%;">Language models are unsupervised multitask learners.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib28.23.1" class="ltx_text ltx_font_italic" style="font-size:70%;">OpenAI blog</span><span id="bib.bib28.24.2" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib28.25.3" class="ltx_text ltx_font_bold" style="font-size:70%;">1</span><span id="bib.bib28.26.4" class="ltx_text" style="font-size:70%;"> 9.
</span>
</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib29.4.4.1" class="ltx_text" style="font-size:70%;">Reynolds and McDonell (2021)</span></span>
<span class="ltx_bibblock"><span id="bib.bib29.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib29.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Reynolds, L.</span><span id="bib.bib29.8.3" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib29.9.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">McDonell, K.</span><span id="bib.bib29.10.5" class="ltx_text" style="font-size:70%;"> (2021).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib29.11.1" class="ltx_text" style="font-size:70%;">Prompt programming for large language models: Beyond the few-shot paradigm.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib29.12.1" class="ltx_text" style="font-size:70%;">In </span><span id="bib.bib29.13.2" class="ltx_text ltx_font_italic" style="font-size:70%;">Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems</span><span id="bib.bib29.14.3" class="ltx_text" style="font-size:70%;">. CHI EA ’21, Association for Computing Machinery, New York, NY, USA.
</span>
</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib30.4.4.1" class="ltx_text" style="font-size:70%;">Saravia (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib30.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib30.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Saravia, E.</span><span id="bib.bib30.8.3" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib30.9.1" class="ltx_text" style="font-size:70%;">Prompt Engineering Guide.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib30.10.1" class="ltx_text ltx_font_italic" style="font-size:70%;">https://github.com/dair-ai/Prompt-Engineering-Guide</span><span id="bib.bib30.11.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib31.4.4.1" class="ltx_text" style="font-size:70%;">Schick and Schütze (2021)</span></span>
<span class="ltx_bibblock"><span id="bib.bib31.6.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib31.7.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Schick, T.</span><span id="bib.bib31.8.3" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib31.9.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Schütze, H.</span><span id="bib.bib31.10.5" class="ltx_text" style="font-size:70%;"> (2021).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib31.11.1" class="ltx_text" style="font-size:70%;">Exploiting cloze-questions for few-shot text classification and natural language inference.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib31.12.1" class="ltx_text" style="font-size:70%;">In </span><span id="bib.bib31.13.2" class="ltx_text ltx_font_italic" style="font-size:70%;">Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume</span><span id="bib.bib31.14.3" class="ltx_text" style="font-size:70%;">.
</span>
</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib32.5.5.1" class="ltx_text" style="font-size:70%;">Shaikh et al. (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib32.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib32.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Shaikh, O.</span><span id="bib.bib32.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib32.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhang, H.</span><span id="bib.bib32.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib32.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Held, W.</span><span id="bib.bib32.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib32.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bernstein, M.</span><span id="bib.bib32.15.9" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib32.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yang, D.</span><span id="bib.bib32.17.11" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib32.18.1" class="ltx_text" style="font-size:70%;">On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib32.19.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2212.08061</span><span id="bib.bib32.20.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib33.5.5.1" class="ltx_text" style="font-size:70%;">Shao et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib33.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib33.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Shao, Z.</span><span id="bib.bib33.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib33.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gong, Y.</span><span id="bib.bib33.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib33.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Shen, Y.</span><span id="bib.bib33.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib33.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Huang, M.</span><span id="bib.bib33.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib33.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Duan, N.</span><span id="bib.bib33.17.11" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib33.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Chen, W.</span><span id="bib.bib33.19.13" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib33.20.1" class="ltx_text" style="font-size:70%;">Synthetic prompting: Generating chain-of-thought demonstrations for large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib33.21.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2302.00618</span><span id="bib.bib33.22.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib34.5.5.1" class="ltx_text" style="font-size:70%;">Shinn et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib34.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib34.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Shinn, N.</span><span id="bib.bib34.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib34.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Cassano, F.</span><span id="bib.bib34.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib34.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Berman, E.</span><span id="bib.bib34.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib34.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gopinath, A.</span><span id="bib.bib34.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib34.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Narasimhan, K.</span><span id="bib.bib34.17.11" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib34.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yao, S.</span><span id="bib.bib34.19.13" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib34.20.1" class="ltx_text" style="font-size:70%;">Reflexion: Language agents with verbal reinforcement learning.
</span>
</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib35.5.5.1" class="ltx_text" style="font-size:70%;">Sorensen et al. (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib35.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib35.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Sorensen, T.</span><span id="bib.bib35.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib35.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Robinson, J.</span><span id="bib.bib35.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib35.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Rytting, C.</span><span id="bib.bib35.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib35.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Shaw, A.</span><span id="bib.bib35.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib35.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Rogers, K.</span><span id="bib.bib35.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib35.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Delorey, A.</span><span id="bib.bib35.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib35.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Khalil, M.</span><span id="bib.bib35.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib35.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Fulda, N.</span><span id="bib.bib35.23.17" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib35.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wingate, D.</span><span id="bib.bib35.25.19" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.26.1" class="ltx_text" style="font-size:70%;">An information-theoretic approach to prompt engineering without ground truth labels.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.27.1" class="ltx_text" style="font-size:70%;">In </span><span id="bib.bib35.28.2" class="ltx_text ltx_font_italic" style="font-size:70%;">Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</span><span id="bib.bib35.29.3" class="ltx_text" style="font-size:70%;">.
</span>
</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib36.5.5.1" class="ltx_text" style="font-size:70%;">Stechly et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib36.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib36.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Stechly, K.</span><span id="bib.bib36.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib36.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Marquez, M.</span><span id="bib.bib36.11.5" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib36.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Kambhampati, S.</span><span id="bib.bib36.13.7" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib36.14.1" class="ltx_text" style="font-size:70%;">Gpt-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems.
</span>
</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib37.5.5.1" class="ltx_text" style="font-size:70%;">Talmor et al. (2019)</span></span>
<span class="ltx_bibblock"><span id="bib.bib37.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib37.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Talmor, A.</span><span id="bib.bib37.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib37.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Herzig, J.</span><span id="bib.bib37.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib37.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lourie, N.</span><span id="bib.bib37.13.7" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib37.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Berant, J.</span><span id="bib.bib37.15.9" class="ltx_text" style="font-size:70%;"> (2019).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.16.1" class="ltx_text" style="font-size:70%;">Commonsenseqa: A question answering challenge targeting commonsense knowledge.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.17.1" class="ltx_text" style="font-size:70%;">In </span><span id="bib.bib37.18.2" class="ltx_text ltx_font_italic" style="font-size:70%;">Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)</span><span id="bib.bib37.19.3" class="ltx_text" style="font-size:70%;">.
</span>
</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib38.5.5.1" class="ltx_text" style="font-size:70%;">Touvron et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib38.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib38.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Touvron, H.</span><span id="bib.bib38.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib38.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Martin, L.</span><span id="bib.bib38.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib38.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Stone, K.</span><span id="bib.bib38.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib38.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Albert, P.</span><span id="bib.bib38.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib38.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Almahairi, A.</span><span id="bib.bib38.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib38.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Babaei, Y.</span><span id="bib.bib38.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib38.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bashlykov, N.</span><span id="bib.bib38.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib38.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Batra, S.</span><span id="bib.bib38.23.17" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib38.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bhargava, P.</span><span id="bib.bib38.25.19" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib38.26.20" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bhosale, S.</span><span id="bib.bib38.27.21" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib38.28.22" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">et al.</span><span id="bib.bib38.29.23" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.30.1" class="ltx_text" style="font-size:70%;">Llama 2: Open foundation and fine-tuned chat models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.31.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2307.09288</span><span id="bib.bib38.32.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib39.5.5.1" class="ltx_text" style="font-size:70%;">Turpin et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib39.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib39.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Turpin, M.</span><span id="bib.bib39.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib39.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Michael, J.</span><span id="bib.bib39.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib39.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Perez, E.</span><span id="bib.bib39.13.7" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib39.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bowman, S. R.</span><span id="bib.bib39.15.9" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib39.16.1" class="ltx_text" style="font-size:70%;">Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.
</span>
</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib40.5.5.1" class="ltx_text" style="font-size:70%;">Wang et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib40.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib40.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wang, J.</span><span id="bib.bib40.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib40.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Shi, E.</span><span id="bib.bib40.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib40.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yu, S.</span><span id="bib.bib40.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib40.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wu, Z.</span><span id="bib.bib40.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib40.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Ma, C.</span><span id="bib.bib40.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib40.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Dai, H.</span><span id="bib.bib40.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib40.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yang, Q.</span><span id="bib.bib40.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib40.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Kang, Y.</span><span id="bib.bib40.23.17" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib40.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wu, J.</span><span id="bib.bib40.25.19" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib40.26.20" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Hu, H.</span><span id="bib.bib40.27.21" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib40.28.22" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">et al.</span><span id="bib.bib40.29.23" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib40.30.1" class="ltx_text" style="font-size:70%;">Prompt engineering for healthcare: Methodologies and applications.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib40.31.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2304.14670</span><span id="bib.bib40.32.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib41.5.5.1" class="ltx_text" style="font-size:70%;">Wang et al. (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib41.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib41.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wang, X.</span><span id="bib.bib41.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib41.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wei, J.</span><span id="bib.bib41.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib41.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Schuurmans, D.</span><span id="bib.bib41.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib41.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Le, Q.</span><span id="bib.bib41.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib41.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Chi, E.</span><span id="bib.bib41.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib41.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Narang, S.</span><span id="bib.bib41.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib41.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Chowdhery, A.</span><span id="bib.bib41.21.15" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib41.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhou, D.</span><span id="bib.bib41.23.17" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.24.1" class="ltx_text" style="font-size:70%;">Self-consistency improves chain of thought reasoning in language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.25.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2203.11171</span><span id="bib.bib41.26.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib42.5.5.1" class="ltx_text" style="font-size:70%;">Wei et al. (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib42.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib42.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wei, J.</span><span id="bib.bib42.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib42.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wang, X.</span><span id="bib.bib42.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib42.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Schuurmans, D.</span><span id="bib.bib42.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib42.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bosma, M.</span><span id="bib.bib42.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib42.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Xia, F.</span><span id="bib.bib42.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib42.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Chi, E.</span><span id="bib.bib42.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib42.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Le, Q. V.</span><span id="bib.bib42.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib42.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhou, D.</span><span id="bib.bib42.23.17" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib42.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">et al.</span><span id="bib.bib42.25.19" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.26.1" class="ltx_text" style="font-size:70%;">Chain-of-thought prompting elicits reasoning in large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.27.1" class="ltx_text ltx_font_italic" style="font-size:70%;">Advances in Neural Information Processing Systems</span><span id="bib.bib42.28.2" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib42.29.3" class="ltx_text ltx_font_bold" style="font-size:70%;">35</span><span id="bib.bib42.30.4" class="ltx_text" style="font-size:70%;"> 24824–24837.
</span>
</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib43.5.5.1" class="ltx_text" style="font-size:70%;">Welleck et al. (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib43.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib43.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Welleck, S.</span><span id="bib.bib43.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib43.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Lu, X.</span><span id="bib.bib43.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib43.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">West, P.</span><span id="bib.bib43.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib43.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Brahman, F.</span><span id="bib.bib43.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib43.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Shen, T.</span><span id="bib.bib43.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib43.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Khashabi, D.</span><span id="bib.bib43.19.13" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib43.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Choi, Y.</span><span id="bib.bib43.21.15" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib43.22.1" class="ltx_text" style="font-size:70%;">Generating sequences by learning to self-correct.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib43.23.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2211.00053</span><span id="bib.bib43.24.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib44.5.5.1" class="ltx_text" style="font-size:70%;">White et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib44.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib44.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">White, J.</span><span id="bib.bib44.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib44.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Fu, Q.</span><span id="bib.bib44.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib44.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Hays, S.</span><span id="bib.bib44.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib44.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Sandborn, M.</span><span id="bib.bib44.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib44.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Olea, C.</span><span id="bib.bib44.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib44.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gilbert, H.</span><span id="bib.bib44.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib44.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Elnashar, A.</span><span id="bib.bib44.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib44.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Spencer-Smith, J.</span><span id="bib.bib44.23.17" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib44.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Schmidt, D. C.</span><span id="bib.bib44.25.19" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib44.26.1" class="ltx_text" style="font-size:70%;">A prompt pattern catalog to enhance prompt engineering with chatgpt.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib44.27.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2302.11382</span><span id="bib.bib44.28.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib45.5.5.1" class="ltx_text" style="font-size:70%;">Williams et al. (2018)</span></span>
<span class="ltx_bibblock"><span id="bib.bib45.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib45.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Williams, A.</span><span id="bib.bib45.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib45.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Nangia, N.</span><span id="bib.bib45.11.5" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib45.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bowman, S.</span><span id="bib.bib45.13.7" class="ltx_text" style="font-size:70%;"> (2018).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.14.1" class="ltx_text" style="font-size:70%;">A broad-coverage challenge corpus for sentence understanding through inference.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.15.1" class="ltx_text" style="font-size:70%;">In </span><span id="bib.bib45.16.2" class="ltx_text ltx_font_italic" style="font-size:70%;">Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)</span><span id="bib.bib45.17.3" class="ltx_text" style="font-size:70%;"> (M. Walker, H. Ji and A. Stent, eds.). Association for Computational Linguistics, New Orleans, Louisiana.
</span>
</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib46.5.5.1" class="ltx_text" style="font-size:70%;">Yan et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib46.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib46.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yan, H.</span><span id="bib.bib46.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib46.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Srivastava, S.</span><span id="bib.bib46.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib46.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Tai, Y.</span><span id="bib.bib46.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib46.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wang, S. I.</span><span id="bib.bib46.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib46.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yih, W.-t.</span><span id="bib.bib46.17.11" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib46.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yao, Z.</span><span id="bib.bib46.19.13" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib46.20.1" class="ltx_text" style="font-size:70%;">Learning to simulate natural language feedback for interactive semantic parsing.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib46.21.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2305.08195</span><span id="bib.bib46.22.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib47.5.5.1" class="ltx_text" style="font-size:70%;">Yang et al. (2022)</span></span>
<span class="ltx_bibblock"><span id="bib.bib47.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib47.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yang, K.</span><span id="bib.bib47.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib47.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Tian, Y.</span><span id="bib.bib47.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib47.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Peng, N.</span><span id="bib.bib47.13.7" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib47.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Klein, D.</span><span id="bib.bib47.15.9" class="ltx_text" style="font-size:70%;"> (2022).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib47.16.1" class="ltx_text" style="font-size:70%;">Re3: Generating longer stories with recursive reprompting and revision.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib47.17.1" class="ltx_text" style="font-size:70%;">In </span><span id="bib.bib47.18.2" class="ltx_text ltx_font_italic" style="font-size:70%;">Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</span><span id="bib.bib47.19.3" class="ltx_text" style="font-size:70%;">.
</span>
</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib48.5.5.1" class="ltx_text" style="font-size:70%;">Yao et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib48.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib48.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yao, S.</span><span id="bib.bib48.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib48.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yu, D.</span><span id="bib.bib48.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib48.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhao, J.</span><span id="bib.bib48.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib48.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Shafran, I.</span><span id="bib.bib48.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib48.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Griffiths, T. L.</span><span id="bib.bib48.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib48.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Cao, Y.</span><span id="bib.bib48.19.13" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib48.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Narasimhan, K.</span><span id="bib.bib48.21.15" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib48.22.1" class="ltx_text" style="font-size:70%;">Tree of thoughts: Deliberate problem solving with large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib48.23.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2305.10601</span><span id="bib.bib48.24.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib49.5.5.1" class="ltx_text" style="font-size:70%;">Yu et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib49.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib49.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Yu, W.</span><span id="bib.bib49.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib49.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhang, Z.</span><span id="bib.bib49.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib49.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Liang, Z.</span><span id="bib.bib49.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib49.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Jiang, M.</span><span id="bib.bib49.15.9" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib49.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Sabharwal, A.</span><span id="bib.bib49.17.11" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib49.18.1" class="ltx_text" style="font-size:70%;">Improving language models via plug-and-play retrieval feedback.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib49.19.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2305.14002</span><span id="bib.bib49.20.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib50.5.5.1" class="ltx_text" style="font-size:70%;">Zhang et al. (2023a)</span></span>
<span class="ltx_bibblock"><span id="bib.bib50.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib50.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhang, M.</span><span id="bib.bib50.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib50.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Press, O.</span><span id="bib.bib50.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib50.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Merrill, W.</span><span id="bib.bib50.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib50.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Liu, A.</span><span id="bib.bib50.15.9" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib50.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Smith, N. A.</span><span id="bib.bib50.17.11" class="ltx_text" style="font-size:70%;"> (2023a).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib50.18.1" class="ltx_text" style="font-size:70%;">How language model hallucinations can snowball.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib50.19.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2305.13534</span><span id="bib.bib50.20.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib51.5.5.1" class="ltx_text" style="font-size:70%;">Zhang et al. (2023b)</span></span>
<span class="ltx_bibblock"><span id="bib.bib51.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib51.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhang, Z.</span><span id="bib.bib51.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib51.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhang, A.</span><span id="bib.bib51.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib51.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Li, M.</span><span id="bib.bib51.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib51.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhao, H.</span><span id="bib.bib51.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib51.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Karypis, G.</span><span id="bib.bib51.17.11" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib51.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Smola, A.</span><span id="bib.bib51.19.13" class="ltx_text" style="font-size:70%;"> (2023b).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib51.20.1" class="ltx_text" style="font-size:70%;">Multimodal chain-of-thought reasoning in language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib51.21.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2302.00923</span><span id="bib.bib51.22.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib52.5.5.1" class="ltx_text" style="font-size:70%;">Zhou et al. (2022a)</span></span>
<span class="ltx_bibblock"><span id="bib.bib52.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib52.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhou, D.</span><span id="bib.bib52.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib52.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Schärli, N.</span><span id="bib.bib52.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib52.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Hou, L.</span><span id="bib.bib52.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib52.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wei, J.</span><span id="bib.bib52.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib52.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Scales, N.</span><span id="bib.bib52.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib52.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Wang, X.</span><span id="bib.bib52.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib52.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Schuurmans, D.</span><span id="bib.bib52.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib52.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Cui, C.</span><span id="bib.bib52.23.17" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib52.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Bousquet, O.</span><span id="bib.bib52.25.19" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib52.26.20" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Le, Q. V.</span><span id="bib.bib52.27.21" class="ltx_text" style="font-size:70%;"> </span><span id="bib.bib52.28.22" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">et al.</span><span id="bib.bib52.29.23" class="ltx_text" style="font-size:70%;"> (2022a).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib52.30.1" class="ltx_text" style="font-size:70%;">Least-to-most prompting enables complex reasoning in large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib52.31.1" class="ltx_text" style="font-size:70%;">In </span><span id="bib.bib52.32.2" class="ltx_text ltx_font_italic" style="font-size:70%;">The Eleventh International Conference on Learning Representations</span><span id="bib.bib52.33.3" class="ltx_text" style="font-size:70%;">.
</span>
</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib53.5.5.1" class="ltx_text" style="font-size:70%;">Zhou et al. (2023)</span></span>
<span class="ltx_bibblock"><span id="bib.bib53.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib53.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhou, P.</span><span id="bib.bib53.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib53.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Madaan, A.</span><span id="bib.bib53.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib53.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Potharaju, S. P.</span><span id="bib.bib53.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib53.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Gupta, A.</span><span id="bib.bib53.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib53.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">McKee, K. R.</span><span id="bib.bib53.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib53.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Holtzman, A.</span><span id="bib.bib53.19.13" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib53.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Pujara, J.</span><span id="bib.bib53.21.15" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib53.22.16" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Ren, X.</span><span id="bib.bib53.23.17" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib53.24.18" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Mishra, S.</span><span id="bib.bib53.25.19" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib53.26.20" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Nematzadeh, A.</span><span id="bib.bib53.27.21" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib53.28.22" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Upadhyay, S.</span><span id="bib.bib53.29.23" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib53.30.24" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Faruqui, M.</span><span id="bib.bib53.31.25" class="ltx_text" style="font-size:70%;"> (2023).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib53.32.1" class="ltx_text" style="font-size:70%;">How far are large language models from agents with theory-of-mind?
</span>
</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib54.5.5.1" class="ltx_text" style="font-size:70%;">Zhou et al. (2022b)</span></span>
<span class="ltx_bibblock"><span id="bib.bib54.7.1" class="ltx_text" style="font-size:70%;">
</span><span id="bib.bib54.8.2" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Zhou, Y.</span><span id="bib.bib54.9.3" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib54.10.4" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Muresanu, A. I.</span><span id="bib.bib54.11.5" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib54.12.6" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Han, Z.</span><span id="bib.bib54.13.7" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib54.14.8" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Paster, K.</span><span id="bib.bib54.15.9" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib54.16.10" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Pitis, S.</span><span id="bib.bib54.17.11" class="ltx_text" style="font-size:70%;">, </span><span id="bib.bib54.18.12" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Chan, H.</span><span id="bib.bib54.19.13" class="ltx_text" style="font-size:70%;"> and </span><span id="bib.bib54.20.14" class="ltx_text ltx_font_smallcaps" style="font-size:70%;">Ba, J.</span><span id="bib.bib54.21.15" class="ltx_text" style="font-size:70%;"> (2022b).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib54.22.1" class="ltx_text" style="font-size:70%;">Large language models are human-level prompt engineers.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib54.23.1" class="ltx_text ltx_font_italic" style="font-size:70%;">arXiv preprint arXiv:2211.01910</span><span id="bib.bib54.24.2" class="ltx_text" style="font-size:70%;"> .
</span>
</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2311.04204" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2311.04205" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2311.04205">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2311.04205" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2311.04206" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Feb 27 21:06:46 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2311.04205
Misunderstandings arise not only in interpersonal communication but also between humans and Large Language Models (LLMs) . Such discrepancies can make LLMs interpret seemingly unambiguous questions in unexpected ways, yielding incorrect responses.
While it is widely acknowledged that the quality of a prompt, such as a question, significantly impacts the quality of the response provided by LLMs, a systematic method for crafting questions that LLMs can better comprehend is still underdeveloped.
In this paper, we present a method named ‘Rephrase and Respond’ (RaR) , which allows LLMs to rephrase and expand questions posed by humans and provide responses in a single prompt. This approach serves as a simple yet effective prompting method for improving performance.
We also introduce a two-step variant of RaR, where a rephrasing LLM first rephrases the question and then passes the original and rephrased questions together to a different responding LLM. This facilitates the effective utilization of rephrased questions generated by one LLM with another. Our experiments demonstrate that our methods significantly improve the performance of different models across a wide range to tasks.
We further provide a comprehensive comparison between RaR and the popular Chain-of-Thought (CoT)  methods, both theoretically and empirically. We show that RaR is complementary to CoT and can be combined with CoT to achieve even better performance. Our work not only contributes to enhancing LLM performance efficiently and effectively but also sheds light on a fair evaluation of LLM capabilities. Data and codes are available athttps://github.com/uclaml/Rephrase-and-Respond.
abstract
Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
Abstract
2311.04205
1IntroductionMisunderstandings in interpersonal communications often arise when individuals, shaped by distinct subjective experiences, interpret the same message differently.
In social science, such phenomena can be attributed to cognitive biases in frames in thought.
A frame represents an individual’s scheme of interpretation, enabling their understanding and response to an input.
A single message, framed in different ways, can lead individuals to different conclusions.
People habitually project their frames onto their received information, and only shift these frames when incongruence arises.
Recently, Large Language Models (LLMs) , such as the GPT series, have witnessed a surge in popularity due to their profound impact on various real-world applications, including question answering, code generation, and conversational agents.
The wide applicability and efficacy of these models have led to rapidly growing research on understanding and improving the use of LLMs.
In this work, we posit that LLMs also exhibit their own frames in thought, and it is not uncommon to observe a disparity between the frames used by humans and LLMs.
It is widely acknowledged that the quality of the prompt generated by human critically influences the response quality of the LLMs, emphasizing the importance of effective queries that prioritize specificity, detail, and precision.
However, because of an individual’s unique frame of thought, it can be challenging for humans to assess the clarity of their questions and to align their frames with those of LLMs.
To illustrate this, we first present a motivating example by investigating a recent workin detail.Figure 1:Motivating examples of our study. We can observe several ambiguities in the question for LLMs. An LLM can interpret “even month” as the month with even number of days, which diverges from human intention. We note that at each independent query, GPT-4 generates different outputs. Please refer to our experiments for more comprehensive results.Motivating Example.In, the authors reported an important finding: LLMs such as GPT-4 may not efficiently reason with their internal knowledge even if they can retrieve information accurately.
As shown in Figure1, when posed with the query, “Was Mother Teresa born on an even month?” GPT-4 might mistakenly assert that August is an odd month.
Based on this observation,suggested that GPT-4 instead requires a Chain-of-Thought process—relying on user-led follow-up questions—to correct its previous wrong answers. When posed with the follow-up question “Do you know what even means?”, GPT-4 will correct itself.
However, we take a step further to investigate the intrinsic reason for LLM’s inefficiency in answering such questions.
As shown in the other three conversations in Figure1, when GPT-4 explains its reasoning, it appears that the model has several ambiguities toward the questions.
For example, it may consider February as odd due to its irregular number of days and sometimes consider an even/odd month to be months with an even/odd number of days.Ambiguity in questions is a recognized concern in benchmark datasets.
For instance, it has been observed that the NLI datasets such as MultiNLIcontains ambiguities, which are challenging even for human interpreters.
Furthermore, our study uncovers that benchmark datasets commonly used for LLM evaluationpossess ambiguities that are imperceptible to humans but challenging for language models.
These ambiguities cause LLMs to provide mistaken responses to unintended queries.
To address this issue, it is imperative to reduce ambiguity and contextualize information in a way that aligns with the existing frame of the LLMs.Figure 2:Comparison with the original question inand one possible self-rephrased question using GPT-4.
The rephrased questions are longer and more detailed. Especially, self-rephrased questions resolve the ambiguity of the concept of ”even/odd” months by specifying ”even-numbered” and listing the examples.
We can observe a significant improvement in GPT-4’s accuracy in answering these rephrased questions over the query ofcelebrities for this task.Our Approach.In this paper, we highlight an often-overlooked aspect of studies in LLMs: the disparity between human and LLM thought frames.
Our research illustrates that this disparity significantly impacts the performance of LLMs. To tackle this problem, we propose to let the LLM to rephrase the question and incorporate additional details for better answering.
We observe that, as opposed to questions asked casually by human, the rephrased questions tend to enhancesemantic clarityand aid in resolvinginherent ambiguity.
For example, the classification questions intend to be short.
Upon rephrasing by the LLM itself, the newly generated question is more detailed and has a clearer question format, as presented in Figure2.
This self-rephrasing technique leads to significant improvement in accuracy compared with, as shown in the barplot of Figure2.
While GPT-4 indeed found the original questions challenging, it demonstrates the ability to effectively answer the rephrased questions it generates.Building upon these insights, we introduce a method namedRephrase and Respond(RaR) , which prompts the LLM to rearticulate the given question and respond in a single prompt.
In addition to the simple RaR prompt, we also present a variation called Two-step RaR.
Two-step RaR employs a rephrasing LLM to generate reworded questions that can be made available to any responding LLM.
Our empirical results across diverse reasoning tasks show the effectiveness of both approaches.
Notably, Two-step RaR facilitates the transfer of rephrased questions from more capable LLMs to clarify ambiguities for less advanced models.
We also present both theoretical and empirical comparisons with the Chain-of-Thought (CoT)  method.
On the one hand, like CoT, RaR is compatible with the black-box nature of the current powerful GPT-3.5/4 that operate through API services.
On the other hand, while CoT focuses on augmentations either at the beginning or the end of a query, RaR directly modifies the query itself.
Therefore, RaR is complimentary to CoT and can be easily combined for improvement, as confirmed by our experimental results.
Furthermore, unlike methods that employ multiple LLMs for iterative prompt engineering based on accuracy scores, our method is bothunsupervisedandtraining-free, making iteconomical and applicable to all questions.
Lastly, our work call forth the importance that the design of human-crafted tasks targeting specific LLM capabilities should be rigorously reviewed by both humans and LLMs to ensure clarity in intention.The remainder of this paper is organized as follows. Section2introduces the RaR method in detail, including One-step RaR (Section2.1)  and Two-step RaR (Section2.2) . In Section3, we present extensive empirical evaluations of the RaR method, including various benchmark tasks (Section3.1) , the performance on GPT-4 (Section3.2)  and other GPT models (Section3.3) . We also discuss the use of multiple rephrasing processes (Section3.4)  in this section.
In Section4, we compare RaR with CoT in detail both theoretically and experimentally. The related works are discussed in Section5and the conclusion is drawn in Section6.
introduction
Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
1Introduction
2311.04205
2Rephrase and RespondIn this section, we introduce our proposed method in detail, encompassing two principled approaches, namelyOne-step RaRandTwo-step RaRto facilitate better responses from LLMs by letting themselves rephrase the questions. In the following presentation, for the sake of simplicity, we will useRaRto refer toOne-step RaR, unless a specific distinction is necessary.2.1One-step RaR: Rephrase and Respond in a Single PromptIn interpersonal communication, rephrasing is a commonly known technique.
People rephrase another person’s question as a process of understanding, to ensure clarity and coherence in responding.
Such a communication strategy can be similarly applied to an LLM, letting it generate a rephrased question first and provide an answer subsequently.
Following this intuition, we proposeRaRto ask the LLMs toRephrase and Responsethe question using a single query. This approach can be viewed as a strategy to directly enhance the quality of the LLM’s response.
In detail, we introduce the following prompt for the question-answering task:"{question}"Rephrase and expand the question, and respond.(2.1) As we will show in experiments, GPT-4 can achieve much better results using RaR prompt (2.1)  across a wide range of tasks, and especially on human-crafted datasets that exhibit ambiguity to LLMs.Figure 3:Demonstration of One-step RaR: one single prompt to ask the LLM to rephrase, expand and respond.2.2Two-step RaR: Rephrase the Question and Respond to the Rephrased QuestionTo further leverage the quality improvement of the questions rephrased by larger models, like GPT-4, we introduce a variation of RaR calledTwo-step RaR.
Intuitively, even among humans, a more detailed and precise question elicits in more accurate and decisive responses. Two-step RaR follows this intuition by designing a two-step procedure to improve the quality of the questions: in the first step, given a queryquestion, we generate a self-rephrased queryrephrased_questionby prompting arephrasing LLMwith the following prompt:"{question}"Given the above question, rephrase and expand it to help youdo better answering. Maintain all information in the original question.(2.2) Then the original question and the rephrased question are combined to prompt aresponding LLMwith the following prompt:(original)  {question}(rephrased)  {rephrased_question}Use your answer for the rephrased question to answer the original question.(2.3) Notably, the rephrasing LLM and the responding LLM can be either the same or different models. As we will show later in experiments, different LLMs exhibit distinct proficiency in question rephrasing. In particular, a question rephrased by GPT-4 can help a weaker LLM like Vicuna to produce more accurate responses.The enhancement in the response quality can be leveraged to improve the benchmark datasets for a fairer evaluation of LLMs: Existing benchmark datasets, crafted by humans, are designed to assess the performance of LLMs across various reasoning skills. However, as demonstrated in our examples in Figure2, these questions may lack the necessary clarity to fully showcase the specific abilities of LLMs. By the ‘Rephrase’ step in Two-step RaR, we can universally improve the question quality and enable a fairer comparison.In addition, compared with the prompt of One-step RaR, in the Two-step version, we maintain the original context by including the user’s question, while adding the LLM-rephrased question to help better understanding. This prevents the possible divergence of LLMs from the original questions.Figure 4:Demonstration of Two-Step RaR: it involves first rephrasing the question and then using the original and rephrased question to improve the response quality.
body
Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
2Rephrase and Respond
2311.04205
3RaR Effectively Improves LLM ResponsesIn this section, we provide a comprehensive assessment of the applicability and efficacy of RaR. The results are presented in four primary dimensions:One-step RaR is a simple and effective prompt to improve LLM performances;Two-step RaR effectively enhances the response accuracy of GPT-4 across diverse tasks;LLMs, while all benefit from Two-Step RaR, have different proficiency in rephrasing questions;a weak LLM can benefit more from a question rephrased by a strong LLM.3.1Benchmark TasksWe first introduce the benchmark tasks we use to evaluate our method.Tasks.We evaluate the capabilities of LLMs across multiple benchmark tasks in different categories.•Knowledge Classification. Sampling a pool of individuals with Wikipedia pages, this task challenges the LLM to decide if a renowned person was born on an even day, month, or year.•Knowledge Comparison. Using the same pool of individuals, this task instructs the LLM to compare the ages of two people and decide who was born earlier.As GPT-4 responds poorly to many of these questions111As the data are not open-sourced, we let GPT-4 generatefamous individuals with their birth dates andChinese idioms in its knowledge., it raises a concern that whether GPT-4, despite its proficiency in retrieving knowledge, falls behind in reasoning with its own knowledge. Furthermore, we consider the following widely-used datasets for a comprehensive evaluation, which are also considered in.•CSQA. The CommonSense QA data encompasses a range of questions that evaluate the ability of commonsense understanding of the world and involves intricate semantics.•Date Understanding. Sourced from Big-bench, the Date Understanding task emphasizes commonsense reasoning and deducing a date from a provided context. The task is also considered in. We consider a more difficult version where we do not provide the choices of potential answers and let the LLM answer directly.•Last Letter Concatenation. The task centers on symbolic reasoning, and asks the LLM to concatenate the final letters of a given list of names. We consider concatenation for two names as well as a more difficult task of concatenation for four names.•Coin Flip222https://huggingface.co/datasets/skrishna/coin_flip. Sourced from Hugging Face, the task asks the LLM if the coin still heads up, given its initial condition and subsequent actions of people who either flipped or did not flip the coin. We add an additional “Flip means reverse.” to the questions.•Sports. Sourced from Big-bench, the Sports Understanding task primarily asks if a sentence is plausible or implausible, where a prominent sports figure is depicted performing specific sports-related actions.The details of all evaluated tasks are summarized in Table1.Table 1:Details of all the tasks considered in our experiments, including dataset name, category, number of questions, and an example question.DatasetCategoryNumberQuestion exampleEven dayKnowledge Classification105Was Donald Trump born in an even day?Even monthKnowledge Classification105Was Joe Biden born in an even month?Even yearKnowledge Classification105Was Elon Musk born in an even year?Compare ageKnowledge Comparison104Was Joe Biden born earlier than Donald Trump?Chinese idiomInverse knowledge search105成语”X事求是”的X是什么字？CSQACommonsense Reasoning220Everybody has some sort of creativity, it’s the best way to what?Choices: A. Create art B. Write C. Sew D. Create idea E. Express yourselfDatesCommonsense Reasoning369This is the last day of 1899. What is the date tomorrow in MM/DD/YYYY?Last letter (2) Symbolic Reasoning220Take the last letters of the words in “Kristen Andrea” and concatenate them.Last letter (4) Symbolic Reasoning220Take the last letters of the words in “Mia Mariana Marcos Xavier” and concatenate them.Coin flipSymbolic Reasoning220A coin is heads up. roxas does not flip the coin.scheideman does not flip the coin. Is the coin still heads up? Flip means reverse.SportsCommonsense Reasoning220Is the following sentence plausible? “Juan Soto shot from beyond the arc”StereoSetStereotypical Bias220Morocco is a country in Africa. Which of these is more accurate?A. It’s a very poor country with low standards of living.B. There are many cultures and classes of people that live there.C. Blue is her favorite color.We use the entire dataset forDates Understanding, and randomly draw subsets of sizefor the rest tasks.
We use accuracy to evaluate the performance of the LLM. The accuracy is firstly estimated using exact matching on the words generated by the LLM. Specifically, an answer is considered correct if it contains the exact word of the correct response and without any incorrect responses.
We subsequently verify and correct the calculations through manual inspection.
For certain tasks, to constrain the response format (e.g., multiple-choice) , we append a consistent prompt when evaluating the original question and RaR, such as “Select the single most appropriate answer”. Details of the prompts are presented in Table8in AppendixA.3.2Performance on GPT-4We conduct experiments on the aforementioned benchmark datasets using GPT-4333We note that all our experiments accessed GPT-4 during 10/01-10/30. We also include results onGPT-4-0613., both with One-step RaR and Two-step RaR. As shown in Figure5, both One-step RaR and Two-step RaR enjoy superior performance compared with using original questions. We will discuss the findings in the sequel.Figure 5:Accuracy (%)  comparison of different prompts using GPT-4. Both One-step RaR and Two-step RaR effectively improve the accuracy of GPT-4 acrosstasks. A larger margin indicates a less well-designed question. For tasks such as CSQA where the question quality is good enough, our method still offers a marginal improvement.3.2.1RaR: A Simple Prompt to Improve LLM PerformanceWe investigate the performance of RaR, which allows the LLM to both rephrase and respond to the question in a single query.
Such an approach can be considered as a simple black-box strategy to improve the LLM’s performance on any question.In Figure5and detailed in Table7in AppendixA, we compare the accuracy of GPT-4 with One-step RaR (i.e., rephrasing and answering a question in a single prompt)  and Two-step RaR (answering a pre-rephrased question in separate queries) . Notably, One-step RaR improves GPT-4’s accuracy, and outperforms Two-step RaR onout oftasks. Indeed, similar to human communication, rephrasing and elaborating a question and then answering is an effective approach. The key takeaway from this experiment in highlighted below.Takeaway #1(One-step)  RaR provides a universal, plug-and-play black-box prompt that allows for efficient and effective performance improvement of LLMs on general tasks.3.2.2Two-step RaR: Rephrased Questions Improve Response QualityWe evaluate the quality improvement of the question using Two-step RaR. In detail, for each query, GPT-4 autonomously generates a rephrased question using the prompt (2.2)  without any external intervention. The rephrased question is then combined with the original question using (2.3)  to prompt GPT-4.
We present the accuracy of GPT-4 using Two-step RaR and compare it with GPT-4 using the original questions as shown in Figure5.
Across a diverse span of tasks that emphasize different aspects of LLM’s capabilities, Two-step RaR consistently yields distinguishable improvements for GPT-4.
Notably, for tasks that GPT-4 originally finds highly challenging (e.g., last letter concatenation) , the Two-step RaR exhibits remarkable improvement even to almostaccuracy. The numerical details of the accuracy are also presented in Table7in AppendixA.
We conclude this experiment by the following takeaway.Takeaway #2•Examining the question quality is pivotal when evaluating the LLM performance on QA tasks.•Two-step RaR provides a universal method for LLMs to improve the question quality autonomously by rephrasing the question.3.3Performance across Various LLMsWe further examine the performance of RaR on various LLMs, including GPT-3.5 and Vicunasize=,color=orange!20!white,]Quanquan: references. In particular, we employ Two-step RaR to investigate (1)  if all these LLMs can provide consistent response improvement by rephrasing the questions; and (2)  if the GPT-4-rephrased questions can improve the performance of other LLMs.3.3.1Can All LLMs Rephrase Questions?Figure 6:Accuracy (%)  ofGPT-4-0613,GPT-3.5-turbo-0613and Vicuna-13b when testing on original and self-rephrased questions using Two-step RaR. GPT-4 enjoys larger improvement on various tasks compared with GPT -3.5 and Vicuna.We investigate the rephrasing abilities of different LLMs by employing Two-step RaR to examine the quality of the rephrased questions. We evaluate the performance of several different LLMs, includingGPT-4-0613,GPT-3.5-turbo-0613, andVicuna-13b-v1.5, using Two-step RaR.
We present the experiment results in Figure6.
Due toVicuna-13b-v1.5’s near-zero performance on Last Letter Concatenation (4) , we exclude this task from the evaluation ofVicuna-13b-v1.5.
Remarkably, all examined LLMs demonstrate enhanced performance with Two-step RaR, resulting in a notable increase in accuracy across the majority of the tasks.
More advanced models, such as GPT-4, benefit from the most significant gains across all tasks, while models of lesser complexity, like Vicuna, achieve modest improvements using our approach.
On certain tasks such as CSQA and Sports, GPT-3.5 and Vicuna even exhibit slightly diminished performance.
In Table2, we closely examine specific examples of self-rephrased questions by different models.
Initial observations suggest thatVicuna-13b-v1.5’s rephrased questions seldom offer substantial clarification, often mirroring the simplicity of their original questions.
In the last instance of Table2,Vicuna-13b-v1.5perturbs the question’s intent by changing “yesterday” to “today”.
While both GPT-3.5 and GPT-4 can elucidate questions, GPT-3.5 occasionally introduces extra details or misinterpretations.
As shown in the second example of Table2, GPT-3.5 misinterprets the concept of even month as “a month with an even number of days”. Similarly, in the third example, GPT-3.5 introduces a wrong constraint of “recent” game.
GPT-4, on the contrary, is able to make clarifications that are mostly close to human intention.
We also observe that GPT-3.5 tends to introduce the following phrase to rephrased questions in Sports (out of)  and Dates (out of) : “Please rephrase and provide additional details if necessary to enhance your response accuracy.”, resulting in an answer with just another rephrased question but not the actual answer. Therefore, we remove all sentences containing “rephrase” for GPT-3.5 on these two datasets.Table 2:Examples of the self-rephrased questions generated by different LLM models. A stronger model makes fewer mistakes in rephrasing the questions.ModelSelf-rephrased QuestionOriginalWas Barack Obama born in an even day?GPT-4-0613Did Barack Obama’s birth occur on aday of the monththat is considered an even number?GPT-3.5-turbo-0613Was Barack Obama born on a day that is evenly divisible by two, resulting in a whole numberwith no remainder when divided by two?Vicuna-13b-v1.5What is the birthday of Barack Obama, and is it an even day?OriginalWas Abraham Lincoln born in an even month?GPT-4-0613Was the former United States president Abraham Lincoln born in a month that is representedby aneven number, from January being ‘1’ to December being ‘12’?GPT-3.5-turbo-0613Can you please clarify whether Abraham Lincoln, the 16th President of the United States, wasborn ina month with an even number of days?Vicuna-13b-v1.5What was the month in which Abraham Lincoln was born, and is that month even or odd?OriginalIs the following sentence plausible? “Amari Cooper scored a touchdown”GPT-4-0613Is it believable or likely that Amari Cooper, who isknown for playing football, scored atouchdown?GPT-3.5-turbo-0613Can you confirm if it is reasonable to say, based on current facts or observations, that AmariCooper,the football player, successfully achieved a touchdown during arecentgame or event?Vicuna-13b-v1.5Is it plausible that Amari Cooper scored a touchdown?OriginalYesterday was April 30, 2021. What is the date tomorrow in MM/DD/YYYY?GPT-4-0613If yesterday was the last day of April in the year 2021, which is 04/30/2021, can we figureout what the date will be the day after today, using the format of the month first, then day, andlastly year (MM/DD/YYYY) ?GPT-3.5-turbo-0613Yesterday, which was April 30, 2021, just passed. I am looking to determine the date thatcomes after today. To provide an accurate response, I need to express the date format asMM/DD/YYYY. Can you please assist me in figuring out the date?Vicuna-13b-v1.5Iftodayis April 30, 2021, what will be the date tomorrow in the format of MM/DD/YYYY?We wrap up this experiment with the following key insight.Takeaway #3All models can benefit from rephrasing questions, with more advanced models expected to gain a larger improvement.3.3.2Are the Rephrased Questions Transferable?Table 3:Comparison of GPT-4’s rephrased questions with Vicuna’s self-rephrased questions. We drop the Even Year task since the rephrased questions by Vicuna are clear enough as those by GPT-4.Even dayEven monthCSQASportsDatesLast letter (2) Vicuna-rephrased66.3674.0958.1850.9133.335.45GPT-4-rephrased72.7376.8259.5560.9139.3010.91Here, we examine if the rephrased questions generated by Two-step RaR are transferable across different LLMs. In particular, we would like to know if the rephrased questions generated by GPT-4 can benefit Vicuna’s performance. We detailVicuna-13b-v1.5’s performance on questions rephrased by GPT-4, as compared to its own rephrased questions in Table3.
Consistent with our expectation that GPT-4 can better align with human intention and clarify the question, we observe that its rephrased questions remarkably enhanceVicuna-13b-v1.5’s performance on several tasks, especially when Vicuna’s self-rephrased questions exhibit low quality.
Indeed, the questions can be clarified further for Vicuna, but more exploration needs to be made on its capability of self-rephrased questions.
We conclude this experiment by the following key message.Takeaway #4The rephrased questions are transferable: the questions rephrased by GPT-4 can improve the response quality on Vicuna.3.4Multiple Rephrasings: Will the Questions Converge?In this subsection, we explore whether iterative self-rephrasing by GPT-4 yields consistent clarifications when using Two-step RaR.
Specifically, we utilize prompt (2.2)  in Two-step RaR to enable GPT-4 to rephrase a question, then feed its output back into the same prompt (2.2)  for a second and third round of rephrasing.
In Table4, we consider “Was Abraham Lincoln born on an even day?” as an example question and use it for three successive self-rephrasings by GPT-4 across different runs.
The key clarification that needs to be made here is on the concept of “even day”.
While humans understand that “even day” refers to whether the day of the month is even, LLMs may understand it as either an even day of the week or year.
We observe that although GPT-4 sometimes might not clarify this concept in its initial attempt, by the third rephrasing, it converges to a consistent explanation of “even day”.
Meanwhile, the question gets more and more elaborate after multiple rephrasings. This conveys the following key message.Table 4:Examples of the self-rephrased questions generated by GPT-4 after the 1st, 2nd and 3rd self-rephrasing. The original question is “Was Abraham Lincoln born in an even day?” The key clarification that needs to be made is on the concept of “even day” of the month instead of even day of the year or the week.Rephrase timesSelf-rephrased Question1stDid the birth of Abraham Lincoln, the sixteenth president of the United States, occur on adayof the monththat is considered an even number?2ndCan you confirm if the birth date of Abraham Lincoln, who was the sixteenth person to hold theoffice of the President of the United States, falls on an even number in themonthly calendar?3rdCould you verify whether theday of the monthon which Abraham Lincoln, the sixteenthpresident of the United States, was born, is an even number? The focus here is on identifying thespecific date and determining if it falls on an even or odd day according to the calendar.1stIs the date of Abraham Lincoln’s birth an even day? Provide more details, if possible, about hisbirth date.2ndCan you confirm whether the day on which Abraham Lincoln was born is an even number ornot? Please provide as much detailed information as possible concerning his birth date.3rdCould you verify if theday of the monthon which Abraham Lincoln was born falls on an evennumber, please provide a comprehensive description around his birth date, while emphasizing onthe day of his birth especially, to assist with this confirmation?Takeaway #5GPT-4 can potentially clarify concepts with multiple rephrasing, even if it fails to make it in the initial attempt.
body
Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
3RaR Effectively Improves LLM Responses
2311.04205
4Comparison with Chain-of-ThoughtIn this section, we compare RaR with CoT. We first present the mathematical formualtions of RaR and CoT and compare them with each other. Then we present experimental results to show that (1)  RaR offers improvements in scenarios where zero-shot CoT is ineffective; and (2)  RaR addresses and corrects the shortcomings inherent in few-shot CoT.4.1Mathematical FormulationIn this subsection, we discuss the formulations of CoT and RaR, respectively. We denote the LLM model by. In detail, LLMs take the sequenceas prompt, and generate the sentence following the distribution. Recently, there has been significant research effort emphasizing the use of instructions to enhance the quality of the generated text. Mathematically, instead of directly using the promptto generate the response following, one can use a different promptaugmented by instructionto generate a different response following. We hypothesize that a successful instructionallows us to extract a better answer from LLM. In this subsection, we employ the symbolto represent the target answer we aim to generate. The notationis used to denote an extended text that encompasses the desired answeras well as additional details, such as the underlying reasoning. Often,is produced by prompting with instructions like a Chain of Thought (CoT) .4.1.1Chain-of-Thought (CoT) The core concept behind the CoT is to generate a textsuch thatincludes intermediate CoT steps and the final answer. In particular,(4.1) whereare intermediate CoT steps that progressively lead to the final answer. In essence, CoT consists of the following primary phases.CoT•Find an instruction, and generate promptfor zero-shot CoT orfor few-shot CoT.•Generatefollowing the “step by step” format of (4.1) .contains the intermediate CoT steps, and the desired question.•Extract the desired responsefrom the sequence.For zero-shot CoT, the instructionis composed of some task-independent tokens, such as “Let’s think step by step”. For few-shot CoT, the instruction/contextconsists of some task-dependent tokens, which include several examples like, whereis the number of in-context examples, i.e.,. We give examples of zero-shot CoT and few-shot CoT as follows.Example 4.1(Zero-shot CoT) .Consider, for zero-shot CoT, we haveas the effective prompt and generate, where•.•.•.•.Example 4.2(Few-shot CoT) .Consider, for two-shot CoT, we haveas the effective prompt, where•.•.•.•.And we obtainas•.4.1.2One-step RaRThe foundation of our (one-step)  RaR method is different from CoT. We generate a rephrased questionthat retains the same semantic content as, and the associated answer. Specifically, we defineas(4.2) whereis the rephrased question that induces the answer. In particular, RaR consists of two primary phases:RaR•Find an instruction, generatefollowing the “Rephrase and Respond” format of (4.2) .contains the rephrased questionand the desired question.•Extract the desired answerfrom the sequence.We provide an example of (One-step)  RaR as follows.Example 4.3(RaR) .Consider the question, for RaR, we have‘‘Rephrase and expand the question, and respond.’’as the instruction and generate, where•.•.Unlike CoT outlined in (4.1) , which generates numerous intermediate steps, RaR in (4.2)  aims to come up with an improved questionefficiently. In this sense, our method RaR is more cost-effective in terms of token usage than CoT.4.1.3Two-step RaRInstead of generating an extended textthat encompasses the answer, the Two-step RaR approach operates in a sequential way. Specifically, it first employs the rephrasing LLM, denoted byto generate the rephrased question. Then, we input both original and rephrased questions to the responding LLM, denoted byto generate the answer.Two-step RaR•(Rephrase Step)  Find an instruction, generate rephrased question.•(Respond Step)  Generate the desired responsefollowing.We also give an example for Two-step RaR below.Example 4.4(Two-step RaR) .Considering the question, for Two-step RaR, we have‘‘Given the above question, rephrase and expand it to help youdo better answering. Maintain all information in the original question.’’as the instruction and generateThen we feedtogether into the LLM and get.In our experiments, we find that Two-step RaR can consistently achieve better performance. The rephrased question can also be used by another LLM, which makes our RaR method more flexible.4.1.4Combining RaR and CoTIn addition, our method is complementary to CoT and can be naturally combined with CoT. For zero-shot CoT, we can simply concatenate the two instructions to obtain, such as “Given the above question, rephrase and expand it to help you
do better answering. Lastly, let’s think step by step to answer”.
For few-shot CoT, where the instruction/context is, we can use Two-step RaR to improve its few-shot examples by the following procedure.RaR + CoT•Use RaR instruction, generatein the Rephrase step.•Applyto get.•Extract the responsefrom the sequenceby eliminating intermediate steps.Remark 4.1.Compared with Two-step RaR, which uses both the original questionand the rephrased questionto prompt the LLM to generate the response, RaR+CoT only uses the rephrased few-shot examplesinstead of combining with the original few-shot examples. This will save the token usage and prevent an increase in the number of in-context examples while maintaining a similar performance.The following example showcases how to combine RaR with CoT.Example 4.5(RaR + Few-shot CoT) .Consider two-shot CoT instruction, where••••Applyingas in Figure9, we get‘‘Q: Identify the last letters of each word in the name "Elon Musk", …together.A: The last letter of "Elon" is "n". …, the result is nk.’’Q: Identify the last letters of each word in the name "Larry Page", …together.A: The last letter of "Larry" is "y". …The result is ye.Our experiment demonstrates that the integration of RaR with few-shot CoT significantly improves the performance of CoT. A comprehensive discussion of the results can be found in Section4.3.
Lastly, we present illustrations of our mathematical formulations for CoT and RaR in Figure7.Figure 7:Demonstration of our mathematical formulations of RaR and CoT methods with question. In particular,A. One-step RaRcan be viewed as using a single promptto generate the rephrasedand answer.B. Two-step RaRcan be viewed as first rephrasing the questionusing prompt, then combining the original questionand the rephrased questionto answer.C. Zero-shot CoTis a promptto enable LLM to generate the CoT stepsthen lead to the answer.D. RaR + Zero-shot CoTis instructing the LLM to first rephrase the questionand then provide the CoT steps before answering.E. Few-shot CoTprovides several human-crafted CoT stepsfor the model to generate the CoT steps for the question and then the answer.F. Two-step RaR + Few-shot CoTinstructs the model to first rephrase the human-crafted CoT steps tobefore using them as examples to generate the CoT steps and the answer. We denote the first prompt used in Two-step RaR (i.e., Prompt (2.2) )  as Two-step1in this figure.4.2Empirical Comparison with Zero-Shot CoTIt is widely known that zero-shot CoT, by appending the instruction “Let’s think step by step.” to queries, can effectively improve the performance of LLMs on reasoning tasks.
However, we highlight some examples where zero-shot CoT fails to deliver improvements, sometimes even leading to diminished performance. In contrast, RaR consistently demonstrates effectiveness.
We also emphasize the importance of question quality with an example, demonstrating that it should be prioritized before enhancing the model’s reasoning capabilities.
Lastly, we note that our method is complementary to zero-shot CoT and can be combined together by simply adding “let’s think step by step” to (2.3)  or (2.1) .Tasks.We examine theChinese Idiomtask as introduced in, specifically the most difficult task of inferring the first letter. This task involves taking widely recognized four-character Chinese idioms and masking one character at each respective position.
The task is to let LLM correctly infer the masked character.
It has been discovered that GPT models suffer from inferring the masked character, particularly when it is located in the first position.
Furthermore, we also use theStereoSettask, which assesses the stereotypical biases present in LLMs with respect to gender, race, profession, and religion.
From the inter-sentence data, we sampleexamples, each comprising a context sentence and three choices: one stereotypical, one anti-stereotypical, and one unrelated.
We adopt the prompt format used by.Evaluation.For the Chinese Idiom task, we evaluate the zero-shot accuracy of GPT-4’s responses, with automated accuracy estimation and further manual checking. For StereoSet, as suggested by, two crucial evaluation metrics should be considered: theLanguage Modeling Score, which assesses whether the LLM selects related options over unrelated ones, and theStereotype Score, which quantifies the percentage of data that a model favors stereotypical choices over anti-stereotypical ones.
As identified by the authors, an ideal model would display no bias toward either stereotypical or anti-stereotypical associations, yielding an optimal score offor the Stereotype Score.
In our examination of GPT-4’s outputs, we observe its capability to actually determine that neither of the two related options can be concluded solely from the context sentence.
Consequently, we categorize such outputs as fair responses and introduce aFair Score, determined by the proportion of these responses, complementing theLanguage Modeling Score. We provide an example of such a response below.The question does not provide enough information to determine the accuracy of optionsA and C. Option B is irrelevant to the given context. Therefore, none of the providedoptions can be selected as the most sensible answer based on the information available.Results.As illustrated in Table5, even though RaR enhances LLM’s performance, accurately inferring the first character of the Chinese Idiom task remains a challenge.
One might then ask: does zero-shot CoT provide consistent improvement to LLM in such tasks as it does on other reasoning tasks?
Our discovery is, in fact, zero-shot CoT may result in worse performances ()  for such hard tasks, as the LLM tends to hallucinate during the intermediate steps—a phenomenon similar to hallucination snowballing.
Furthermore, asdiscovered on other language models, zero-shot CoT may result in undesired reasoning towards bias and toxicity.
Also in Table5, we demonstrate the performance of GPT-4 on StereoSet.
We can observe that, while zero-shot CoT fails to improve the Language Modeling Score, rephrased questions improve it significantly to. This implies that, with RaR, the LLM rarely opts for unrelated choices.
Moreover, while zero-shot CoT improves the percentage of fair responses (choosing neither of them) , RaR achieves the best performance.Table 5:Performance of GPT-4 using original questions, zero-shot CoT prompt and RaR. We consider two tasks, Chinese Idiom and StereoSet. Zero-shot CoT results in even worse accuracy of Chinese Idiom due to hallucination in the reasoning steps. On StereoSet, evaluations include Language Modeling Score and Fair Score, where RaR demonstrated the best performance.DatasetEvaluationOriginalZero-shot CoTRaRChinese IdiomAccuracy () 32.3831.4335.24StereoSetLanguage Modeling Score () 84.0984.0997.73Fair Score () 6.8235.0042.27Question quality comes before reasoning.With the following example, we emphasize that the attention to question quality is more important before considering to improve model reasoning. We examine the originalCoin Flipquestions. Specifically, an example question isA coin is heads up. aluino flips the coin. arthor flips the coin. Is the coin still heads up?This question, as originally crafted by Humans, appears clear to human interpreters that “flipping” the coin here means reversing the coin.
However, LLMs like GPT-4 might perceive the flipping as a random toss.
As shown in Figure8, such misconception persists even when prompting the LLM to think step by step, which therefore results in an incorrect answer. Once we add a clarifying sentence stating that flip means reverse, GPT-4 can finally start answering the question as we desire. Since the clarification is also created by humans, LLM still exhibit an unsatisfactory performance of. With self-rephrased questions, the accuracy can finally be improved to.
In light of such instances, we advocate for careful examination of human-crafted questions when evaluating LLMs, ensuring the removal of ambiguities for a fair assessment.Figure 8:Demonstration of the importance of question quality as compared to reasoning in the coin flip question. We highlight the incorrect responses inred. As shown in the left, when the concept of “flip” is not clarified for GPT-4, prompting it to further reasoning will not resolve the problem as the ambiguity persists through the reasoning steps. Only after clarifying that “flip” means the action of reverse will enable the LLM to correctly respond to the question. Lastly, the human-crafted clarification still results in false response. Meanwhile, the application of RaR can effectively improve any human-crafted phrases.4.3Empirical Improvement on Few-Shot CoTFew-shot CoThas been the most effective CoT technique. It employs a small set of human-crafted QA examples to facilitate LLMs in addressing similar questions with a congruent structure. LLMs, particularly advanced models such as GPT-4, are adept at extrapolating from the provided examples to improve their performance on new questions.
Providing question-answer pairs effectively communicates the human-desired logical structure to the LLM.
Instead of aligning the question to what the LLM best receives, few-shot CoT guides the LLM to reason using the supplied human logic.
Nonetheless, a concern emerges: How do LLMs respond when the human-crafted examples are flawed or contain errors?
As corroborated by a recent parallel study, we similarly observe that LLMs can be adversely influenced by bad few-shot examples.Tasks.We revisit theLast Letter Concatenationtask and refer to the few-shot examples provided in. As shown in Figure9, the examples follow a specific logic: (1)  obtain the last letter of the first word; (2)  obtain the last letter of the second word; (3)  concatenate these letters; resulting in (4)  the answer. Such few-shot examples have been demonstrated to most effectively enhance the performance of a language model, achieving an accuracy ofwhen concatenating the last letters of two words. Conversely, we explore an example that employs the following logic: (1)  obtain thefirstletter of the first word; (2)  obtain thefirstletter of the second word; (3)  concatenate these letters; providing (4)  the answer forlastletter concatenation.
Our aim is to investigate how this alternative few-shot prompt, despite bearing a logic similar to the original prompt and the correct answer, influences the performance of the GPT-4.Results.As illustrated in Figure9, GPT-4 tends to stick to the logic of our modified prompt, resulting in an incorrect answer.
It accurately concatenates all first letters, but concludes with a seemingly arbitrary final answer.
In Table6, we demonstrate the results of one-shot and four-shot CoT using such examples. We observe that the performance of the one-shot CoT evidently degraded with just one flawed example.
As the number of these flawed examples increases, the performance of GPT-4 in a 4-shot setting for last letter concatenation of four words drops to only.
This observation reveals a potential pitfall in employing few-shot CoT: given that these examples are user-crafted, their quality becomes vital.
Meanwhile, we discovered that RaR enables GPT-4 to correct any pitfalls in the logic of the given examples.Figure 9:Few-shot CoT can effectively instruct the LLM to reason in a similar manner to the provided examples. A badly crafted QA example, as shown inred, result in the LLM following the provided logic but reaching an arbitrary answer.
Slight modification to the few-shot CoT prompt results in notable degradation of LLM’s performance. Meanwhile, our prompt can successfully correct the pitfalls in the few-shot examples and improve the robustness and efficacy of few-shot CoT.Table 6:Accuracy (%)  of GPT-4 on Last Letter Concatenation task. We use one-shot and four-shot examples from, only changing the intermediate steps to be “first letter” as shown in our example.
Based on our modified examples, the accuracy of GPT-4 are not satisfactory, especially on the task of concatenating four words. Moreover, more of such examples would result in further diminished performance. Meanwhile, such potential risk can be resolved by applying RaR to improve the human-crafted few-shot examples, which increase the accuracy to near.Last letter (2) Last letter (4) One-shot CoT89.0478.18RaR+One-shot CoT10093.64Four-shot CoT87.2152.27RaR+Four-shot CoT10095.45
body
Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
4Comparison with Chain-of-Thought
2311.04205
5Related Work5.1Prompting Methods for LLMsSince the advent of recent LLMs, a growing body of research has focused on prompt engineering for LLMs.
Manual guidelines have emerged to guide users in designing and revising their prompts.
Notably, studies have shown that a well-crafted system message, such as ”You are a helpful assistant who always provides explanations,” preceding the main query can encourage an LLM to respond with greater subject expertise.has also offered general recommendations for crafting queries, emphasizing specificity, detail, and precision. However, individuals often find it challenging to refine their own questions for clarity or to include necessary details for LLMs, as the questions are clear enough for humans themselves.Subsequent researchhas concentrated on the autonomous refinement of prompts. These methods often employ multiple LLMs to generate candidate prompts, evaluate and score these prompts, and iteratively refine them until a satisfactory prompt is produced.
The evaluation of a prompt typically relies on either the accuracy of an LLM’s response (supervised,)  or the mutual information of the question (unsupervised,) . Given the nature of iterative computation and the necessity for qualitative evaluation, such methods are employed for refining single prompt templates; applying them universally to all questions would be expensive. Consequently, these techniques are less frequently adopted in daily user cases.The method most frequently used by users and closely aligned with our approach is the Chain-of-Thought (CoT)  prompting, which can be either zero-shotor few-shot. Given that these techniques do not require evaluation and iterative selection, they have gained widespread popularity and inspired a series of subsequent studies. However, CoT methods are not without their limitations, as observed in our study. Recent investigations have also highlighted challenges with the reliability of both zero-shot CoTand few-shot CoT. Most recently,propose Foresee and Reflect similarly as a zero-shot prompting method that targets the proposed task Thinking for Doing (T4D) .
Lastly, it is worth noting that our method is complementary to all the prompting techniques mentioned above and can be combined with them.5.2Self-correction Methods for LLMsAnother line of work aims at enhancing LLM performanceby leveraging the LLM to refine its ownresponses, a concept known as post-hoc prompting. This encompasses terms such as ”self-correction,” ”self-refine,” and ”self-critique,” where LLMs revise their own responses drawing upon various feedback sources orcritic models. As classified by, automated critic models generally employ the LLM’s self-feedback, other trained LLMs, or external references. Yet, recent studiesexamine the self-correction capacities of LLMs and find potential limitations, suggesting that LLMs may not be able to self-correct their reasoning processes. Their findings reveal that self-correction is no better than self-consistency. Contrary to allowing the LLM to self-refine its responses, our methodology let the LLM instead rephrase questions originally crafted by humans.
body
Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
5Related Work
2311.04205
6ConclusionIn this paper, we have investigated the existing misunderstandings that occur between humans and LLMs and demonstrated that questions that appear clear to humans may still be misinterpreted by LLMs.
Building on this insight, we introduced Rephrase and Respond (RaR) , a novel approach that prompts an LLM to first rephrase and clarify the question before answering it.
We also presented Two-step RaR, a variation of RaR that employs a rephrasing LLM to refine questions for subsequent use by any responding LLM.
Our empirical evaluations, conducted across a range of benchmark datasets, confirm the effectiveness of our proposed methods.
Further analysis reveals that while all models gain enhanced performance through question rephrasing, the more sophisticated models exhibit more substantial improvements.
Crucially, we have found that the enhancement in question quality achieved through rephrasing is transferable across models.
In addition to these findings, we have made comparisons with CoT methods through both mathematical formulation and empirical investigations. We also demonstrated that RaR is complementary to CoT, and can be leveraged to achieve additional performance gains.
conclusion
Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
6Conclusion