<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2402.12219] Reformatted Alignment</title><meta property="og:description" content="The quality of finetuning data is crucial for aligning large language models (LLMs) with human values.
Current methods to improve data quality are either labor-intensive or prone to factual errors caused by LLM halluci…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Reformatted Alignment">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Reformatted Alignment">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2402.12219">

<!--Generated on Tue Mar  5 17:45:32 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<div id="p1" class="ltx_para">
<p id="p1.1" class="ltx_p">showstringspaces = false,
keywords = false,true,
alsoletter = 0123456789.,
morestring = [s]””,
stringstyle = ,
MoreSelectCharTable =<span id="p1.1.1" class="ltx_ERROR undefined">\lst@DefSaveDef</span>‘:<span id="p1.1.2" class="ltx_ERROR undefined">\colon@json</span><span id="p1.1.3" class="ltx_ERROR undefined">\processColon@json</span>,
basicstyle = <span id="p1.1.4" class="ltx_text ltx_font_typewriter">,
keywordstyle = <span id="p1.1.4.1" class="ltx_text ltx_font_bold">,
</span></span>















<span id="p1.1.5" class="ltx_text" lang="en"></span></p>
</div>
<h1 class="ltx_title ltx_title_document" lang="en">Reformatted Alignment</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname" lang="en">Run-Ze Fan<sup id="id1.1.id1" class="ltx_sup">1,6</sup>       Xuefeng Li<sup id="id2.2.id2" class="ltx_sup">1,6</sup>       Haoyang Zou<sup id="id3.3.id3" class="ltx_sup">3,6</sup>       Junlong Li<sup id="id4.4.id4" class="ltx_sup">1,6</sup>       <span id="id5.5.id5" class="ltx_text ltx_font_bold">Shwai He<sup id="id5.5.id5.1" class="ltx_sup"><span id="id5.5.id5.1.1" class="ltx_text ltx_font_medium">4</span></sup>
<br class="ltx_break">Ethan Chern<sup id="id5.5.id5.2" class="ltx_sup"><span id="id5.5.id5.2.1" class="ltx_text ltx_font_medium">1,6</span></sup></span>      <span id="id6.6.id6" class="ltx_text ltx_font_bold">Jiewen Hu<sup id="id6.6.id6.1" class="ltx_sup"><span id="id6.6.id6.1.1" class="ltx_text ltx_font_medium">5,6</span></sup></span>      <span id="id7.7.id7" class="ltx_text ltx_font_bold">Pengfei Liu<sup id="id7.7.id7.1" class="ltx_sup"><span id="id7.7.id7.1.1" class="ltx_text ltx_font_medium">1,2,6</span></sup>
<br class="ltx_break"><sup id="id7.7.id7.2" class="ltx_sup"><span id="id7.7.id7.2.1" class="ltx_text ltx_font_medium">1</span></sup></span>Shanghai Jiao Tong University  <sup id="id8.8.id8" class="ltx_sup">2</sup>Shanghai Artificial Intelligence Laboratory 
<br class="ltx_break"><sup id="id9.9.id9" class="ltx_sup">3</sup>Fudan University   <sup id="id10.10.id10" class="ltx_sup">4</sup>University of Maryland, College Park 
<br class="ltx_break"><sup id="id11.11.id11" class="ltx_sup">5</sup>CMU  <sup id="id12.12.id12" class="ltx_sup">6</sup>Generative AI Research Lab (GAIR)
<br class="ltx_break"><span id="id13.13.id13" class="ltx_text ltx_font_typewriter">runze.fan@icloud.com</span>,  <span id="id14.14.id14" class="ltx_text ltx_font_typewriter">pengfei@sjtu.edu.cn</span>
</span><span class="ltx_author_notes">  Corresponding author</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id15.id1" class="ltx_p"><span id="id15.id1.1" class="ltx_text" lang="en">The quality of finetuning data is crucial for aligning large language models (LLMs) with human values.
Current methods to improve data quality are either labor-intensive or prone to factual errors caused by LLM hallucinations.
This paper explores elevating the quality of existing instruction data to better align with human values, introducing a simple and effective approach named <span id="id15.id1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>, which <span id="id15.id1.1.2" class="ltx_text ltx_font_italic">reformats</span> the responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence.
This approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques.
Experimentally, <span id="id15.id1.1.3" class="ltx_text ltx_font_smallcaps">ReAlign</span> significantly boosts the general alignment ability, math reasoning, factuality, and readability of the LLMs.</span></p>
<p id="id16.id2" class="ltx_p"><span id="id16.id2.1" class="ltx_text" lang="en">Encouragingly, <em id="id16.id2.1.1" class="ltx_emph ltx_font_italic">without</em> introducing any additional data or advanced training techniques, and merely by reformatting the response, LLaMA-2-13B’s mathematical reasoning ability on <span id="id16.id2.1.2" class="ltx_text ltx_font_typewriter">GSM8K</span> can be improved <span id="id16.id2.1.3" class="ltx_text ltx_font_bold">from 46.77% to 56.63%</span> in accuracy.
Additionally, a mere 5% of <span id="id16.id2.1.4" class="ltx_text ltx_font_smallcaps">ReAlign</span> data yields a 67% boost in general alignment ability measured by the Alpaca dataset.
This work highlights the need for further research into the <em id="id16.id2.1.5" class="ltx_emph ltx_font_italic">science</em> and <em id="id16.id2.1.6" class="ltx_emph ltx_font_italic">mechanistic interpretability</em> of LLMs. We have made the associated code and data publicly accessible to support future studies at
<a target="_blank" href="https://github.com/GAIR-NLP/ReAlign" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/GAIR-NLP/ReAlign</a>.

<br class="ltx_break">
<br class="ltx_break">
<br class="ltx_break">
<br class="ltx_break"></span></p>
</div>
<figure id="S0.F1" class="ltx_figure" lang="en">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S0.F1.sf1" class="ltx_figure ltx_figure_panel ltx_align_center">
<div id="S0.F1.sf1.1" class="ltx_block ltx_minipage ltx_align_bottom" style="width:208.1pt;">
<img src="/html/2402.12219/assets/x1.png" id="S0.F1.sf1.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="281" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>GSM8K</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S0.F1.sf2" class="ltx_figure ltx_figure_panel ltx_align_center">
<div id="S0.F1.sf2.1" class="ltx_block ltx_minipage ltx_align_bottom" style="width:208.1pt;">
<img src="/html/2402.12219/assets/x2.png" id="S0.F1.sf2.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="287" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>MATH (OOD Setting)</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>The accuracy of the GSM8K test set for LLaMA-2-13B and Mistral-7B models fine-tuned on the training set of GSM8K and MATH with and without <span id="S0.F1.2.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>. (a): Training and testing on GSM8K. (b): Training on MATH and testing on GSM8K (Out-of-Distribution Setting).</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="S1" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Alignment has been witnessed to be an effective technique for aligning large language models (LLMs) to human values and human intent <cite class="ltx_cite ltx_citemacro_cite">Ouyang et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite>, which usually requires fine-tuning on a large amount of synthetic data derived from LLMs <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib35" title="" class="ltx_ref">2023b</a>); Honovich et al. (<a href="#bib.bib10" title="" class="ltx_ref">2023</a>); Peng et al. (<a href="#bib.bib26" title="" class="ltx_ref">2023</a>); Xu et al. (<a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite> or human-annotated instruction data <cite class="ltx_cite ltx_citemacro_cite">Ouyang et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>); Köpf et al. (<a href="#bib.bib14" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Recent studies, notably by <cite class="ltx_cite ltx_citemacro_citet">Zhou et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite> highlight the critical role of instruction data quality in this process.
Numerous works <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib35" title="" class="ltx_ref">2023b</a>); Zhou et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>); Cao et al. (<a href="#bib.bib2" title="" class="ltx_ref">2023</a>); Chen et al. (<a href="#bib.bib3" title="" class="ltx_ref">2023</a>); Li et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023a</a>); Lu et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite> have contributed to enhancing instruction quality by focusing on the diversity and complexity of input queries as well as the quality of responses. These efforts can be divided into two primary approaches.
The first approach, advocated by <cite class="ltx_cite ltx_citemacro_citet">Ouyang et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Touvron et al. (<a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite>, involves the manual creation of high-quality data. Although this method creates complex queries and factually correct and highly readable responses, it is labor-intensive and challenging to scale.
The second approach revolves around the automated extraction of high-quality instructions from existing datasets due to their extensive availability <cite class="ltx_cite ltx_citemacro_cite">Cao et al. (<a href="#bib.bib2" title="" class="ltx_ref">2023</a>); Chen et al. (<a href="#bib.bib3" title="" class="ltx_ref">2023</a>); Li et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023a</a>); Lu et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite>.
However, this method inherits the limitations associated with distilled data, such as containing factually incorrect content <cite class="ltx_cite ltx_citemacro_citep">(Ji et al., <a href="#bib.bib11" title="" class="ltx_ref">2023</a>; Gudibande et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite> and the format and style of the generated response are often determined by distilled LLMs’ preference.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this paper, instead of focusing on the creation of instruction data from scratch, we investigate how existing instruction data can be made higher quality and better aligned with human values.
We propose a simple and effective method, named <span id="S1.p3.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>, which is orthogonal to the above existing approaches.
Specifically, <span id="S1.p3.1.2" class="ltx_text ltx_font_smallcaps">ReAlign</span> necessitates a base instruction dataset, which can be sourced from extensive existing supervised datasets (e.g., <span id="S1.p3.1.3" class="ltx_text ltx_font_typewriter">GSM8K</span> <cite class="ltx_cite ltx_citemacro_cite">Cobbe et al. (<a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite>), or publicly available instruction data compiled through various methods (e.g., Self-Instruct <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib35" title="" class="ltx_ref">2023b</a>)</cite>, Evol-Instruct <cite class="ltx_cite ltx_citemacro_cite">Xu et al. (<a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite>, and Self-Alignment <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib18" title="" class="ltx_ref">2023b</a>)</cite>).
The <span id="S1.p3.1.4" class="ltx_text ltx_font_smallcaps">ReAlign</span> process unfolds in three main steps.
The first step involves <span id="S1.p3.1.5" class="ltx_text ltx_font_bold">criteria definition</span> (§<a href="#S3.SS1" title="3.1 Criteria Definition ‣ 3 ReAlign ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>), where humans define their preferences (e.g., the preferred format of responses) in various scenarios in the form of natural language.
In this paper, we meticulously define criteria for 46 distinct scenarios.
The second step, <span id="S1.p3.1.6" class="ltx_text ltx_font_bold">retrieval augmentation</span> (§<a href="#S3.SS2" title="3.2 Retrieval Augmentation ‣ 3 ReAlign ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>), broadens the knowledge base for knowledge-intensive tasks like open-domain QA and fact verification. This is achieved by incorporating additional information, thereby improving the factuality and informativeness of responses.
The final step, <span id="S1.p3.1.7" class="ltx_text ltx_font_bold">reformatting</span> (§<a href="#S3.SS3" title="3.3 Reformatting ‣ 3 ReAlign ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>), aims to re-align the responses with the pre-established criteria and the collated evidence, guaranteeing outputs that are both structured and substantiated.
As demonstrated in Fig. <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, the realigned response provides a better format and a clearer chain of thoughts.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">The underlying <em id="S1.p4.1.1" class="ltx_emph ltx_font_italic">philosophy</em> of <span id="S1.p4.1.2" class="ltx_text ltx_font_smallcaps">ReAlign</span> is to re-coordinate the roles of humans and LLMs in the alignment process, leveraging their complementary strengths – humans articulate their preferences, and LLMs, in turn, reconstruct instructions based on their generative power (e.g., instruction-following ability), without directly using distilled LLM knowledge.
Through this collaborative synergy, we expect the generated instruction data to be not only more contextually precise but also more closely aligned with human preferences.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">We operationalize this idea on five types of existing instruction data, where three are general datasets (i.e., <span id="S1.p5.1.1" class="ltx_text ltx_font_typewriter">Open-Platypus</span> <cite class="ltx_cite ltx_citemacro_cite">Lee et al. (<a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>, <span id="S1.p5.1.2" class="ltx_text ltx_font_typewriter">No Robots</span> <cite class="ltx_cite ltx_citemacro_cite">Rajani et al. (<a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite>, and <span id="S1.p5.1.3" class="ltx_text ltx_font_typewriter">Alpaca</span> <cite class="ltx_cite ltx_citemacro_cite">Taori et al. (<a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite>) and two are mathematical datasets (i.e., <span id="S1.p5.1.4" class="ltx_text ltx_font_typewriter">GSM8K</span> <cite class="ltx_cite ltx_citemacro_cite">Cobbe et al. (<a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite> and <span id="S1.p5.1.5" class="ltx_text ltx_font_typewriter">MATH</span> <cite class="ltx_cite ltx_citemacro_cite">Hendrycks et al. (<a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite>).
The performance of <span id="S1.p5.1.6" class="ltx_text ltx_font_smallcaps">ReAlign</span> has been validated across various well-established benchmarks, including AlpacaEval <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib19" title="" class="ltx_ref">2023c</a>)</cite>, MT-Bench <cite class="ltx_cite ltx_citemacro_cite">Zheng et al. (<a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite>, and Vicuna-Bench <cite class="ltx_cite ltx_citemacro_cite">Chiang et al. (<a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite> for general alignment, as well as <span id="S1.p5.1.7" class="ltx_text ltx_font_typewriter">GSM8K</span> and <span id="S1.p5.1.8" class="ltx_text ltx_font_typewriter">MATH</span> for mathematical reasoning. Additionally, it has also been evaluated for factuality and readability, demonstrating its proficiency.
In particular, <span id="S1.p5.1.9" class="ltx_text ltx_font_smallcaps">ReAlign</span> significantly boosts math reasoning, even up to 9.86% on <span id="S1.p5.1.10" class="ltx_text ltx_font_typewriter">GSM8K</span> for LLaMA-2-13B.
Notably, we find that only 5% of the <span id="S1.p5.1.11" class="ltx_text ltx_font_smallcaps">ReAlign</span> data yields a 67% boost in general alignment ability compared to the full <span id="S1.p5.1.12" class="ltx_text ltx_font_smallcaps">ReAlign</span> data based on the <span id="S1.p5.1.13" class="ltx_text ltx_font_typewriter">Alpaca</span> dataset, indicating that only a small amount of data is required to learn style and format.</p>
</div>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2402.12219/assets/x3.png" id="S1.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="368" height="286" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span><span id="S1.F2.3.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> realigns the original response with the pre-defined criteria to be a better format. The original response is from the <span id="S1.F2.4.2" class="ltx_text ltx_font_typewriter">Open-Platypus</span> <cite class="ltx_cite ltx_citemacro_cite">Lee et al. (<a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite> dataset. The complete version is shown in Tab. <a href="#A5.T13" title="Table 13 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>.</figcaption>
</figure>
</section>
<section id="S2" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Instruction Creation</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">Creating instructional data significantly improves LLMs’ alignment abilities.
High-quality instruction generation traditionally depends on human annotation for tasks like query writing, response drafting, and preference indication. This approach produces premium open-source datasets (e.g., <span id="S2.SS1.p1.1.1" class="ltx_text ltx_font_typewriter">Open-Platypus</span> <cite class="ltx_cite ltx_citemacro_cite">Lee et al. (<a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite> and <span id="S2.SS1.p1.1.2" class="ltx_text ltx_font_typewriter">OpenAssistant</span> <cite class="ltx_cite ltx_citemacro_cite">Köpf et al. (<a href="#bib.bib14" title="" class="ltx_ref">2023</a>)</cite>) and supports advanced LLMs (e.g., LIMA <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite> and LLaMA-2 <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite>), but it’s hard to scale due to high labor costs and the need for domain-specific expertise.
Many studies have explored using LLMs (e.g., GPT-3 <cite class="ltx_cite ltx_citemacro_cite">Brown et al. (<a href="#bib.bib1" title="" class="ltx_ref">2020</a>)</cite> and GPT-4 <cite class="ltx_cite ltx_citemacro_cite">OpenAI (<a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>) to generate instruction data. Techniques like unnatural instructions <cite class="ltx_cite ltx_citemacro_cite">Honovich et al. (<a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite> and self-instruct <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib35" title="" class="ltx_ref">2023b</a>)</cite> utilize GPT-3’s in-context learning with seed data to generate instructions, while evol-instruct <cite class="ltx_cite ltx_citemacro_cite">Xu et al. (<a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite> generates more complex and varied instructions through ChatGPT. Recently, training with self-generated data has yielded excellent results, achieving self-alignment <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib18" title="" class="ltx_ref">2023b</a>); Yuan et al. (<a href="#bib.bib40" title="" class="ltx_ref">2024</a>)</cite>. While it can be easily scaled up, this approach inherits the drawbacks of LLMs (e.g., factual errors) <cite class="ltx_cite ltx_citemacro_cite">Gudibande et al. (<a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite>.
Our proposed method contrasts by offering automatic data quality improvement with minimal labor and reduced factual errors.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Instruction Selection</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">After the discovery of “quality is all you need” <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>); Touvron et al. (<a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite>, instruction selection has been paid attention to, aiming at selecting a small number of the highest-quality samples from a large amount of instruction data as a training dataset.
<cite class="ltx_cite ltx_citemacro_citet">Cao et al. (<a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite> evaluates the dataset’s quality by utilizing the evaluation dataset loss to fit the natural language indicators of the dataset.
<cite class="ltx_cite ltx_citemacro_citet">Chen et al. (<a href="#bib.bib3" title="" class="ltx_ref">2023</a>)</cite> proposes to use ChatGPT directly to score the data, while <cite class="ltx_cite ltx_citemacro_citet">Li et al. (<a href="#bib.bib18" title="" class="ltx_ref">2023b</a>)</cite> proposes to score the data using the trained model directly to save costs.
<cite class="ltx_cite ltx_citemacro_citet">Lu et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite> proposes to tag samples within SFT datasets based on semantics and intentions and define instruction diversity and complexity regarding tags to rank data.
<cite class="ltx_cite ltx_citemacro_citet">Li et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023a</a>)</cite> introduces a self-guided approach that utilizes a new indicator, Instruction-Following Difficulty (IDF), to score data by identifying gaps in a model’s responses versus its autonomous generation capability.
<cite class="ltx_cite ltx_citemacro_citet">Liu et al. (<a href="#bib.bib20" title="" class="ltx_ref">2023</a>)</cite> trains two scorers to evaluate the complexity of the instruction and the quality of the response, respectively, and then uses the embedding distance to determine the diversity to select high-quality data.
However, the above works usually mine from distilled datasets because the large scale of distilled datasets is available, thereby inheriting the drawbacks of distilled data and suffering from the hallucination of LLMs.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Instruction Tuning</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">Instruction tuning aims to reinforce the model’s instruction-following capabilities and align LLMs to human values.
Early instruction tuning was designed to improve cross-task generalization capabilities, in which they usually scale up the quantity and the diversity of tasks <cite class="ltx_cite ltx_citemacro_cite">Mishra et al. (<a href="#bib.bib22" title="" class="ltx_ref">2022</a>); Wei et al. (<a href="#bib.bib37" title="" class="ltx_ref">2022a</a>); Sanh et al. (<a href="#bib.bib29" title="" class="ltx_ref">2022</a>); Wang et al. (<a href="#bib.bib36" title="" class="ltx_ref">2022</a>)</cite>. Recent works no longer explicitly define tasks, but extend to more generalized capabilities, especially for scenarios of real-world questions <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib35" title="" class="ltx_ref">2023b</a>); Honovich et al. (<a href="#bib.bib10" title="" class="ltx_ref">2023</a>); Peng et al. (<a href="#bib.bib26" title="" class="ltx_ref">2023</a>); Xu et al. (<a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite>.
Differently, our work utilizes the future of the task to design a better format for it, which further improves the quality of the data.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<table id="S2.T1.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S2.T1.1.1" class="ltx_tr">
<td id="S2.T1.1.1.1" class="ltx_td ltx_align_left ltx_border_tt"><span id="S2.T1.1.1.1.1" class="ltx_text ltx_font_bold">Group</span></td>
<td id="S2.T1.1.1.2" class="ltx_td ltx_align_left ltx_border_tt"><span id="S2.T1.1.1.2.1" class="ltx_text ltx_font_bold">Tasks</span></td>
</tr>
<tr id="S2.T1.1.2" class="ltx_tr">
<td id="S2.T1.1.2.1" class="ltx_td ltx_align_left ltx_border_t">Generation</td>
<td id="S2.T1.1.2.2" class="ltx_td ltx_align_left ltx_border_t">
<span id="S2.T1.1.2.2.1" class="ltx_text"></span><span id="S2.T1.1.2.2.2" class="ltx_text">
<span id="S2.T1.1.2.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.2.2.2.1.1" class="ltx_tr">
<span id="S2.T1.1.2.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">question generation; story generation; poem generation; email generation; data generation; text-to-text translation</span></span>
</span></span><span id="S2.T1.1.2.2.3" class="ltx_text"></span></td>
</tr>
<tr id="S2.T1.1.3" class="ltx_tr">
<td id="S2.T1.1.3.1" class="ltx_td ltx_align_left ltx_border_t">Brainstorming</td>
<td id="S2.T1.1.3.2" class="ltx_td ltx_align_left ltx_border_t">
<span id="S2.T1.1.3.2.1" class="ltx_text"></span><span id="S2.T1.1.3.2.2" class="ltx_text">
<span id="S2.T1.1.3.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.3.2.2.1.1" class="ltx_tr">
<span id="S2.T1.1.3.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">advice giving; recommendations; how-to generation; planning</span></span>
</span></span><span id="S2.T1.1.3.2.3" class="ltx_text"></span></td>
</tr>
<tr id="S2.T1.1.4" class="ltx_tr">
<td id="S2.T1.1.4.1" class="ltx_td ltx_align_left ltx_border_t">Code</td>
<td id="S2.T1.1.4.2" class="ltx_td ltx_align_left ltx_border_t">
<span id="S2.T1.1.4.2.1" class="ltx_text"></span><span id="S2.T1.1.4.2.2" class="ltx_text">
<span id="S2.T1.1.4.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.4.2.2.1.1" class="ltx_tr">
<span id="S2.T1.1.4.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">code correction; code simplification; explain code; text-to-code translation; code-to-code translation;</span></span>
<span id="S2.T1.1.4.2.2.1.2" class="ltx_tr">
<span id="S2.T1.1.4.2.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left">language learning questions; code language classification; code-to-text-translation</span></span>
</span></span><span id="S2.T1.1.4.2.3" class="ltx_text"></span></td>
</tr>
<tr id="S2.T1.1.5" class="ltx_tr">
<td id="S2.T1.1.5.1" class="ltx_td ltx_align_left ltx_border_t">Rewriting</td>
<td id="S2.T1.1.5.2" class="ltx_td ltx_align_left ltx_border_t">
<span id="S2.T1.1.5.2.1" class="ltx_text"></span><span id="S2.T1.1.5.2.2" class="ltx_text">
<span id="S2.T1.1.5.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.5.2.2.1.1" class="ltx_tr">
<span id="S2.T1.1.5.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">instructional rewriting; language polishing; paraphrasing; text correction</span></span>
</span></span><span id="S2.T1.1.5.2.3" class="ltx_text"></span></td>
</tr>
<tr id="S2.T1.1.6" class="ltx_tr">
<td id="S2.T1.1.6.1" class="ltx_td ltx_align_left ltx_border_t">Extraction</td>
<td id="S2.T1.1.6.2" class="ltx_td ltx_align_left ltx_border_t">
<span id="S2.T1.1.6.2.1" class="ltx_text"></span><span id="S2.T1.1.6.2.2" class="ltx_text">
<span id="S2.T1.1.6.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.6.2.2.1.1" class="ltx_tr">
<span id="S2.T1.1.6.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">information extraction; keywords extraction; table extraction</span></span>
</span></span><span id="S2.T1.1.6.2.3" class="ltx_text"></span></td>
</tr>
<tr id="S2.T1.1.7" class="ltx_tr">
<td id="S2.T1.1.7.1" class="ltx_td ltx_align_left ltx_border_t">Summarization</td>
<td id="S2.T1.1.7.2" class="ltx_td ltx_align_left ltx_border_t">title generation; text summarization; note summarization</td>
</tr>
<tr id="S2.T1.1.8" class="ltx_tr">
<td id="S2.T1.1.8.1" class="ltx_td ltx_align_left ltx_border_t">Conversation</td>
<td id="S2.T1.1.8.2" class="ltx_td ltx_align_left ltx_border_t">
<span id="S2.T1.1.8.2.1" class="ltx_text"></span><span id="S2.T1.1.8.2.2" class="ltx_text">
<span id="S2.T1.1.8.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.8.2.2.1.1" class="ltx_tr">
<span id="S2.T1.1.8.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">open qa; closed qa; fact verification; value judgment; roleplay; explain answer</span></span>
</span></span><span id="S2.T1.1.8.2.3" class="ltx_text"></span></td>
</tr>
<tr id="S2.T1.1.9" class="ltx_tr">
<td id="S2.T1.1.9.1" class="ltx_td ltx_align_left ltx_border_t">Education.</td>
<td id="S2.T1.1.9.2" class="ltx_td ltx_align_left ltx_border_t">
<span id="S2.T1.1.9.2.1" class="ltx_text"></span><span id="S2.T1.1.9.2.2" class="ltx_text">
<span id="S2.T1.1.9.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.9.2.2.1.1" class="ltx_tr">
<span id="S2.T1.1.9.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">natural language tutor; exam problem tutor; ai tutor; math puzzles; fill in the blank</span></span>
</span></span><span id="S2.T1.1.9.2.3" class="ltx_text"></span></td>
</tr>
<tr id="S2.T1.1.10" class="ltx_tr">
<td id="S2.T1.1.10.1" class="ltx_td ltx_align_left ltx_border_t">Classification</td>
<td id="S2.T1.1.10.2" class="ltx_td ltx_align_left ltx_border_t">
<span id="S2.T1.1.10.2.1" class="ltx_text"></span><span id="S2.T1.1.10.2.2" class="ltx_text">
<span id="S2.T1.1.10.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.10.2.2.1.1" class="ltx_tr">
<span id="S2.T1.1.10.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">general classification; ordering; sentiment analysis; language classification; topic classification</span></span>
</span></span><span id="S2.T1.1.10.2.3" class="ltx_text"></span></td>
</tr>
<tr id="S2.T1.1.11" class="ltx_tr">
<td id="S2.T1.1.11.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">Others</td>
<td id="S2.T1.1.11.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">rejecting; others</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>The category of tasks. “Education.” denotes Specialized Educational Dialog.</figcaption>
</figure>
<figure id="S2.T2" class="ltx_table">
<table id="S2.T2.9" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S2.T2.9.10" class="ltx_tr">
<td id="S2.T2.9.10.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S2.T2.9.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T2.9.10.1.1.1" class="ltx_p" style="width:368.6pt;"><span id="S2.T2.9.10.1.1.1.1" class="ltx_text ltx_font_bold">Email Generation</span></span>
</span>
</td>
</tr>
<tr id="S2.T2.9.11" class="ltx_tr">
<td id="S2.T2.9.11.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="S2.T2.9.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T2.9.11.1.1.1" class="ltx_p" style="width:368.6pt;"><span id="S2.T2.9.11.1.1.1.1" class="ltx_text"></span><span id="S2.T2.9.11.1.1.1.2" class="ltx_text">
<span id="S2.T2.9.11.1.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T2.9.11.1.1.1.2.1.1" class="ltx_tr">
<span id="S2.T2.9.11.1.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">It is an email-writing task. Here is a general guideline for creating a well-structured and professional email:</span></span>
</span></span></span>
<span id="S2.T2.9.11.1.1.2" class="ltx_p"></span>
</span>
</td>
</tr>
<tr id="S2.T2.9.9" class="ltx_tr">
<td id="S2.T2.9.9.9" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top"><span id="S2.T2.9.9.9.9" class="ltx_inline-logical-block ltx_align_top">
<span id="S2.T2.6.6.6.6.p6" class="ltx_para">
<span id="S2.T2.6.6.6.6.p6.1" class="ltx_p" style="width:368.6pt;"><span id="S2.T2.6.6.6.6.p6.1.1" class="ltx_text"></span><span id="S2.T2.6.6.6.6.p6.1.2" class="ltx_text">
<span id="S2.T2.6.6.6.6.p6.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T2.6.6.6.6.p6.1.2.1.1" class="ltx_tr">
<span id="S2.T2.6.6.6.6.p6.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S2.T2.6.6.6.6.p6.1.2.1.1.1.1" class="ltx_text" style="color:#000000;background-color:#DAE3F3;">1. Subject Line:</span> Write a clear and concise subject line that accurately summarizes the content of your email …</span></span>
</span></span> <span id="S2.T2.6.6.6.6.p6.1.3" class="ltx_text"></span></span>
</span>
<span id="S2.T2.1.1.1.1.p1" class="ltx_para ltx_noindent">
<span id="S2.T2.1.1.1.1.p1.1" class="ltx_p"><span id="S2.T2.1.1.1.1.p1.1.1" class="ltx_text"></span><span id="S2.T2.1.1.1.1.p1.1.2" class="ltx_text">
<span id="S2.T2.1.1.1.1.p1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T2.1.1.1.1.p1.1.2.1.1" class="ltx_tr">
<span id="S2.T2.1.1.1.1.p1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S2.T2.1.1.1.1.p1.1.2.1.1.1.1" class="ltx_text" style="color:#000000;background-color:#DAE3F3;">2. Salutation:</span> Begin your email with a formal salutation such as ”Dear [Recipient’s Name],” …</span></span>
</span></span> <span id="S2.T2.1.1.1.1.p1.1.3" class="ltx_text"></span></span>
</span>
<span id="S2.T2.7.7.7.7.p7" class="ltx_para">
<span id="S2.T2.7.7.7.7.p7.1" class="ltx_p"><span id="S2.T2.7.7.7.7.p7.1.1" class="ltx_text" style="color:#000000;background-color:#DAE3F3;">3. Introduction:</span> Start your email with a brief introduction …</span>
<span id="S2.T2.7.7.7.7.p7.2" class="ltx_p"><span id="S2.T2.7.7.7.7.p7.2.1" class="ltx_text" style="color:#000000;background-color:#DAE3F3;">4. Body:</span> This is the main content of your email …</span>
</span>
<span id="S2.T2.2.2.2.2.p2" class="ltx_para ltx_noindent">
<span id="S2.T2.2.2.2.2.p2.1" class="ltx_p"><span id="S2.T2.2.2.2.2.p2.1.1" class="ltx_text"></span><span id="S2.T2.2.2.2.2.p2.1.2" class="ltx_text">
<span id="S2.T2.2.2.2.2.p2.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T2.2.2.2.2.p2.1.2.1.1" class="ltx_tr">
<span id="S2.T2.2.2.2.2.p2.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S2.T2.2.2.2.2.p2.1.2.1.1.1.1" class="ltx_text" style="color:#000000;background-color:#DAE3F3;">5. Politeness and Tone:</span> Maintain a polite and respectful tone throughout your email …</span></span>
</span></span> <span id="S2.T2.2.2.2.2.p2.1.3" class="ltx_text"></span></span>
</span>
<span id="S2.T2.3.3.3.3.p3" class="ltx_para ltx_noindent">
<span id="S2.T2.3.3.3.3.p3.1" class="ltx_p"><span id="S2.T2.3.3.3.3.p3.1.1" class="ltx_text"></span><span id="S2.T2.3.3.3.3.p3.1.2" class="ltx_text">
<span id="S2.T2.3.3.3.3.p3.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T2.3.3.3.3.p3.1.2.1.1" class="ltx_tr">
<span id="S2.T2.3.3.3.3.p3.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S2.T2.3.3.3.3.p3.1.2.1.1.1.1" class="ltx_text" style="color:#000000;background-color:#DAE3F3;">6. Closing:</span> Conclude your email with a closing remark, such as ”Thank you,” or ”Best regards,” followed by your name …</span></span>
</span></span> <span id="S2.T2.3.3.3.3.p3.1.3" class="ltx_text"></span></span>
</span>
<span id="S2.T2.4.4.4.4.p4" class="ltx_para ltx_noindent">
<span id="S2.T2.4.4.4.4.p4.1" class="ltx_p"><span id="S2.T2.4.4.4.4.p4.1.1" class="ltx_text"></span><span id="S2.T2.4.4.4.4.p4.1.2" class="ltx_text">
<span id="S2.T2.4.4.4.4.p4.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T2.4.4.4.4.p4.1.2.1.1" class="ltx_tr">
<span id="S2.T2.4.4.4.4.p4.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S2.T2.4.4.4.4.p4.1.2.1.1.1.1" class="ltx_text" style="color:#000000;background-color:#DAE3F3;">7. Signature:</span> Include your full name, job title, and contact information (e.g., phone number, email address) …</span></span>
</span></span> <span id="S2.T2.4.4.4.4.p4.1.3" class="ltx_text"></span></span>
</span>
<span id="S2.T2.8.8.8.8.p8" class="ltx_para">
<span id="S2.T2.8.8.8.8.p8.1" class="ltx_p"><span id="S2.T2.8.8.8.8.p8.1.1" class="ltx_text" style="color:#000000;background-color:#DAE3F3;">8. Attachments:</span> If you need to include attachments, mention them …</span>
</span>
<span id="S2.T2.5.5.5.5.p5" class="ltx_para ltx_noindent">
<span id="S2.T2.5.5.5.5.p5.1" class="ltx_p"><span id="S2.T2.5.5.5.5.p5.1.1" class="ltx_text"></span><span id="S2.T2.5.5.5.5.p5.1.2" class="ltx_text">
<span id="S2.T2.5.5.5.5.p5.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T2.5.5.5.5.p5.1.2.1.1" class="ltx_tr">
<span id="S2.T2.5.5.5.5.p5.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S2.T2.5.5.5.5.p5.1.2.1.1.1.1" class="ltx_text" style="color:#000000;background-color:#DAE3F3;">9. Proofread:</span> Before sending the email, proofread it for any grammatical or spelling errors …</span></span>
</span></span></span>
</span>
<span id="S2.T2.9.9.9.9.p9" class="ltx_para">
<span id="S2.T2.9.9.9.9.p9.1" class="ltx_p"></span>
</span></span></td>
</tr>
<tr id="S2.T2.9.12" class="ltx_tr">
<td id="S2.T2.9.12.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S2.T2.9.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T2.9.12.1.1.1" class="ltx_p" style="width:368.6pt;"><span id="S2.T2.9.12.1.1.1.1" class="ltx_text"></span><span id="S2.T2.9.12.1.1.1.2" class="ltx_text">
<span id="S2.T2.9.12.1.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T2.9.12.1.1.1.2.1.1" class="ltx_tr">
<span id="S2.T2.9.12.1.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">The best emails are short, direct, professional, and scannable for the recipient. Follow a formal business email structure unless you have</span></span>
<span id="S2.T2.9.12.1.1.1.2.1.2" class="ltx_tr">
<span id="S2.T2.9.12.1.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left">an established casual rapport with the recipient.</span></span>
</span></span><span id="S2.T2.9.12.1.1.1.3" class="ltx_text"></span></span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>An example of the format for the “email generation” task.</figcaption>
</figure>
<figure id="S2.F3" class="ltx_figure"><img src="/html/2402.12219/assets/x4.png" id="S2.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="196" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>An overview of our <span id="S2.F3.2.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> including three steps. KILT denotes Knowledge Intensive Language Tasks.</figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span><span id="S3.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.3" class="ltx_p">Given a base instruction dataset <math id="S3.p1.1.m1.3" class="ltx_Math" alttext="\mathcal{D}=\{(q_{1},r_{1}),\cdots,(q_{n},r_{n})\}" display="inline"><semantics id="S3.p1.1.m1.3a"><mrow id="S3.p1.1.m1.3.3" xref="S3.p1.1.m1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.1.m1.3.3.4" xref="S3.p1.1.m1.3.3.4.cmml">𝒟</mi><mo id="S3.p1.1.m1.3.3.3" xref="S3.p1.1.m1.3.3.3.cmml">=</mo><mrow id="S3.p1.1.m1.3.3.2.2" xref="S3.p1.1.m1.3.3.2.3.cmml"><mo stretchy="false" id="S3.p1.1.m1.3.3.2.2.3" xref="S3.p1.1.m1.3.3.2.3.cmml">{</mo><mrow id="S3.p1.1.m1.2.2.1.1.1.2" xref="S3.p1.1.m1.2.2.1.1.1.3.cmml"><mo stretchy="false" id="S3.p1.1.m1.2.2.1.1.1.2.3" xref="S3.p1.1.m1.2.2.1.1.1.3.cmml">(</mo><msub id="S3.p1.1.m1.2.2.1.1.1.1.1" xref="S3.p1.1.m1.2.2.1.1.1.1.1.cmml"><mi id="S3.p1.1.m1.2.2.1.1.1.1.1.2" xref="S3.p1.1.m1.2.2.1.1.1.1.1.2.cmml">q</mi><mn id="S3.p1.1.m1.2.2.1.1.1.1.1.3" xref="S3.p1.1.m1.2.2.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S3.p1.1.m1.2.2.1.1.1.2.4" xref="S3.p1.1.m1.2.2.1.1.1.3.cmml">,</mo><msub id="S3.p1.1.m1.2.2.1.1.1.2.2" xref="S3.p1.1.m1.2.2.1.1.1.2.2.cmml"><mi id="S3.p1.1.m1.2.2.1.1.1.2.2.2" xref="S3.p1.1.m1.2.2.1.1.1.2.2.2.cmml">r</mi><mn id="S3.p1.1.m1.2.2.1.1.1.2.2.3" xref="S3.p1.1.m1.2.2.1.1.1.2.2.3.cmml">1</mn></msub><mo stretchy="false" id="S3.p1.1.m1.2.2.1.1.1.2.5" xref="S3.p1.1.m1.2.2.1.1.1.3.cmml">)</mo></mrow><mo id="S3.p1.1.m1.3.3.2.2.4" xref="S3.p1.1.m1.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">⋯</mi><mo id="S3.p1.1.m1.3.3.2.2.5" xref="S3.p1.1.m1.3.3.2.3.cmml">,</mo><mrow id="S3.p1.1.m1.3.3.2.2.2.2" xref="S3.p1.1.m1.3.3.2.2.2.3.cmml"><mo stretchy="false" id="S3.p1.1.m1.3.3.2.2.2.2.3" xref="S3.p1.1.m1.3.3.2.2.2.3.cmml">(</mo><msub id="S3.p1.1.m1.3.3.2.2.2.1.1" xref="S3.p1.1.m1.3.3.2.2.2.1.1.cmml"><mi id="S3.p1.1.m1.3.3.2.2.2.1.1.2" xref="S3.p1.1.m1.3.3.2.2.2.1.1.2.cmml">q</mi><mi id="S3.p1.1.m1.3.3.2.2.2.1.1.3" xref="S3.p1.1.m1.3.3.2.2.2.1.1.3.cmml">n</mi></msub><mo id="S3.p1.1.m1.3.3.2.2.2.2.4" xref="S3.p1.1.m1.3.3.2.2.2.3.cmml">,</mo><msub id="S3.p1.1.m1.3.3.2.2.2.2.2" xref="S3.p1.1.m1.3.3.2.2.2.2.2.cmml"><mi id="S3.p1.1.m1.3.3.2.2.2.2.2.2" xref="S3.p1.1.m1.3.3.2.2.2.2.2.2.cmml">r</mi><mi id="S3.p1.1.m1.3.3.2.2.2.2.2.3" xref="S3.p1.1.m1.3.3.2.2.2.2.2.3.cmml">n</mi></msub><mo stretchy="false" id="S3.p1.1.m1.3.3.2.2.2.2.5" xref="S3.p1.1.m1.3.3.2.2.2.3.cmml">)</mo></mrow><mo stretchy="false" id="S3.p1.1.m1.3.3.2.2.6" xref="S3.p1.1.m1.3.3.2.3.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.3b"><apply id="S3.p1.1.m1.3.3.cmml" xref="S3.p1.1.m1.3.3"><eq id="S3.p1.1.m1.3.3.3.cmml" xref="S3.p1.1.m1.3.3.3"></eq><ci id="S3.p1.1.m1.3.3.4.cmml" xref="S3.p1.1.m1.3.3.4">𝒟</ci><set id="S3.p1.1.m1.3.3.2.3.cmml" xref="S3.p1.1.m1.3.3.2.2"><interval closure="open" id="S3.p1.1.m1.2.2.1.1.1.3.cmml" xref="S3.p1.1.m1.2.2.1.1.1.2"><apply id="S3.p1.1.m1.2.2.1.1.1.1.1.cmml" xref="S3.p1.1.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.p1.1.m1.2.2.1.1.1.1.1.1.cmml" xref="S3.p1.1.m1.2.2.1.1.1.1.1">subscript</csymbol><ci id="S3.p1.1.m1.2.2.1.1.1.1.1.2.cmml" xref="S3.p1.1.m1.2.2.1.1.1.1.1.2">𝑞</ci><cn type="integer" id="S3.p1.1.m1.2.2.1.1.1.1.1.3.cmml" xref="S3.p1.1.m1.2.2.1.1.1.1.1.3">1</cn></apply><apply id="S3.p1.1.m1.2.2.1.1.1.2.2.cmml" xref="S3.p1.1.m1.2.2.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.p1.1.m1.2.2.1.1.1.2.2.1.cmml" xref="S3.p1.1.m1.2.2.1.1.1.2.2">subscript</csymbol><ci id="S3.p1.1.m1.2.2.1.1.1.2.2.2.cmml" xref="S3.p1.1.m1.2.2.1.1.1.2.2.2">𝑟</ci><cn type="integer" id="S3.p1.1.m1.2.2.1.1.1.2.2.3.cmml" xref="S3.p1.1.m1.2.2.1.1.1.2.2.3">1</cn></apply></interval><ci id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">⋯</ci><interval closure="open" id="S3.p1.1.m1.3.3.2.2.2.3.cmml" xref="S3.p1.1.m1.3.3.2.2.2.2"><apply id="S3.p1.1.m1.3.3.2.2.2.1.1.cmml" xref="S3.p1.1.m1.3.3.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.p1.1.m1.3.3.2.2.2.1.1.1.cmml" xref="S3.p1.1.m1.3.3.2.2.2.1.1">subscript</csymbol><ci id="S3.p1.1.m1.3.3.2.2.2.1.1.2.cmml" xref="S3.p1.1.m1.3.3.2.2.2.1.1.2">𝑞</ci><ci id="S3.p1.1.m1.3.3.2.2.2.1.1.3.cmml" xref="S3.p1.1.m1.3.3.2.2.2.1.1.3">𝑛</ci></apply><apply id="S3.p1.1.m1.3.3.2.2.2.2.2.cmml" xref="S3.p1.1.m1.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.p1.1.m1.3.3.2.2.2.2.2.1.cmml" xref="S3.p1.1.m1.3.3.2.2.2.2.2">subscript</csymbol><ci id="S3.p1.1.m1.3.3.2.2.2.2.2.2.cmml" xref="S3.p1.1.m1.3.3.2.2.2.2.2.2">𝑟</ci><ci id="S3.p1.1.m1.3.3.2.2.2.2.2.3.cmml" xref="S3.p1.1.m1.3.3.2.2.2.2.2.3">𝑛</ci></apply></interval></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.3c">\mathcal{D}=\{(q_{1},r_{1}),\cdots,(q_{n},r_{n})\}</annotation></semantics></math>, where <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="q" display="inline"><semantics id="S3.p1.2.m2.1a"><mi id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><ci id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">q</annotation></semantics></math> and <math id="S3.p1.3.m3.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.p1.3.m3.1a"><mi id="S3.p1.3.m3.1.1" xref="S3.p1.3.m3.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.p1.3.m3.1b"><ci id="S3.p1.3.m3.1.1.cmml" xref="S3.p1.3.m3.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m3.1c">r</annotation></semantics></math> are the input query and response respectively, <span id="S3.p1.3.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> aims to improve the quality of responses by three steps as shown in Fig. <a href="#S2.F3" title="Figure 3 ‣ 2.3 Instruction Tuning ‣ 2 Related Work ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>:
(1) Criteria Definition: defining the criteria including tasks and formats for each task, (2) Retrieval Augmentation: retrieving relevant external information for the knowledge-intensive tasks, and (3) Reformatting: reformatting the original response based on the guidance consisting of hand-written format and the retrieved information. An overview of our method is shown in Fig. <a href="#S2.F3" title="Figure 3 ‣ 2.3 Instruction Tuning ‣ 2 Related Work ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Criteria Definition</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">The predefined criteria consist of the tasks and the corresponding formats:</p>
</div>
<section id="S3.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Tasks.</h5>

<div id="S3.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px1.p1.2" class="ltx_p">Clearly defining tasks is crucial to subsequently devising tailored formats, as the optimal format varies across distinct tasks.
In this paper, we follow <cite class="ltx_cite ltx_citemacro_citet">Li et al. (<a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite> to define 46 different tasks <math id="S3.SS1.SSS0.Px1.p1.1.m1.3" class="ltx_Math" alttext="\{T_{1},\cdots,T_{N=46}\}" display="inline"><semantics id="S3.SS1.SSS0.Px1.p1.1.m1.3a"><mrow id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.3.cmml"><mo stretchy="false" id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.3" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.3.cmml">{</mo><msub id="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1" xref="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.cmml"><mi id="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.2" xref="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.2.cmml">T</mi><mn id="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.3" xref="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.4" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS1.SSS0.Px1.p1.1.m1.1.1.cmml">⋯</mi><mo id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.5" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.3.cmml">,</mo><msub id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.cmml"><mi id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.2" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.2.cmml">T</mi><mrow id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.cmml"><mi id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.2" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.2.cmml">N</mi><mo id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.1" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.1.cmml">=</mo><mn id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.3" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.3.cmml">46</mn></mrow></msub><mo stretchy="false" id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.6" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.3.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px1.p1.1.m1.3b"><set id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.3.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2"><apply id="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.1.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1">subscript</csymbol><ci id="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.2.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.2">𝑇</ci><cn type="integer" id="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.3.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.2.2.1.1.3">1</cn></apply><ci id="S3.SS1.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.1.1">⋯</ci><apply id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.1.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2">subscript</csymbol><ci id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.2.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.2">𝑇</ci><apply id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3"><eq id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.1.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.1"></eq><ci id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.2.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.2">𝑁</ci><cn type="integer" id="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.3.cmml" xref="S3.SS1.SSS0.Px1.p1.1.m1.3.3.2.2.3.3">46</cn></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px1.p1.1.m1.3c">\{T_{1},\cdots,T_{N=46}\}</annotation></semantics></math>, categorized into 10 major groups, as shown in Tab. <a href="#S2.T1" title="Table 1 ‣ 2.3 Instruction Tuning ‣ 2 Related Work ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. The detailed description for each task is shown in Tab. <a href="#A4.T8" title="Table 8 ‣ Appendix D The Description of Auto-J ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>, §<a href="#A2" title="Appendix B Task Description ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>. We also train a task classifier <math id="S3.SS1.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="\mathtt{C}" display="inline"><semantics id="S3.SS1.SSS0.Px1.p1.2.m2.1a"><mi id="S3.SS1.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS1.SSS0.Px1.p1.2.m2.1.1.cmml">𝙲</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px1.p1.2.m2.1b"><ci id="S3.SS1.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS1.SSS0.Px1.p1.2.m2.1.1">𝙲</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px1.p1.2.m2.1c">\mathtt{C}</annotation></semantics></math>, detailed in §<a href="#A3" title="Appendix C Training Details of Task Classifier ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Format.</h5>

<div id="S3.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px2.p1.1" class="ltx_p">Due to the distinct formatting requisites associated with diverse tasks, we meticulously devised tailored formats <math id="S3.SS1.SSS0.Px2.p1.1.m1.3" class="ltx_Math" alttext="\{F_{1},\cdots,F_{N=46}\}" display="inline"><semantics id="S3.SS1.SSS0.Px2.p1.1.m1.3a"><mrow id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.3.cmml"><mo stretchy="false" id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.3" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.3.cmml">{</mo><msub id="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1" xref="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.cmml"><mi id="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.2" xref="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.2.cmml">F</mi><mn id="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.3" xref="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.4" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS1.SSS0.Px2.p1.1.m1.1.1.cmml">⋯</mi><mo id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.5" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.3.cmml">,</mo><msub id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.cmml"><mi id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.2" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.2.cmml">F</mi><mrow id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.cmml"><mi id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.2" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.2.cmml">N</mi><mo id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.1" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.1.cmml">=</mo><mn id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.3" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.3.cmml">46</mn></mrow></msub><mo stretchy="false" id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.6" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.3.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px2.p1.1.m1.3b"><set id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.3.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2"><apply id="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.1.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1">subscript</csymbol><ci id="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.2.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.2">𝐹</ci><cn type="integer" id="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.3.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.2.2.1.1.3">1</cn></apply><ci id="S3.SS1.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.1.1">⋯</ci><apply id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.1.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2">subscript</csymbol><ci id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.2.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.2">𝐹</ci><apply id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3"><eq id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.1.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.1"></eq><ci id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.2.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.2">𝑁</ci><cn type="integer" id="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.3.cmml" xref="S3.SS1.SSS0.Px2.p1.1.m1.3.3.2.2.3.3">46</cn></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px2.p1.1.m1.3c">\{F_{1},\cdots,F_{N=46}\}</annotation></semantics></math> for each task based on the task definition and description, encompassing considerations such as organizational structure, section content, and output modality.
This format, which has been carefully designed for different tasks, is more readable than the generic format.
Each format has a task name and a detailed format description. We show an example of a format for “email generation” in Tab. <a href="#S2.T2" title="Table 2 ‣ 2.3 Instruction Tuning ‣ 2 Related Work ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (The complete version is shown in Tab. <a href="#A5.T14" title="Table 14 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>).</p>
</div>
<div id="S3.SS1.SSS0.Px2.p2" class="ltx_para">
<p id="S3.SS1.SSS0.Px2.p2.3" class="ltx_p">In this step, we input query <math id="S3.SS1.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S3.SS1.SSS0.Px2.p2.1.m1.1a"><msub id="S3.SS1.SSS0.Px2.p2.1.m1.1.1" xref="S3.SS1.SSS0.Px2.p2.1.m1.1.1.cmml"><mi id="S3.SS1.SSS0.Px2.p2.1.m1.1.1.2" xref="S3.SS1.SSS0.Px2.p2.1.m1.1.1.2.cmml">q</mi><mi id="S3.SS1.SSS0.Px2.p2.1.m1.1.1.3" xref="S3.SS1.SSS0.Px2.p2.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px2.p2.1.m1.1b"><apply id="S3.SS1.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S3.SS1.SSS0.Px2.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS0.Px2.p2.1.m1.1.1.1.cmml" xref="S3.SS1.SSS0.Px2.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.SSS0.Px2.p2.1.m1.1.1.2.cmml" xref="S3.SS1.SSS0.Px2.p2.1.m1.1.1.2">𝑞</ci><ci id="S3.SS1.SSS0.Px2.p2.1.m1.1.1.3.cmml" xref="S3.SS1.SSS0.Px2.p2.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px2.p2.1.m1.1c">q_{i}</annotation></semantics></math> to the task classifier <math id="S3.SS1.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="\mathtt{C}" display="inline"><semantics id="S3.SS1.SSS0.Px2.p2.2.m2.1a"><mi id="S3.SS1.SSS0.Px2.p2.2.m2.1.1" xref="S3.SS1.SSS0.Px2.p2.2.m2.1.1.cmml">𝙲</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px2.p2.2.m2.1b"><ci id="S3.SS1.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S3.SS1.SSS0.Px2.p2.2.m2.1.1">𝙲</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px2.p2.2.m2.1c">\mathtt{C}</annotation></semantics></math> (detailed in §<a href="#A3" title="Appendix C Training Details of Task Classifier ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>) to acquire the category <math id="S3.SS1.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="t_{i}" display="inline"><semantics id="S3.SS1.SSS0.Px2.p2.3.m3.1a"><msub id="S3.SS1.SSS0.Px2.p2.3.m3.1.1" xref="S3.SS1.SSS0.Px2.p2.3.m3.1.1.cmml"><mi id="S3.SS1.SSS0.Px2.p2.3.m3.1.1.2" xref="S3.SS1.SSS0.Px2.p2.3.m3.1.1.2.cmml">t</mi><mi id="S3.SS1.SSS0.Px2.p2.3.m3.1.1.3" xref="S3.SS1.SSS0.Px2.p2.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px2.p2.3.m3.1b"><apply id="S3.SS1.SSS0.Px2.p2.3.m3.1.1.cmml" xref="S3.SS1.SSS0.Px2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS0.Px2.p2.3.m3.1.1.1.cmml" xref="S3.SS1.SSS0.Px2.p2.3.m3.1.1">subscript</csymbol><ci id="S3.SS1.SSS0.Px2.p2.3.m3.1.1.2.cmml" xref="S3.SS1.SSS0.Px2.p2.3.m3.1.1.2">𝑡</ci><ci id="S3.SS1.SSS0.Px2.p2.3.m3.1.1.3.cmml" xref="S3.SS1.SSS0.Px2.p2.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px2.p2.3.m3.1c">t_{i}</annotation></semantics></math>:</p>
<table id="S3.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.Ex1.m1.1" class="ltx_Math" alttext="t_{i}=\mathtt{C}(q_{i})," display="block"><semantics id="S3.Ex1.m1.1a"><mrow id="S3.Ex1.m1.1.1.1" xref="S3.Ex1.m1.1.1.1.1.cmml"><mrow id="S3.Ex1.m1.1.1.1.1" xref="S3.Ex1.m1.1.1.1.1.cmml"><msub id="S3.Ex1.m1.1.1.1.1.3" xref="S3.Ex1.m1.1.1.1.1.3.cmml"><mi id="S3.Ex1.m1.1.1.1.1.3.2" xref="S3.Ex1.m1.1.1.1.1.3.2.cmml">t</mi><mi id="S3.Ex1.m1.1.1.1.1.3.3" xref="S3.Ex1.m1.1.1.1.1.3.3.cmml">i</mi></msub><mo id="S3.Ex1.m1.1.1.1.1.2" xref="S3.Ex1.m1.1.1.1.1.2.cmml">=</mo><mrow id="S3.Ex1.m1.1.1.1.1.1" xref="S3.Ex1.m1.1.1.1.1.1.cmml"><mi id="S3.Ex1.m1.1.1.1.1.1.3" xref="S3.Ex1.m1.1.1.1.1.1.3.cmml">𝙲</mi><mo lspace="0em" rspace="0em" id="S3.Ex1.m1.1.1.1.1.1.2" xref="S3.Ex1.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.Ex1.m1.1.1.1.1.1.1.1" xref="S3.Ex1.m1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.Ex1.m1.1.1.1.1.1.1.1.2" xref="S3.Ex1.m1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.Ex1.m1.1.1.1.1.1.1.1.1" xref="S3.Ex1.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.Ex1.m1.1.1.1.1.1.1.1.1.2" xref="S3.Ex1.m1.1.1.1.1.1.1.1.1.2.cmml">q</mi><mi id="S3.Ex1.m1.1.1.1.1.1.1.1.1.3" xref="S3.Ex1.m1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.Ex1.m1.1.1.1.1.1.1.1.3" xref="S3.Ex1.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.Ex1.m1.1.1.1.2" xref="S3.Ex1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex1.m1.1b"><apply id="S3.Ex1.m1.1.1.1.1.cmml" xref="S3.Ex1.m1.1.1.1"><eq id="S3.Ex1.m1.1.1.1.1.2.cmml" xref="S3.Ex1.m1.1.1.1.1.2"></eq><apply id="S3.Ex1.m1.1.1.1.1.3.cmml" xref="S3.Ex1.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.Ex1.m1.1.1.1.1.3.1.cmml" xref="S3.Ex1.m1.1.1.1.1.3">subscript</csymbol><ci id="S3.Ex1.m1.1.1.1.1.3.2.cmml" xref="S3.Ex1.m1.1.1.1.1.3.2">𝑡</ci><ci id="S3.Ex1.m1.1.1.1.1.3.3.cmml" xref="S3.Ex1.m1.1.1.1.1.3.3">𝑖</ci></apply><apply id="S3.Ex1.m1.1.1.1.1.1.cmml" xref="S3.Ex1.m1.1.1.1.1.1"><times id="S3.Ex1.m1.1.1.1.1.1.2.cmml" xref="S3.Ex1.m1.1.1.1.1.1.2"></times><ci id="S3.Ex1.m1.1.1.1.1.1.3.cmml" xref="S3.Ex1.m1.1.1.1.1.1.3">𝙲</ci><apply id="S3.Ex1.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex1.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.Ex1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex1.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.Ex1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex1.m1.1.1.1.1.1.1.1.1.2">𝑞</ci><ci id="S3.Ex1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex1.m1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex1.m1.1c">t_{i}=\mathtt{C}(q_{i}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS1.SSS0.Px2.p2.4" class="ltx_p">and then obtain the corresponding format <math id="S3.SS1.SSS0.Px2.p2.4.m1.1" class="ltx_Math" alttext="f_{i}" display="inline"><semantics id="S3.SS1.SSS0.Px2.p2.4.m1.1a"><msub id="S3.SS1.SSS0.Px2.p2.4.m1.1.1" xref="S3.SS1.SSS0.Px2.p2.4.m1.1.1.cmml"><mi id="S3.SS1.SSS0.Px2.p2.4.m1.1.1.2" xref="S3.SS1.SSS0.Px2.p2.4.m1.1.1.2.cmml">f</mi><mi id="S3.SS1.SSS0.Px2.p2.4.m1.1.1.3" xref="S3.SS1.SSS0.Px2.p2.4.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px2.p2.4.m1.1b"><apply id="S3.SS1.SSS0.Px2.p2.4.m1.1.1.cmml" xref="S3.SS1.SSS0.Px2.p2.4.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS0.Px2.p2.4.m1.1.1.1.cmml" xref="S3.SS1.SSS0.Px2.p2.4.m1.1.1">subscript</csymbol><ci id="S3.SS1.SSS0.Px2.p2.4.m1.1.1.2.cmml" xref="S3.SS1.SSS0.Px2.p2.4.m1.1.1.2">𝑓</ci><ci id="S3.SS1.SSS0.Px2.p2.4.m1.1.1.3.cmml" xref="S3.SS1.SSS0.Px2.p2.4.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px2.p2.4.m1.1c">f_{i}</annotation></semantics></math>.</p>
</div>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Retrieval Augmentation</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.4" class="ltx_p">Knowledge-intensive language tasks (KILT), such as open-domain QA and fact verification, usually require large and external knowledge sources as the evidence to ensure the factuality <cite class="ltx_cite ltx_citemacro_cite">Petroni et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite>.
Thus, we follow <cite class="ltx_cite ltx_citemacro_citet">Petroni et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite> to choose five knowledge-intensive tasks and use the query <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><msub id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml"><mi id="S3.SS2.p1.1.m1.1.1.2" xref="S3.SS2.p1.1.m1.1.1.2.cmml">q</mi><mi id="S3.SS2.p1.1.m1.1.1.3" xref="S3.SS2.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><apply id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.p1.1.m1.1.1.2">𝑞</ci><ci id="S3.SS2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">q_{i}</annotation></semantics></math> to retrieve relevant information as our evidence. The tasks for retrieval augmentation are shown in Tab. <a href="#A4.T8" title="Table 8 ‣ Appendix D The Description of Auto-J ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>. Specifically, we follow <cite class="ltx_cite ltx_citemacro_citet">Chern et al. (<a href="#bib.bib4" title="" class="ltx_ref">2023</a>)</cite> and use the Google Search API as our retriever <math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="\mathtt{R}" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><mi id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">𝚁</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><ci id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">𝚁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">\mathtt{R}</annotation></semantics></math> provided by Serper<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span><a target="_blank" href="https://serper.dev/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://serper.dev/</a></span></span></span> to retrieve the most relevant search snippets included in the API’s answer. We then parse the response to obtain different types of snippets such as answer boxes, knowledge graphs, and organic search results. Finally, we choose the top-<math id="S3.SS2.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS2.p1.3.m3.1a"><mi id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.1b"><ci id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.1c">k</annotation></semantics></math> snippets and filter them as our evidence <math id="S3.SS2.p1.4.m4.3" class="ltx_Math" alttext="E_{i}={e_{i1},\cdots,e_{ik}}" display="inline"><semantics id="S3.SS2.p1.4.m4.3a"><mrow id="S3.SS2.p1.4.m4.3.3" xref="S3.SS2.p1.4.m4.3.3.cmml"><msub id="S3.SS2.p1.4.m4.3.3.4" xref="S3.SS2.p1.4.m4.3.3.4.cmml"><mi id="S3.SS2.p1.4.m4.3.3.4.2" xref="S3.SS2.p1.4.m4.3.3.4.2.cmml">E</mi><mi id="S3.SS2.p1.4.m4.3.3.4.3" xref="S3.SS2.p1.4.m4.3.3.4.3.cmml">i</mi></msub><mo id="S3.SS2.p1.4.m4.3.3.3" xref="S3.SS2.p1.4.m4.3.3.3.cmml">=</mo><mrow id="S3.SS2.p1.4.m4.3.3.2.2" xref="S3.SS2.p1.4.m4.3.3.2.3.cmml"><msub id="S3.SS2.p1.4.m4.2.2.1.1.1" xref="S3.SS2.p1.4.m4.2.2.1.1.1.cmml"><mi id="S3.SS2.p1.4.m4.2.2.1.1.1.2" xref="S3.SS2.p1.4.m4.2.2.1.1.1.2.cmml">e</mi><mrow id="S3.SS2.p1.4.m4.2.2.1.1.1.3" xref="S3.SS2.p1.4.m4.2.2.1.1.1.3.cmml"><mi id="S3.SS2.p1.4.m4.2.2.1.1.1.3.2" xref="S3.SS2.p1.4.m4.2.2.1.1.1.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.2.2.1.1.1.3.1" xref="S3.SS2.p1.4.m4.2.2.1.1.1.3.1.cmml">​</mo><mn id="S3.SS2.p1.4.m4.2.2.1.1.1.3.3" xref="S3.SS2.p1.4.m4.2.2.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S3.SS2.p1.4.m4.3.3.2.2.3" xref="S3.SS2.p1.4.m4.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.p1.4.m4.1.1" xref="S3.SS2.p1.4.m4.1.1.cmml">⋯</mi><mo id="S3.SS2.p1.4.m4.3.3.2.2.4" xref="S3.SS2.p1.4.m4.3.3.2.3.cmml">,</mo><msub id="S3.SS2.p1.4.m4.3.3.2.2.2" xref="S3.SS2.p1.4.m4.3.3.2.2.2.cmml"><mi id="S3.SS2.p1.4.m4.3.3.2.2.2.2" xref="S3.SS2.p1.4.m4.3.3.2.2.2.2.cmml">e</mi><mrow id="S3.SS2.p1.4.m4.3.3.2.2.2.3" xref="S3.SS2.p1.4.m4.3.3.2.2.2.3.cmml"><mi id="S3.SS2.p1.4.m4.3.3.2.2.2.3.2" xref="S3.SS2.p1.4.m4.3.3.2.2.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.3.3.2.2.2.3.1" xref="S3.SS2.p1.4.m4.3.3.2.2.2.3.1.cmml">​</mo><mi id="S3.SS2.p1.4.m4.3.3.2.2.2.3.3" xref="S3.SS2.p1.4.m4.3.3.2.2.2.3.3.cmml">k</mi></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m4.3b"><apply id="S3.SS2.p1.4.m4.3.3.cmml" xref="S3.SS2.p1.4.m4.3.3"><eq id="S3.SS2.p1.4.m4.3.3.3.cmml" xref="S3.SS2.p1.4.m4.3.3.3"></eq><apply id="S3.SS2.p1.4.m4.3.3.4.cmml" xref="S3.SS2.p1.4.m4.3.3.4"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.3.3.4.1.cmml" xref="S3.SS2.p1.4.m4.3.3.4">subscript</csymbol><ci id="S3.SS2.p1.4.m4.3.3.4.2.cmml" xref="S3.SS2.p1.4.m4.3.3.4.2">𝐸</ci><ci id="S3.SS2.p1.4.m4.3.3.4.3.cmml" xref="S3.SS2.p1.4.m4.3.3.4.3">𝑖</ci></apply><list id="S3.SS2.p1.4.m4.3.3.2.3.cmml" xref="S3.SS2.p1.4.m4.3.3.2.2"><apply id="S3.SS2.p1.4.m4.2.2.1.1.1.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.2.2.1.1.1.1.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1">subscript</csymbol><ci id="S3.SS2.p1.4.m4.2.2.1.1.1.2.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1.2">𝑒</ci><apply id="S3.SS2.p1.4.m4.2.2.1.1.1.3.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1.3"><times id="S3.SS2.p1.4.m4.2.2.1.1.1.3.1.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1.3.1"></times><ci id="S3.SS2.p1.4.m4.2.2.1.1.1.3.2.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.SS2.p1.4.m4.2.2.1.1.1.3.3.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1.3.3">1</cn></apply></apply><ci id="S3.SS2.p1.4.m4.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1">⋯</ci><apply id="S3.SS2.p1.4.m4.3.3.2.2.2.cmml" xref="S3.SS2.p1.4.m4.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.3.3.2.2.2.1.cmml" xref="S3.SS2.p1.4.m4.3.3.2.2.2">subscript</csymbol><ci id="S3.SS2.p1.4.m4.3.3.2.2.2.2.cmml" xref="S3.SS2.p1.4.m4.3.3.2.2.2.2">𝑒</ci><apply id="S3.SS2.p1.4.m4.3.3.2.2.2.3.cmml" xref="S3.SS2.p1.4.m4.3.3.2.2.2.3"><times id="S3.SS2.p1.4.m4.3.3.2.2.2.3.1.cmml" xref="S3.SS2.p1.4.m4.3.3.2.2.2.3.1"></times><ci id="S3.SS2.p1.4.m4.3.3.2.2.2.3.2.cmml" xref="S3.SS2.p1.4.m4.3.3.2.2.2.3.2">𝑖</ci><ci id="S3.SS2.p1.4.m4.3.3.2.2.2.3.3.cmml" xref="S3.SS2.p1.4.m4.3.3.2.2.2.3.3">𝑘</ci></apply></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m4.3c">E_{i}={e_{i1},\cdots,e_{ik}}</annotation></semantics></math>:</p>
<table id="S3.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.Ex2.m1.1" class="ltx_Math" alttext="E_{i}=\mathtt{R}(q_{i})." display="block"><semantics id="S3.Ex2.m1.1a"><mrow id="S3.Ex2.m1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.cmml"><mrow id="S3.Ex2.m1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.cmml"><msub id="S3.Ex2.m1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.3.2" xref="S3.Ex2.m1.1.1.1.1.3.2.cmml">E</mi><mi id="S3.Ex2.m1.1.1.1.1.3.3" xref="S3.Ex2.m1.1.1.1.1.3.3.cmml">i</mi></msub><mo id="S3.Ex2.m1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.2.cmml">=</mo><mrow id="S3.Ex2.m1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.1.3.cmml">𝚁</mi><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.Ex2.m1.1.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.Ex2.m1.1.1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.Ex2.m1.1.1.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.2.cmml">q</mi><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.Ex2.m1.1.1.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S3.Ex2.m1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex2.m1.1b"><apply id="S3.Ex2.m1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1"><eq id="S3.Ex2.m1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2"></eq><apply id="S3.Ex2.m1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.3">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.3.2">𝐸</ci><ci id="S3.Ex2.m1.1.1.1.1.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.3.3">𝑖</ci></apply><apply id="S3.Ex2.m1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1"><times id="S3.Ex2.m1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.2"></times><ci id="S3.Ex2.m1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.3">𝚁</ci><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.2">𝑞</ci><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex2.m1.1c">E_{i}=\mathtt{R}(q_{i}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS2.p1.5" class="ltx_p">We show an example of a knowledge-intensive language task in Tab. <a href="#A5.T15" title="Table 15 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>, demonstrating that retrieval augmentation enables the response more factual and informative.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Reformatting</h3>

<section id="S3.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.3.1 </span>Rewriting</h4>

<div id="S3.SS3.SSS1.p1" class="ltx_para">
<p id="S3.SS3.SSS1.p1.3" class="ltx_p">In this step, we leverage large language models (e.g., ChatGPT) to rewrite the response <math id="S3.SS3.SSS1.p1.1.m1.1" class="ltx_Math" alttext="r_{i}" display="inline"><semantics id="S3.SS3.SSS1.p1.1.m1.1a"><msub id="S3.SS3.SSS1.p1.1.m1.1.1" xref="S3.SS3.SSS1.p1.1.m1.1.1.cmml"><mi id="S3.SS3.SSS1.p1.1.m1.1.1.2" xref="S3.SS3.SSS1.p1.1.m1.1.1.2.cmml">r</mi><mi id="S3.SS3.SSS1.p1.1.m1.1.1.3" xref="S3.SS3.SSS1.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.p1.1.m1.1b"><apply id="S3.SS3.SSS1.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS1.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS1.p1.1.m1.1.1.2">𝑟</ci><ci id="S3.SS3.SSS1.p1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS1.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.p1.1.m1.1c">r_{i}</annotation></semantics></math> based on the given format <math id="S3.SS3.SSS1.p1.2.m2.1" class="ltx_Math" alttext="f_{i}" display="inline"><semantics id="S3.SS3.SSS1.p1.2.m2.1a"><msub id="S3.SS3.SSS1.p1.2.m2.1.1" xref="S3.SS3.SSS1.p1.2.m2.1.1.cmml"><mi id="S3.SS3.SSS1.p1.2.m2.1.1.2" xref="S3.SS3.SSS1.p1.2.m2.1.1.2.cmml">f</mi><mi id="S3.SS3.SSS1.p1.2.m2.1.1.3" xref="S3.SS3.SSS1.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.p1.2.m2.1b"><apply id="S3.SS3.SSS1.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.p1.2.m2.1.1.1.cmml" xref="S3.SS3.SSS1.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.p1.2.m2.1.1.2.cmml" xref="S3.SS3.SSS1.p1.2.m2.1.1.2">𝑓</ci><ci id="S3.SS3.SSS1.p1.2.m2.1.1.3.cmml" xref="S3.SS3.SSS1.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.p1.2.m2.1c">f_{i}</annotation></semantics></math> and retrieved evidence <math id="S3.SS3.SSS1.p1.3.m3.1" class="ltx_Math" alttext="E_{i}" display="inline"><semantics id="S3.SS3.SSS1.p1.3.m3.1a"><msub id="S3.SS3.SSS1.p1.3.m3.1.1" xref="S3.SS3.SSS1.p1.3.m3.1.1.cmml"><mi id="S3.SS3.SSS1.p1.3.m3.1.1.2" xref="S3.SS3.SSS1.p1.3.m3.1.1.2.cmml">E</mi><mi id="S3.SS3.SSS1.p1.3.m3.1.1.3" xref="S3.SS3.SSS1.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.p1.3.m3.1b"><apply id="S3.SS3.SSS1.p1.3.m3.1.1.cmml" xref="S3.SS3.SSS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.p1.3.m3.1.1.1.cmml" xref="S3.SS3.SSS1.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.p1.3.m3.1.1.2.cmml" xref="S3.SS3.SSS1.p1.3.m3.1.1.2">𝐸</ci><ci id="S3.SS3.SSS1.p1.3.m3.1.1.3.cmml" xref="S3.SS3.SSS1.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.p1.3.m3.1c">E_{i}</annotation></semantics></math> (for knowledge-intensive tasks).
Since certain queries have additional requirements (e.g., specific formatting or specified information), an adaptive rewriting strategy is employed.
This approach involves initially using LLMs to determine whether the format matches the query requirements.
Subsequently, if it matches, the LLMs rewrite the response accordingly.
We divide the tasks into two categories:</p>
</div>
<section id="S3.SS3.SSS1.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Non-knowledge-intensive tasks</h5>

<div id="S3.SS3.SSS1.Px1.p1" class="ltx_para">
<p id="S3.SS3.SSS1.Px1.p1.4" class="ltx_p">For the non-knowledge-intensive tasks, we decide to rewrite a part of the tasks. This decision stems from the observation that certain tasks are not amenable to a standardized format, exemplified by instances such as story generation and poem generation (see Tab. <a href="#A4.T8" title="Table 8 ‣ Appendix D The Description of Auto-J ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> for details). We guide LLMs to rewrite the original responses <math id="S3.SS3.SSS1.Px1.p1.1.m1.1" class="ltx_Math" alttext="r_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px1.p1.1.m1.1a"><msub id="S3.SS3.SSS1.Px1.p1.1.m1.1.1" xref="S3.SS3.SSS1.Px1.p1.1.m1.1.1.cmml"><mi id="S3.SS3.SSS1.Px1.p1.1.m1.1.1.2" xref="S3.SS3.SSS1.Px1.p1.1.m1.1.1.2.cmml">r</mi><mi id="S3.SS3.SSS1.Px1.p1.1.m1.1.1.3" xref="S3.SS3.SSS1.Px1.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px1.p1.1.m1.1b"><apply id="S3.SS3.SSS1.Px1.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS1.Px1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS1.Px1.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS1.Px1.p1.1.m1.1.1.2">𝑟</ci><ci id="S3.SS3.SSS1.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS1.Px1.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px1.p1.1.m1.1c">r_{i}</annotation></semantics></math>, organizing the query <math id="S3.SS3.SSS1.Px1.p1.2.m2.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px1.p1.2.m2.1a"><msub id="S3.SS3.SSS1.Px1.p1.2.m2.1.1" xref="S3.SS3.SSS1.Px1.p1.2.m2.1.1.cmml"><mi id="S3.SS3.SSS1.Px1.p1.2.m2.1.1.2" xref="S3.SS3.SSS1.Px1.p1.2.m2.1.1.2.cmml">q</mi><mi id="S3.SS3.SSS1.Px1.p1.2.m2.1.1.3" xref="S3.SS3.SSS1.Px1.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px1.p1.2.m2.1b"><apply id="S3.SS3.SSS1.Px1.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS1.Px1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px1.p1.2.m2.1.1.1.cmml" xref="S3.SS3.SSS1.Px1.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.Px1.p1.2.m2.1.1.2.cmml" xref="S3.SS3.SSS1.Px1.p1.2.m2.1.1.2">𝑞</ci><ci id="S3.SS3.SSS1.Px1.p1.2.m2.1.1.3.cmml" xref="S3.SS3.SSS1.Px1.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px1.p1.2.m2.1c">q_{i}</annotation></semantics></math>, original response <math id="S3.SS3.SSS1.Px1.p1.3.m3.1" class="ltx_Math" alttext="r_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px1.p1.3.m3.1a"><msub id="S3.SS3.SSS1.Px1.p1.3.m3.1.1" xref="S3.SS3.SSS1.Px1.p1.3.m3.1.1.cmml"><mi id="S3.SS3.SSS1.Px1.p1.3.m3.1.1.2" xref="S3.SS3.SSS1.Px1.p1.3.m3.1.1.2.cmml">r</mi><mi id="S3.SS3.SSS1.Px1.p1.3.m3.1.1.3" xref="S3.SS3.SSS1.Px1.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px1.p1.3.m3.1b"><apply id="S3.SS3.SSS1.Px1.p1.3.m3.1.1.cmml" xref="S3.SS3.SSS1.Px1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px1.p1.3.m3.1.1.1.cmml" xref="S3.SS3.SSS1.Px1.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.Px1.p1.3.m3.1.1.2.cmml" xref="S3.SS3.SSS1.Px1.p1.3.m3.1.1.2">𝑟</ci><ci id="S3.SS3.SSS1.Px1.p1.3.m3.1.1.3.cmml" xref="S3.SS3.SSS1.Px1.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px1.p1.3.m3.1c">r_{i}</annotation></semantics></math>, and the format <math id="S3.SS3.SSS1.Px1.p1.4.m4.1" class="ltx_Math" alttext="f_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px1.p1.4.m4.1a"><msub id="S3.SS3.SSS1.Px1.p1.4.m4.1.1" xref="S3.SS3.SSS1.Px1.p1.4.m4.1.1.cmml"><mi id="S3.SS3.SSS1.Px1.p1.4.m4.1.1.2" xref="S3.SS3.SSS1.Px1.p1.4.m4.1.1.2.cmml">f</mi><mi id="S3.SS3.SSS1.Px1.p1.4.m4.1.1.3" xref="S3.SS3.SSS1.Px1.p1.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px1.p1.4.m4.1b"><apply id="S3.SS3.SSS1.Px1.p1.4.m4.1.1.cmml" xref="S3.SS3.SSS1.Px1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px1.p1.4.m4.1.1.1.cmml" xref="S3.SS3.SSS1.Px1.p1.4.m4.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.Px1.p1.4.m4.1.1.2.cmml" xref="S3.SS3.SSS1.Px1.p1.4.m4.1.1.2">𝑓</ci><ci id="S3.SS3.SSS1.Px1.p1.4.m4.1.1.3.cmml" xref="S3.SS3.SSS1.Px1.p1.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px1.p1.4.m4.1c">f_{i}</annotation></semantics></math> together via the prompt in Tab. <a href="#A5.T11" title="Table 11 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>:</p>
<table id="S3.Ex3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.Ex3.m1.1" class="ltx_Math" alttext="\hat{r}_{i}=\mathtt{LLM}(q_{i},r_{i},f_{i})," display="block"><semantics id="S3.Ex3.m1.1a"><mrow id="S3.Ex3.m1.1.1.1" xref="S3.Ex3.m1.1.1.1.1.cmml"><mrow id="S3.Ex3.m1.1.1.1.1" xref="S3.Ex3.m1.1.1.1.1.cmml"><msub id="S3.Ex3.m1.1.1.1.1.5" xref="S3.Ex3.m1.1.1.1.1.5.cmml"><mover accent="true" id="S3.Ex3.m1.1.1.1.1.5.2" xref="S3.Ex3.m1.1.1.1.1.5.2.cmml"><mi id="S3.Ex3.m1.1.1.1.1.5.2.2" xref="S3.Ex3.m1.1.1.1.1.5.2.2.cmml">r</mi><mo id="S3.Ex3.m1.1.1.1.1.5.2.1" xref="S3.Ex3.m1.1.1.1.1.5.2.1.cmml">^</mo></mover><mi id="S3.Ex3.m1.1.1.1.1.5.3" xref="S3.Ex3.m1.1.1.1.1.5.3.cmml">i</mi></msub><mo id="S3.Ex3.m1.1.1.1.1.4" xref="S3.Ex3.m1.1.1.1.1.4.cmml">=</mo><mrow id="S3.Ex3.m1.1.1.1.1.3" xref="S3.Ex3.m1.1.1.1.1.3.cmml"><mi id="S3.Ex3.m1.1.1.1.1.3.5" xref="S3.Ex3.m1.1.1.1.1.3.5.cmml">𝙻𝙻𝙼</mi><mo lspace="0em" rspace="0em" id="S3.Ex3.m1.1.1.1.1.3.4" xref="S3.Ex3.m1.1.1.1.1.3.4.cmml">​</mo><mrow id="S3.Ex3.m1.1.1.1.1.3.3.3" xref="S3.Ex3.m1.1.1.1.1.3.3.4.cmml"><mo stretchy="false" id="S3.Ex3.m1.1.1.1.1.3.3.3.4" xref="S3.Ex3.m1.1.1.1.1.3.3.4.cmml">(</mo><msub id="S3.Ex3.m1.1.1.1.1.1.1.1.1" xref="S3.Ex3.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.Ex3.m1.1.1.1.1.1.1.1.1.2" xref="S3.Ex3.m1.1.1.1.1.1.1.1.1.2.cmml">q</mi><mi id="S3.Ex3.m1.1.1.1.1.1.1.1.1.3" xref="S3.Ex3.m1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.Ex3.m1.1.1.1.1.3.3.3.5" xref="S3.Ex3.m1.1.1.1.1.3.3.4.cmml">,</mo><msub id="S3.Ex3.m1.1.1.1.1.2.2.2.2" xref="S3.Ex3.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S3.Ex3.m1.1.1.1.1.2.2.2.2.2" xref="S3.Ex3.m1.1.1.1.1.2.2.2.2.2.cmml">r</mi><mi id="S3.Ex3.m1.1.1.1.1.2.2.2.2.3" xref="S3.Ex3.m1.1.1.1.1.2.2.2.2.3.cmml">i</mi></msub><mo id="S3.Ex3.m1.1.1.1.1.3.3.3.6" xref="S3.Ex3.m1.1.1.1.1.3.3.4.cmml">,</mo><msub id="S3.Ex3.m1.1.1.1.1.3.3.3.3" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3.cmml"><mi id="S3.Ex3.m1.1.1.1.1.3.3.3.3.2" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3.2.cmml">f</mi><mi id="S3.Ex3.m1.1.1.1.1.3.3.3.3.3" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3.3.cmml">i</mi></msub><mo stretchy="false" id="S3.Ex3.m1.1.1.1.1.3.3.3.7" xref="S3.Ex3.m1.1.1.1.1.3.3.4.cmml">)</mo></mrow></mrow></mrow><mo id="S3.Ex3.m1.1.1.1.2" xref="S3.Ex3.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex3.m1.1b"><apply id="S3.Ex3.m1.1.1.1.1.cmml" xref="S3.Ex3.m1.1.1.1"><eq id="S3.Ex3.m1.1.1.1.1.4.cmml" xref="S3.Ex3.m1.1.1.1.1.4"></eq><apply id="S3.Ex3.m1.1.1.1.1.5.cmml" xref="S3.Ex3.m1.1.1.1.1.5"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.5.1.cmml" xref="S3.Ex3.m1.1.1.1.1.5">subscript</csymbol><apply id="S3.Ex3.m1.1.1.1.1.5.2.cmml" xref="S3.Ex3.m1.1.1.1.1.5.2"><ci id="S3.Ex3.m1.1.1.1.1.5.2.1.cmml" xref="S3.Ex3.m1.1.1.1.1.5.2.1">^</ci><ci id="S3.Ex3.m1.1.1.1.1.5.2.2.cmml" xref="S3.Ex3.m1.1.1.1.1.5.2.2">𝑟</ci></apply><ci id="S3.Ex3.m1.1.1.1.1.5.3.cmml" xref="S3.Ex3.m1.1.1.1.1.5.3">𝑖</ci></apply><apply id="S3.Ex3.m1.1.1.1.1.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3"><times id="S3.Ex3.m1.1.1.1.1.3.4.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4"></times><ci id="S3.Ex3.m1.1.1.1.1.3.5.cmml" xref="S3.Ex3.m1.1.1.1.1.3.5">𝙻𝙻𝙼</ci><vector id="S3.Ex3.m1.1.1.1.1.3.3.4.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3"><apply id="S3.Ex3.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex3.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex3.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex3.m1.1.1.1.1.1.1.1.1.2">𝑞</ci><ci id="S3.Ex3.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex3.m1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.Ex3.m1.1.1.1.1.2.2.2.2.cmml" xref="S3.Ex3.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.Ex3.m1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.Ex3.m1.1.1.1.1.2.2.2.2.2">𝑟</ci><ci id="S3.Ex3.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.Ex3.m1.1.1.1.1.2.2.2.2.3">𝑖</ci></apply><apply id="S3.Ex3.m1.1.1.1.1.3.3.3.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.3.3.3.3.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.3.3.3.3.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3.2">𝑓</ci><ci id="S3.Ex3.m1.1.1.1.1.3.3.3.3.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3.3">𝑖</ci></apply></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex3.m1.1c">\hat{r}_{i}=\mathtt{LLM}(q_{i},r_{i},f_{i}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS3.SSS1.Px1.p1.5" class="ltx_p">where <math id="S3.SS3.SSS1.Px1.p1.5.m1.1" class="ltx_Math" alttext="\hat{r}_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px1.p1.5.m1.1a"><msub id="S3.SS3.SSS1.Px1.p1.5.m1.1.1" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1.cmml"><mover accent="true" id="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2.cmml"><mi id="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2.2" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2.2.cmml">r</mi><mo id="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2.1" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2.1.cmml">^</mo></mover><mi id="S3.SS3.SSS1.Px1.p1.5.m1.1.1.3" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px1.p1.5.m1.1b"><apply id="S3.SS3.SSS1.Px1.p1.5.m1.1.1.cmml" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px1.p1.5.m1.1.1.1.cmml" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1">subscript</csymbol><apply id="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2.cmml" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2"><ci id="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2.1.cmml" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2.1">^</ci><ci id="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2.2.cmml" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1.2.2">𝑟</ci></apply><ci id="S3.SS3.SSS1.Px1.p1.5.m1.1.1.3.cmml" xref="S3.SS3.SSS1.Px1.p1.5.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px1.p1.5.m1.1c">\hat{r}_{i}</annotation></semantics></math> is the reformatted response.</p>
</div>
</section>
<section id="S3.SS3.SSS1.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Knowledge-intensive tasks.</h5>

<div id="S3.SS3.SSS1.Px2.p1" class="ltx_para">
<p id="S3.SS3.SSS1.Px2.p1.6" class="ltx_p">For the knowledge-intensive tasks, we additionally utilize the retrieved evidence <math id="S3.SS3.SSS1.Px2.p1.1.m1.1" class="ltx_Math" alttext="E_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px2.p1.1.m1.1a"><msub id="S3.SS3.SSS1.Px2.p1.1.m1.1.1" xref="S3.SS3.SSS1.Px2.p1.1.m1.1.1.cmml"><mi id="S3.SS3.SSS1.Px2.p1.1.m1.1.1.2" xref="S3.SS3.SSS1.Px2.p1.1.m1.1.1.2.cmml">E</mi><mi id="S3.SS3.SSS1.Px2.p1.1.m1.1.1.3" xref="S3.SS3.SSS1.Px2.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px2.p1.1.m1.1b"><apply id="S3.SS3.SSS1.Px2.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px2.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.Px2.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS1.Px2.p1.1.m1.1.1.2">𝐸</ci><ci id="S3.SS3.SSS1.Px2.p1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS1.Px2.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px2.p1.1.m1.1c">E_{i}</annotation></semantics></math> compared to non-knowledge-intensive tasks. Specifically, We guide LLM to rewrite the original response <math id="S3.SS3.SSS1.Px2.p1.2.m2.1" class="ltx_Math" alttext="r_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px2.p1.2.m2.1a"><msub id="S3.SS3.SSS1.Px2.p1.2.m2.1.1" xref="S3.SS3.SSS1.Px2.p1.2.m2.1.1.cmml"><mi id="S3.SS3.SSS1.Px2.p1.2.m2.1.1.2" xref="S3.SS3.SSS1.Px2.p1.2.m2.1.1.2.cmml">r</mi><mi id="S3.SS3.SSS1.Px2.p1.2.m2.1.1.3" xref="S3.SS3.SSS1.Px2.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px2.p1.2.m2.1b"><apply id="S3.SS3.SSS1.Px2.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px2.p1.2.m2.1.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.Px2.p1.2.m2.1.1.2.cmml" xref="S3.SS3.SSS1.Px2.p1.2.m2.1.1.2">𝑟</ci><ci id="S3.SS3.SSS1.Px2.p1.2.m2.1.1.3.cmml" xref="S3.SS3.SSS1.Px2.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px2.p1.2.m2.1c">r_{i}</annotation></semantics></math>, organizing the query <math id="S3.SS3.SSS1.Px2.p1.3.m3.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px2.p1.3.m3.1a"><msub id="S3.SS3.SSS1.Px2.p1.3.m3.1.1" xref="S3.SS3.SSS1.Px2.p1.3.m3.1.1.cmml"><mi id="S3.SS3.SSS1.Px2.p1.3.m3.1.1.2" xref="S3.SS3.SSS1.Px2.p1.3.m3.1.1.2.cmml">q</mi><mi id="S3.SS3.SSS1.Px2.p1.3.m3.1.1.3" xref="S3.SS3.SSS1.Px2.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px2.p1.3.m3.1b"><apply id="S3.SS3.SSS1.Px2.p1.3.m3.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px2.p1.3.m3.1.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.Px2.p1.3.m3.1.1.2.cmml" xref="S3.SS3.SSS1.Px2.p1.3.m3.1.1.2">𝑞</ci><ci id="S3.SS3.SSS1.Px2.p1.3.m3.1.1.3.cmml" xref="S3.SS3.SSS1.Px2.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px2.p1.3.m3.1c">q_{i}</annotation></semantics></math>, original response <math id="S3.SS3.SSS1.Px2.p1.4.m4.1" class="ltx_Math" alttext="r_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px2.p1.4.m4.1a"><msub id="S3.SS3.SSS1.Px2.p1.4.m4.1.1" xref="S3.SS3.SSS1.Px2.p1.4.m4.1.1.cmml"><mi id="S3.SS3.SSS1.Px2.p1.4.m4.1.1.2" xref="S3.SS3.SSS1.Px2.p1.4.m4.1.1.2.cmml">r</mi><mi id="S3.SS3.SSS1.Px2.p1.4.m4.1.1.3" xref="S3.SS3.SSS1.Px2.p1.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px2.p1.4.m4.1b"><apply id="S3.SS3.SSS1.Px2.p1.4.m4.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px2.p1.4.m4.1.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.4.m4.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.Px2.p1.4.m4.1.1.2.cmml" xref="S3.SS3.SSS1.Px2.p1.4.m4.1.1.2">𝑟</ci><ci id="S3.SS3.SSS1.Px2.p1.4.m4.1.1.3.cmml" xref="S3.SS3.SSS1.Px2.p1.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px2.p1.4.m4.1c">r_{i}</annotation></semantics></math>, format <math id="S3.SS3.SSS1.Px2.p1.5.m5.1" class="ltx_Math" alttext="f_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px2.p1.5.m5.1a"><msub id="S3.SS3.SSS1.Px2.p1.5.m5.1.1" xref="S3.SS3.SSS1.Px2.p1.5.m5.1.1.cmml"><mi id="S3.SS3.SSS1.Px2.p1.5.m5.1.1.2" xref="S3.SS3.SSS1.Px2.p1.5.m5.1.1.2.cmml">f</mi><mi id="S3.SS3.SSS1.Px2.p1.5.m5.1.1.3" xref="S3.SS3.SSS1.Px2.p1.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px2.p1.5.m5.1b"><apply id="S3.SS3.SSS1.Px2.p1.5.m5.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px2.p1.5.m5.1.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.5.m5.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.Px2.p1.5.m5.1.1.2.cmml" xref="S3.SS3.SSS1.Px2.p1.5.m5.1.1.2">𝑓</ci><ci id="S3.SS3.SSS1.Px2.p1.5.m5.1.1.3.cmml" xref="S3.SS3.SSS1.Px2.p1.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px2.p1.5.m5.1c">f_{i}</annotation></semantics></math>, and the retrieved evidence <math id="S3.SS3.SSS1.Px2.p1.6.m6.1" class="ltx_Math" alttext="E_{i}" display="inline"><semantics id="S3.SS3.SSS1.Px2.p1.6.m6.1a"><msub id="S3.SS3.SSS1.Px2.p1.6.m6.1.1" xref="S3.SS3.SSS1.Px2.p1.6.m6.1.1.cmml"><mi id="S3.SS3.SSS1.Px2.p1.6.m6.1.1.2" xref="S3.SS3.SSS1.Px2.p1.6.m6.1.1.2.cmml">E</mi><mi id="S3.SS3.SSS1.Px2.p1.6.m6.1.1.3" xref="S3.SS3.SSS1.Px2.p1.6.m6.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.Px2.p1.6.m6.1b"><apply id="S3.SS3.SSS1.Px2.p1.6.m6.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS1.Px2.p1.6.m6.1.1.1.cmml" xref="S3.SS3.SSS1.Px2.p1.6.m6.1.1">subscript</csymbol><ci id="S3.SS3.SSS1.Px2.p1.6.m6.1.1.2.cmml" xref="S3.SS3.SSS1.Px2.p1.6.m6.1.1.2">𝐸</ci><ci id="S3.SS3.SSS1.Px2.p1.6.m6.1.1.3.cmml" xref="S3.SS3.SSS1.Px2.p1.6.m6.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.Px2.p1.6.m6.1c">E_{i}</annotation></semantics></math> together via the prompt in Tab. <a href="#A5.T12" title="Table 12 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>:</p>
</div>
<div id="S3.SS3.SSS1.Px2.p2" class="ltx_para">
<table id="S3.Ex4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.Ex4.m1.1" class="ltx_Math" alttext="\hat{r}_{i}=\mathtt{LLM}(q_{i},r_{i},f_{i},E_{i})." display="block"><semantics id="S3.Ex4.m1.1a"><mrow id="S3.Ex4.m1.1.1.1" xref="S3.Ex4.m1.1.1.1.1.cmml"><mrow id="S3.Ex4.m1.1.1.1.1" xref="S3.Ex4.m1.1.1.1.1.cmml"><msub id="S3.Ex4.m1.1.1.1.1.6" xref="S3.Ex4.m1.1.1.1.1.6.cmml"><mover accent="true" id="S3.Ex4.m1.1.1.1.1.6.2" xref="S3.Ex4.m1.1.1.1.1.6.2.cmml"><mi id="S3.Ex4.m1.1.1.1.1.6.2.2" xref="S3.Ex4.m1.1.1.1.1.6.2.2.cmml">r</mi><mo id="S3.Ex4.m1.1.1.1.1.6.2.1" xref="S3.Ex4.m1.1.1.1.1.6.2.1.cmml">^</mo></mover><mi id="S3.Ex4.m1.1.1.1.1.6.3" xref="S3.Ex4.m1.1.1.1.1.6.3.cmml">i</mi></msub><mo id="S3.Ex4.m1.1.1.1.1.5" xref="S3.Ex4.m1.1.1.1.1.5.cmml">=</mo><mrow id="S3.Ex4.m1.1.1.1.1.4" xref="S3.Ex4.m1.1.1.1.1.4.cmml"><mi id="S3.Ex4.m1.1.1.1.1.4.6" xref="S3.Ex4.m1.1.1.1.1.4.6.cmml">𝙻𝙻𝙼</mi><mo lspace="0em" rspace="0em" id="S3.Ex4.m1.1.1.1.1.4.5" xref="S3.Ex4.m1.1.1.1.1.4.5.cmml">​</mo><mrow id="S3.Ex4.m1.1.1.1.1.4.4.4" xref="S3.Ex4.m1.1.1.1.1.4.4.5.cmml"><mo stretchy="false" id="S3.Ex4.m1.1.1.1.1.4.4.4.5" xref="S3.Ex4.m1.1.1.1.1.4.4.5.cmml">(</mo><msub id="S3.Ex4.m1.1.1.1.1.1.1.1.1" xref="S3.Ex4.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.Ex4.m1.1.1.1.1.1.1.1.1.2" xref="S3.Ex4.m1.1.1.1.1.1.1.1.1.2.cmml">q</mi><mi id="S3.Ex4.m1.1.1.1.1.1.1.1.1.3" xref="S3.Ex4.m1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.Ex4.m1.1.1.1.1.4.4.4.6" xref="S3.Ex4.m1.1.1.1.1.4.4.5.cmml">,</mo><msub id="S3.Ex4.m1.1.1.1.1.2.2.2.2" xref="S3.Ex4.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S3.Ex4.m1.1.1.1.1.2.2.2.2.2" xref="S3.Ex4.m1.1.1.1.1.2.2.2.2.2.cmml">r</mi><mi id="S3.Ex4.m1.1.1.1.1.2.2.2.2.3" xref="S3.Ex4.m1.1.1.1.1.2.2.2.2.3.cmml">i</mi></msub><mo id="S3.Ex4.m1.1.1.1.1.4.4.4.7" xref="S3.Ex4.m1.1.1.1.1.4.4.5.cmml">,</mo><msub id="S3.Ex4.m1.1.1.1.1.3.3.3.3" xref="S3.Ex4.m1.1.1.1.1.3.3.3.3.cmml"><mi id="S3.Ex4.m1.1.1.1.1.3.3.3.3.2" xref="S3.Ex4.m1.1.1.1.1.3.3.3.3.2.cmml">f</mi><mi id="S3.Ex4.m1.1.1.1.1.3.3.3.3.3" xref="S3.Ex4.m1.1.1.1.1.3.3.3.3.3.cmml">i</mi></msub><mo id="S3.Ex4.m1.1.1.1.1.4.4.4.8" xref="S3.Ex4.m1.1.1.1.1.4.4.5.cmml">,</mo><msub id="S3.Ex4.m1.1.1.1.1.4.4.4.4" xref="S3.Ex4.m1.1.1.1.1.4.4.4.4.cmml"><mi id="S3.Ex4.m1.1.1.1.1.4.4.4.4.2" xref="S3.Ex4.m1.1.1.1.1.4.4.4.4.2.cmml">E</mi><mi id="S3.Ex4.m1.1.1.1.1.4.4.4.4.3" xref="S3.Ex4.m1.1.1.1.1.4.4.4.4.3.cmml">i</mi></msub><mo stretchy="false" id="S3.Ex4.m1.1.1.1.1.4.4.4.9" xref="S3.Ex4.m1.1.1.1.1.4.4.5.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S3.Ex4.m1.1.1.1.2" xref="S3.Ex4.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex4.m1.1b"><apply id="S3.Ex4.m1.1.1.1.1.cmml" xref="S3.Ex4.m1.1.1.1"><eq id="S3.Ex4.m1.1.1.1.1.5.cmml" xref="S3.Ex4.m1.1.1.1.1.5"></eq><apply id="S3.Ex4.m1.1.1.1.1.6.cmml" xref="S3.Ex4.m1.1.1.1.1.6"><csymbol cd="ambiguous" id="S3.Ex4.m1.1.1.1.1.6.1.cmml" xref="S3.Ex4.m1.1.1.1.1.6">subscript</csymbol><apply id="S3.Ex4.m1.1.1.1.1.6.2.cmml" xref="S3.Ex4.m1.1.1.1.1.6.2"><ci id="S3.Ex4.m1.1.1.1.1.6.2.1.cmml" xref="S3.Ex4.m1.1.1.1.1.6.2.1">^</ci><ci id="S3.Ex4.m1.1.1.1.1.6.2.2.cmml" xref="S3.Ex4.m1.1.1.1.1.6.2.2">𝑟</ci></apply><ci id="S3.Ex4.m1.1.1.1.1.6.3.cmml" xref="S3.Ex4.m1.1.1.1.1.6.3">𝑖</ci></apply><apply id="S3.Ex4.m1.1.1.1.1.4.cmml" xref="S3.Ex4.m1.1.1.1.1.4"><times id="S3.Ex4.m1.1.1.1.1.4.5.cmml" xref="S3.Ex4.m1.1.1.1.1.4.5"></times><ci id="S3.Ex4.m1.1.1.1.1.4.6.cmml" xref="S3.Ex4.m1.1.1.1.1.4.6">𝙻𝙻𝙼</ci><vector id="S3.Ex4.m1.1.1.1.1.4.4.5.cmml" xref="S3.Ex4.m1.1.1.1.1.4.4.4"><apply id="S3.Ex4.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex4.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.Ex4.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex4.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.Ex4.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex4.m1.1.1.1.1.1.1.1.1.2">𝑞</ci><ci id="S3.Ex4.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex4.m1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.Ex4.m1.1.1.1.1.2.2.2.2.cmml" xref="S3.Ex4.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.Ex4.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.Ex4.m1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S3.Ex4.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.Ex4.m1.1.1.1.1.2.2.2.2.2">𝑟</ci><ci id="S3.Ex4.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.Ex4.m1.1.1.1.1.2.2.2.2.3">𝑖</ci></apply><apply id="S3.Ex4.m1.1.1.1.1.3.3.3.3.cmml" xref="S3.Ex4.m1.1.1.1.1.3.3.3.3"><csymbol cd="ambiguous" id="S3.Ex4.m1.1.1.1.1.3.3.3.3.1.cmml" xref="S3.Ex4.m1.1.1.1.1.3.3.3.3">subscript</csymbol><ci id="S3.Ex4.m1.1.1.1.1.3.3.3.3.2.cmml" xref="S3.Ex4.m1.1.1.1.1.3.3.3.3.2">𝑓</ci><ci id="S3.Ex4.m1.1.1.1.1.3.3.3.3.3.cmml" xref="S3.Ex4.m1.1.1.1.1.3.3.3.3.3">𝑖</ci></apply><apply id="S3.Ex4.m1.1.1.1.1.4.4.4.4.cmml" xref="S3.Ex4.m1.1.1.1.1.4.4.4.4"><csymbol cd="ambiguous" id="S3.Ex4.m1.1.1.1.1.4.4.4.4.1.cmml" xref="S3.Ex4.m1.1.1.1.1.4.4.4.4">subscript</csymbol><ci id="S3.Ex4.m1.1.1.1.1.4.4.4.4.2.cmml" xref="S3.Ex4.m1.1.1.1.1.4.4.4.4.2">𝐸</ci><ci id="S3.Ex4.m1.1.1.1.1.4.4.4.4.3.cmml" xref="S3.Ex4.m1.1.1.1.1.4.4.4.4.3">𝑖</ci></apply></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex4.m1.1c">\hat{r}_{i}=\mathtt{LLM}(q_{i},r_{i},f_{i},E_{i}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</section>
</section>
<section id="S3.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.3.2 </span>Post-processing</h4>

<section id="S3.SS3.SSS2.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Length filtering.</h5>

<div id="S3.SS3.SSS2.Px1.p1" class="ltx_para">
<p id="S3.SS3.SSS2.Px1.p1.1" class="ltx_p">We find that LLMs sometimes fail to reformat and only output the changed sentences, whose output length plummets.
To filter out the data that fails to be reformatted, we keep the original response instead of using the reformatted response that is less than half the length of the original response.</p>
</div>
</section>
<section id="S3.SS3.SSS2.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Task-based filtering.</h5>

<div id="S3.SS3.SSS2.Px2.p1" class="ltx_para">
<p id="S3.SS3.SSS2.Px2.p1.1" class="ltx_p">To mitigate the problem of error propagation in task classification, we design filtering rules for specific tasks:

<span id="S3.I1" class="ltx_inline-enumerate">
<span id="S3.I1.i1" class="ltx_inline-item"><span class="ltx_tag ltx_tag_inline-item">(i)</span> <span id="S3.I1.i1.1" class="ltx_text">For code-related tasks (e.g., “<span id="S3.I1.i1.1.1" class="ltx_text ltx_font_typewriter">code correction</span>”), the keyword matching rule is employed to ascertain whether both the original and the reformatted versions contain code. If only one of the original responses or the reformatted response incorporates code, it signifies a failure in reformatting, and the original response is retained.
</span></span>
<span id="S3.I1.i2" class="ltx_inline-item"><span class="ltx_tag ltx_tag_inline-item">(ii)</span> <span id="S3.I1.i2.1" class="ltx_text">For the “<span id="S3.I1.i2.1.1" class="ltx_text ltx_font_typewriter">exam problem tutor</span>” task, reformatted responses that do not contain the accurate result will not be accepted.
</span></span>
<span id="S3.I1.i3" class="ltx_inline-item"><span class="ltx_tag ltx_tag_inline-item">(iii)</span> <span id="S3.I1.i3.1" class="ltx_text">For the “<span id="S3.I1.i3.1.1" class="ltx_text ltx_font_typewriter">planning</span>” task, if the query does not contain a planning-related keyword (e.g., plan or planning), the original answer is retained.
</span></span>
</span></p>
</div>
<div id="S3.SS3.SSS2.Px2.p2" class="ltx_para">
<p id="S3.SS3.SSS2.Px2.p2.1" class="ltx_p">Finally, we could acquire the reformatted dataset <math id="S3.SS3.SSS2.Px2.p2.1.m1.3" class="ltx_Math" alttext="\mathcal{\hat{D}}=\{(q_{1},\hat{r}_{1}),\cdots,(q_{n},\hat{r}_{n})\}" display="inline"><semantics id="S3.SS3.SSS2.Px2.p2.1.m1.3a"><mrow id="S3.SS3.SSS2.Px2.p2.1.m1.3.3" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.cmml"><mover accent="true" id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4.2.cmml">𝒟</mi><mo id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4.1" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4.1.cmml">^</mo></mover><mo id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.3" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.3.cmml">=</mo><mrow id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.3.cmml"><mo stretchy="false" id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.3" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.3.cmml">{</mo><mrow id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.3.cmml"><mo stretchy="false" id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.3" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.3.cmml">(</mo><msub id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.cmml"><mi id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.2.cmml">q</mi><mn id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.3" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.4" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.3.cmml">,</mo><msub id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.cmml"><mover accent="true" id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2.cmml"><mi id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2.2.cmml">r</mi><mo id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2.1" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2.1.cmml">^</mo></mover><mn id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.3" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.3.cmml">1</mn></msub><mo stretchy="false" id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.5" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.3.cmml">)</mo></mrow><mo id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.4" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.SS3.SSS2.Px2.p2.1.m1.1.1" xref="S3.SS3.SSS2.Px2.p2.1.m1.1.1.cmml">⋯</mi><mo id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.5" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.3.cmml">,</mo><mrow id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.3" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.3.cmml">(</mo><msub id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.cmml"><mi id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.2.cmml">q</mi><mi id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.3" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.3.cmml">n</mi></msub><mo id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.4" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.3.cmml">,</mo><msub id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.cmml"><mover accent="true" id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2.cmml"><mi id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2.2" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2.2.cmml">r</mi><mo id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2.1" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2.1.cmml">^</mo></mover><mi id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.3" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.3.cmml">n</mi></msub><mo stretchy="false" id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.5" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.3.cmml">)</mo></mrow><mo stretchy="false" id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.6" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.3.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS2.Px2.p2.1.m1.3b"><apply id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3"><eq id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.3.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.3"></eq><apply id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4"><ci id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4.1.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4.1">^</ci><ci id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4.2.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.4.2">𝒟</ci></apply><set id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.3.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2"><interval closure="open" id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.3.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2"><apply id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.2">𝑞</ci><cn type="integer" id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.1.1.3">1</cn></apply><apply id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.1.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2">subscript</csymbol><apply id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2"><ci id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2.1.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2.1">^</ci><ci id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2.2.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.2.2">𝑟</ci></apply><cn type="integer" id="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.3.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.2.2.1.1.1.2.2.3">1</cn></apply></interval><ci id="S3.SS3.SSS2.Px2.p2.1.m1.1.1.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.1.1">⋯</ci><interval closure="open" id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.3.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2"><apply id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.1.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1">subscript</csymbol><ci id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.2.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.2">𝑞</ci><ci id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.3.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.1.1.3">𝑛</ci></apply><apply id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.1.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2">subscript</csymbol><apply id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2"><ci id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2.1.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2.1">^</ci><ci id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2.2.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.2.2">𝑟</ci></apply><ci id="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.3.cmml" xref="S3.SS3.SSS2.Px2.p2.1.m1.3.3.2.2.2.2.2.3">𝑛</ci></apply></interval></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS2.Px2.p2.1.m1.3c">\mathcal{\hat{D}}=\{(q_{1},\hat{r}_{1}),\cdots,(q_{n},\hat{r}_{n})\}</annotation></semantics></math> (denotes as <span id="S3.SS3.SSS2.Px2.p2.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> dataset).</p>
</div>
</section>
</section>
</section>
</section>
<section id="S4" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Datasets</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">For evaluation of general ability, we select two high-quality manual datasets and one distillation dataset for instruction tuning:
(1) <span id="S4.SS1.p1.1.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Open-Platypus</span> <cite class="ltx_cite ltx_citemacro_cite">Lee et al. (<a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite> is an amalgamation of 11 open-source datasets, carefully curated to enhance LLM performance in STEM and logical domains. It consists of 25k questions, with around 90% written by humans and the rest generated by LLM.
(2) <span id="S4.SS1.p1.1.2" class="ltx_text ltx_font_typewriter ltx_font_bold">No Robots</span> <cite class="ltx_cite ltx_citemacro_cite">Rajani et al. (<a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite> is a high-quality dataset of 10k instructions and demonstrations created by skilled human annotators.
(3) <span id="S4.SS1.p1.1.3" class="ltx_text ltx_font_typewriter ltx_font_bold">Alpaca</span> <cite class="ltx_cite ltx_citemacro_cite">Taori et al. (<a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite> is an open-source instruction tuning dataset generated from text-davinci-003 <cite class="ltx_cite ltx_citemacro_cite">Ouyang et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite> by the <span id="S4.SS1.p1.1.4" class="ltx_text ltx_font_italic">Self-Instruct</span> <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib35" title="" class="ltx_ref">2023b</a>)</cite> method, containing 52k samples.
Additionally, we also choose two manual datasets to evaluate the math reasoning after using <span id="S4.SS1.p1.1.5" class="ltx_text ltx_font_smallcaps">ReAlign</span>:
(4) <span id="S4.SS1.p1.1.6" class="ltx_text ltx_font_typewriter ltx_font_bold">GSM8K</span> <cite class="ltx_cite ltx_citemacro_cite">Cobbe et al. (<a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite> is a high-quality grade school math problems dataset created by human problem writers, consisting of 7.5k training problems and 1k test problems.
(5) <span id="S4.SS1.p1.1.7" class="ltx_text ltx_font_typewriter ltx_font_bold">MATH</span> <cite class="ltx_cite ltx_citemacro_cite">Hendrycks et al. (<a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite> is a dataset of mathematics competitions problems, including 7.5k for training and 5k for testing.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Models</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">We select two well-known open-source base models for fine-tuning:
(1) <span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_bold">LLaMA-2-13B</span> <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite> is a open-source pre-trained model using 2T tokens.
(2) <span id="S4.SS2.p1.1.2" class="ltx_text ltx_font_bold">Mistral-7B</span> <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a href="#bib.bib12" title="" class="ltx_ref">2023</a>)</cite> is the current state-of-the-art base language model at the 7B parameter scale.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Evaluation</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">We evaluate <span id="S4.SS3.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> on general alignment and specific alignment ability including math reasoning, factuality, and readability.</p>
</div>
<section id="S4.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.1 </span>General Alignment</h4>

<div id="S4.SS3.SSS1.p1" class="ltx_para">
<p id="S4.SS3.SSS1.p1.1" class="ltx_p">To evaluate the general alignment ability, we follow <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib34" title="" class="ltx_ref">2023a</a>)</cite> to employ the most widely recognized benchmarks, including: <span id="S4.SS3.SSS1.p1.1.1" class="ltx_text ltx_font_bold">AlpacaEval</span> <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib19" title="" class="ltx_ref">2023c</a>)</cite>, <span id="S4.SS3.SSS1.p1.1.2" class="ltx_text ltx_font_bold">MT-Bench</span> <cite class="ltx_cite ltx_citemacro_cite">Zheng et al. (<a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite>, <span id="S4.SS3.SSS1.p1.1.3" class="ltx_text ltx_font_bold">Vicuna-Bench</span> <cite class="ltx_cite ltx_citemacro_cite">Chiang et al. (<a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite>. Specifically, we use GPT-3.5 and Auto-J (detailed in §<a href="#A4" title="Appendix D The Description of Auto-J ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a>) as the evaluators for AlpacaEval due to the cost of GPT-4, which has an extremely strong correlation with human <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib16" title="" class="ltx_ref">2024</a>); Sun et al. (<a href="#bib.bib30" title="" class="ltx_ref">2024</a>)</cite>, and GPT-4 for MT-Bench and Vicuna-Bench.</p>
</div>
</section>
<section id="S4.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.2 </span>Specific Alignment</h4>

<div id="S4.SS3.SSS2.p1" class="ltx_para">
<p id="S4.SS3.SSS2.p1.1" class="ltx_p">We evaluate specific perspectives for alignment, including math reasoning, factuality, and readability.</p>
</div>
<figure id="S4.T3" class="ltx_table">
<table id="S4.T3.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T3.1.1" class="ltx_tr">
<td id="S4.T3.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="2"><span id="S4.T3.1.1.1.1" class="ltx_text">Model</span></td>
<td id="S4.T3.1.1.2" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="2"><span id="S4.T3.1.1.2.1" class="ltx_text">Dataset</span></td>
<td id="S4.T3.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="2">AlpacaEval</td>
<td id="S4.T3.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="3">MT-Bench</td>
<td id="S4.T3.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="2"><span id="S4.T3.1.1.5.1" class="ltx_text">Vicuna-Bench</span></td>
<td id="S4.T3.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="2"><span id="S4.T3.1.1.6.1" class="ltx_text">Overall</span></td>
</tr>
<tr id="S4.T3.1.2" class="ltx_tr">
<td id="S4.T3.1.2.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">GPT-3.5 (%)</td>
<td id="S4.T3.1.2.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Auto-J</td>
<td id="S4.T3.1.2.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">First</td>
<td id="S4.T3.1.2.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Second</td>
<td id="S4.T3.1.2.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Average</td>
</tr>
<tr id="S4.T3.1.3" class="ltx_tr">
<td id="S4.T3.1.3.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="6"><span id="S4.T3.1.3.1.1" class="ltx_text">LLaMA-2-13B</span></td>
<td id="S4.T3.1.3.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.3.2.1" class="ltx_text ltx_font_typewriter">Open-Platypus</span></td>
<td id="S4.T3.1.3.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">55.71</td>
<td id="S4.T3.1.3.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.3.4.1" class="ltx_text ltx_font_bold">4.93</span></td>
<td id="S4.T3.1.3.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.69</td>
<td id="S4.T3.1.3.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.3.6.1" class="ltx_text ltx_font_bold">5.16</span></td>
<td id="S4.T3.1.3.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.3.7.1" class="ltx_text ltx_font_bold">5.94</span></td>
<td id="S4.T3.1.3.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">8.28</td>
<td id="S4.T3.1.3.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.18</td>
</tr>
<tr id="S4.T3.1.4" class="ltx_tr">
<td id="S4.T3.1.4.1" class="ltx_td ltx_align_left" style="padding-left:5.0pt;padding-right:5.0pt;">+ <span id="S4.T3.1.4.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T3.1.4.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.4.2.1" class="ltx_text ltx_font_bold">58.20</span></td>
<td id="S4.T3.1.4.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">4.81</td>
<td id="S4.T3.1.4.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.4.4.1" class="ltx_text ltx_font_bold">6.89</span></td>
<td id="S4.T3.1.4.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">4.86</td>
<td id="S4.T3.1.4.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.88</td>
<td id="S4.T3.1.4.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.4.7.1" class="ltx_text ltx_font_bold">8.45</span></td>
<td id="S4.T3.1.4.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.4.8.1" class="ltx_text ltx_font_bold">6.24</span></td>
</tr>
<tr id="S4.T3.1.5" class="ltx_tr">
<td id="S4.T3.1.5.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.5.1.1" class="ltx_text ltx_font_typewriter">No Robots</span></td>
<td id="S4.T3.1.5.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">44.25</td>
<td id="S4.T3.1.5.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">4.56</td>
<td id="S4.T3.1.5.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.80</td>
<td id="S4.T3.1.5.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.15</td>
<td id="S4.T3.1.5.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.48</td>
<td id="S4.T3.1.5.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.31</td>
<td id="S4.T3.1.5.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.44</td>
</tr>
<tr id="S4.T3.1.6" class="ltx_tr">
<td id="S4.T3.1.6.1" class="ltx_td ltx_align_left" style="padding-left:5.0pt;padding-right:5.0pt;">+ <span id="S4.T3.1.6.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T3.1.6.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.6.2.1" class="ltx_text ltx_font_bold">48.13</span></td>
<td id="S4.T3.1.6.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.6.3.1" class="ltx_text ltx_font_bold">4.65</span></td>
<td id="S4.T3.1.6.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.6.4.1" class="ltx_text ltx_font_bold">6.04</span></td>
<td id="S4.T3.1.6.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.6.5.1" class="ltx_text ltx_font_bold">5.20</span></td>
<td id="S4.T3.1.6.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.6.6.1" class="ltx_text ltx_font_bold">5.62</span></td>
<td id="S4.T3.1.6.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.6.7.1" class="ltx_text ltx_font_bold">7.51</span></td>
<td id="S4.T3.1.6.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.6.8.1" class="ltx_text ltx_font_bold">5.65</span></td>
</tr>
<tr id="S4.T3.1.7" class="ltx_tr">
<td id="S4.T3.1.7.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.7.1.1" class="ltx_text ltx_font_typewriter">Alpaca</span></td>
<td id="S4.T3.1.7.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">46.08</td>
<td id="S4.T3.1.7.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">4.65</td>
<td id="S4.T3.1.7.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.55</td>
<td id="S4.T3.1.7.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">4.16</td>
<td id="S4.T3.1.7.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">4.86</td>
<td id="S4.T3.1.7.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.55</td>
<td id="S4.T3.1.7.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.17</td>
</tr>
<tr id="S4.T3.1.8" class="ltx_tr">
<td id="S4.T3.1.8.1" class="ltx_td ltx_align_left" style="padding-left:5.0pt;padding-right:5.0pt;">+ <span id="S4.T3.1.8.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T3.1.8.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.8.2.1" class="ltx_text ltx_font_bold">49.19</span></td>
<td id="S4.T3.1.8.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.8.3.1" class="ltx_text ltx_font_bold">4.74</span></td>
<td id="S4.T3.1.8.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.8.4.1" class="ltx_text ltx_font_bold">5.83</span></td>
<td id="S4.T3.1.8.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.8.5.1" class="ltx_text ltx_font_bold">4.71</span></td>
<td id="S4.T3.1.8.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.8.6.1" class="ltx_text ltx_font_bold">5.27</span></td>
<td id="S4.T3.1.8.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.8.7.1" class="ltx_text ltx_font_bold">6.84</span></td>
<td id="S4.T3.1.8.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.8.8.1" class="ltx_text ltx_font_bold">5.44</span></td>
</tr>
<tr id="S4.T3.1.9" class="ltx_tr">
<td id="S4.T3.1.9.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="8"><span id="S4.T3.1.9.1.1" class="ltx_text">Mistral-7B</span></td>
<td id="S4.T3.1.9.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.9.2.1" class="ltx_text ltx_font_typewriter">Open-Platypus</span></td>
<td id="S4.T3.1.9.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">59.63</td>
<td id="S4.T3.1.9.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.15</td>
<td id="S4.T3.1.9.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.29</td>
<td id="S4.T3.1.9.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.88</td>
<td id="S4.T3.1.9.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.58</td>
<td id="S4.T3.1.9.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.9.8.1" class="ltx_text ltx_font_bold">8.96</span></td>
<td id="S4.T3.1.9.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.66</td>
</tr>
<tr id="S4.T3.1.10" class="ltx_tr">
<td id="S4.T3.1.10.1" class="ltx_td ltx_align_left" style="padding-left:5.0pt;padding-right:5.0pt;">+ <span id="S4.T3.1.10.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T3.1.10.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.10.2.1" class="ltx_text ltx_font_bold">61.33</span></td>
<td id="S4.T3.1.10.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.10.3.1" class="ltx_text ltx_font_bold">5.15</span></td>
<td id="S4.T3.1.10.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.10.4.1" class="ltx_text ltx_font_bold">7.43</span></td>
<td id="S4.T3.1.10.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.10.5.1" class="ltx_text ltx_font_bold">6.18</span></td>
<td id="S4.T3.1.10.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.10.6.1" class="ltx_text ltx_font_bold">6.80</span></td>
<td id="S4.T3.1.10.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">8.86</td>
<td id="S4.T3.1.10.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.10.8.1" class="ltx_text ltx_font_bold">6.74</span></td>
</tr>
<tr id="S4.T3.1.11" class="ltx_tr">
<td id="S4.T3.1.11.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.11.1.1" class="ltx_text ltx_font_typewriter">No Robots</span></td>
<td id="S4.T3.1.11.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">44.22</td>
<td id="S4.T3.1.11.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">4.62</td>
<td id="S4.T3.1.11.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.95</td>
<td id="S4.T3.1.11.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.11.5.1" class="ltx_text ltx_font_bold">4.94</span></td>
<td id="S4.T3.1.11.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.44</td>
<td id="S4.T3.1.11.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.32</td>
<td id="S4.T3.1.11.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.45</td>
</tr>
<tr id="S4.T3.1.12" class="ltx_tr">
<td id="S4.T3.1.12.1" class="ltx_td ltx_align_left" style="padding-left:5.0pt;padding-right:5.0pt;">+ <span id="S4.T3.1.12.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T3.1.12.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.12.2.1" class="ltx_text ltx_font_bold">48.26</span></td>
<td id="S4.T3.1.12.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.12.3.1" class="ltx_text ltx_font_bold">4.76</span></td>
<td id="S4.T3.1.12.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.12.4.1" class="ltx_text ltx_font_bold">6.14</span></td>
<td id="S4.T3.1.12.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">4.79</td>
<td id="S4.T3.1.12.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.12.6.1" class="ltx_text ltx_font_bold">5.46</span></td>
<td id="S4.T3.1.12.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.12.7.1" class="ltx_text ltx_font_bold">7.68</span></td>
<td id="S4.T3.1.12.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.12.8.1" class="ltx_text ltx_font_bold">5.68</span></td>
</tr>
<tr id="S4.T3.1.13" class="ltx_tr">
<td id="S4.T3.1.13.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.13.1.1" class="ltx_text ltx_font_typewriter">Alpaca</span></td>
<td id="S4.T3.1.13.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">51.24</td>
<td id="S4.T3.1.13.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">4.77</td>
<td id="S4.T3.1.13.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.06</td>
<td id="S4.T3.1.13.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.13.5.1" class="ltx_text ltx_font_bold">5.26</span></td>
<td id="S4.T3.1.13.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.66</td>
<td id="S4.T3.1.13.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.14</td>
<td id="S4.T3.1.13.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.67</td>
</tr>
<tr id="S4.T3.1.14" class="ltx_tr">
<td id="S4.T3.1.14.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">+ <span id="S4.T3.1.14.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T3.1.14.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.14.2.1" class="ltx_text ltx_font_bold">52.67</span></td>
<td id="S4.T3.1.14.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.14.3.1" class="ltx_text ltx_font_bold">4.82</span></td>
<td id="S4.T3.1.14.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.14.4.1" class="ltx_text ltx_font_bold">6.50</span></td>
<td id="S4.T3.1.14.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">5.03</td>
<td id="S4.T3.1.14.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.14.6.1" class="ltx_text ltx_font_bold">5.76</span></td>
<td id="S4.T3.1.14.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.14.7.1" class="ltx_text ltx_font_bold">7.33</span></td>
<td id="S4.T3.1.14.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.1.14.8.1" class="ltx_text ltx_font_bold">5.79</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>The results of the general alignment ability on the original datasets and the <span id="S4.T3.4.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> datasets. <span id="S4.T3.5.2" class="ltx_text ltx_font_bold">Bold</span> indicates the best result on each dataset. For AlpacaEval, GPT-3.5 denotes the winning rate obtained by using GPT-3.5 as the evaluator. Auto-J denotes the quality of the model’s responses evaluated in a point-wise manner using Auto-J <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite>. For MT-Bench, we report the result of the first turn, the second turn, and the average, respectively. For Overall, we calculate the average of AlpacaEval’s winning rate for GPT-3.5 divided by 10, the results for Auto-J, the average MT-Bench results, and the results for Vicuna-Bench.</figcaption>
</figure>
<section id="S4.SS3.SSS2.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Math Reasoning.</h5>

<div id="S4.SS3.SSS2.Px1.p1" class="ltx_para">
<p id="S4.SS3.SSS2.Px1.p1.1" class="ltx_p">To evaluate math reasoning, we finetune LLaMA-2-13B and Mistral-7B on <span id="S4.SS3.SSS2.Px1.p1.1.1" class="ltx_text ltx_font_typewriter">GSM8K</span> and <span id="S4.SS3.SSS2.Px1.p1.1.2" class="ltx_text ltx_font_typewriter">MATH</span> training datasets, respectively, and test afterward. The prompt template for training and testing is <span id="S4.SS3.SSS2.Px1.p1.1.3" class="ltx_text ltx_font_typewriter">‘‘Question:\n
<br class="ltx_break">{input}\n Answer:\nLet’s think step by step.\n’’</span>.
Since both datasets consist of math problems in the same style, we apply forced rewriting instead of adaptive, which does not require the determination of whether the query and format match but rather mandates a rewriting. We determine the accuracy by extracting the last number from the responses and comparing it directly to the ground truth.</p>
</div>
<figure id="S4.T4" class="ltx_table">
<div class="ltx_flex_figure ltx_flex_table">
<div class="ltx_flex_cell ltx_flex_size_1">
<table id="S4.T4.1" class="ltx_tabular ltx_centering ltx_figure_panel ltx_align_middle">
<tr id="S4.T4.1.1" class="ltx_tr">
<td id="S4.T4.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;">Model</td>
<td id="S4.T4.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;">Dataset</td>
<td id="S4.T4.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.1.3.1" class="ltx_text ltx_font_typewriter">GSM8K</span></td>
<td id="S4.T4.1.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.1.4.1" class="ltx_text ltx_font_typewriter">MATH</span></td>
<td id="S4.T4.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;">Overall</td>
</tr>
<tr id="S4.T4.1.2" class="ltx_tr">
<td id="S4.T4.1.2.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="S4.T4.1.2.1.1" class="ltx_text">LLaMA-2-13B</span></td>
<td id="S4.T4.1.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.2.2.1" class="ltx_text ltx_font_typewriter">GSM8K</span></td>
<td id="S4.T4.1.2.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">46.77</td>
<td id="S4.T4.1.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.02</td>
<td id="S4.T4.1.2.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">25.90</td>
</tr>
<tr id="S4.T4.1.3" class="ltx_tr">
<td id="S4.T4.1.3.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">+ <span id="S4.T4.1.3.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T4.1.3.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.3.2.1" class="ltx_text ltx_font_bold">56.63</span></td>
<td id="S4.T4.1.3.3" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.3.3.1" class="ltx_text ltx_font_bold">5.46</span></td>
<td id="S4.T4.1.3.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.3.4.1" class="ltx_text ltx_font_bold">31.05</span></td>
</tr>
<tr id="S4.T4.1.4" class="ltx_tr">
<td id="S4.T4.1.4.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.4.1.1" class="ltx_text ltx_font_typewriter">MATH</span></td>
<td id="S4.T4.1.4.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">14.48</td>
<td id="S4.T4.1.4.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.14</td>
<td id="S4.T4.1.4.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">10.31</td>
</tr>
<tr id="S4.T4.1.5" class="ltx_tr">
<td id="S4.T4.1.5.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">+ <span id="S4.T4.1.5.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T4.1.5.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.5.2.1" class="ltx_text ltx_font_bold">25.17</span></td>
<td id="S4.T4.1.5.3" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.5.3.1" class="ltx_text ltx_font_bold">7.14</span></td>
<td id="S4.T4.1.5.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.5.4.1" class="ltx_text ltx_font_bold">16.16</span></td>
</tr>
<tr id="S4.T4.1.6" class="ltx_tr">
<td id="S4.T4.1.6.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="S4.T4.1.6.1.1" class="ltx_text">Mistral-7B</span></td>
<td id="S4.T4.1.6.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.6.2.1" class="ltx_text ltx_font_typewriter">GSM8K</span></td>
<td id="S4.T4.1.6.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">57.62</td>
<td id="S4.T4.1.6.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.68</td>
<td id="S4.T4.1.6.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">32.65</td>
</tr>
<tr id="S4.T4.1.7" class="ltx_tr">
<td id="S4.T4.1.7.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">+ <span id="S4.T4.1.7.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T4.1.7.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.7.2.1" class="ltx_text ltx_font_bold">62.47</span></td>
<td id="S4.T4.1.7.3" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.7.3.1" class="ltx_text ltx_font_bold">9.02</span></td>
<td id="S4.T4.1.7.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.7.4.1" class="ltx_text ltx_font_bold">35.75</span></td>
</tr>
<tr id="S4.T4.1.8" class="ltx_tr">
<td id="S4.T4.1.8.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.8.1.1" class="ltx_text ltx_font_typewriter">MATH</span></td>
<td id="S4.T4.1.8.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">28.35</td>
<td id="S4.T4.1.8.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">13.18</td>
<td id="S4.T4.1.8.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">20.77</td>
</tr>
<tr id="S4.T4.1.9" class="ltx_tr">
<td id="S4.T4.1.9.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">+ <span id="S4.T4.1.9.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T4.1.9.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.9.2.1" class="ltx_text ltx_font_bold">38.21</span></td>
<td id="S4.T4.1.9.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.9.3.1" class="ltx_text ltx_font_bold">15.30</span></td>
<td id="S4.T4.1.9.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T4.1.9.4.1" class="ltx_text ltx_font_bold">26.76</span></td>
</tr>
</table>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>The results of math reasoning on <span id="S4.T4.8.1" class="ltx_text ltx_font_typewriter">GSM8K</span>, <span id="S4.T4.9.2" class="ltx_text ltx_font_typewriter">MATH</span> and them + <span id="S4.T4.10.3" class="ltx_text ltx_font_smallcaps">ReAlign</span> based on LLaMA-2-13B and Mistral-7B. We test models on both <span id="S4.T4.11.4" class="ltx_text ltx_font_typewriter">GSM8K</span> and <span id="S4.T4.12.5" class="ltx_text ltx_font_typewriter">MATH</span> test sets. We report the accuracy by exact matching. <span id="S4.T4.13.6" class="ltx_text ltx_font_bold">Bold</span> indicates the best result.</figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S4.T4.14" class="ltx_p ltx_figure_panel ltx_align_center">.</p>
</div>
</div>
</figure>
</section>
<section id="S4.SS3.SSS2.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Factuality.</h5>

<div id="S4.SS3.SSS2.Px2.p1" class="ltx_para">
<p id="S4.SS3.SSS2.Px2.p1.1" class="ltx_p">To evaluate the factuality of models, we randomly select 100 cases from the Natural Questions dataset (NQ) <cite class="ltx_cite ltx_citemacro_cite">Kwiatkowski et al. (<a href="#bib.bib13" title="" class="ltx_ref">2019</a>)</cite>, a public Q&amp;A dataset rich in fact-based queries and their verified answers.
We employ GPT-4 to rate these instances on a factuality scale of 1 to 10, considering the question, the response, and the ground truth (referred to as the factuality score). The evaluation prompt is shown in Tab. <a href="#A5.T17" title="Table 17 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">17</span></a>.</p>
</div>
</section>
<section id="S4.SS3.SSS2.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Readability.</h5>

<div id="S4.SS3.SSS2.Px3.p1" class="ltx_para">
<p id="S4.SS3.SSS2.Px3.p1.1" class="ltx_p">To evaluate the readability, we compare a model trained on the original dataset against another model on the dataset enhanced with <span id="S4.SS3.SSS2.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>, using human and GPT-4 evaluations on the Vicuna-Bench dataset <cite class="ltx_cite ltx_citemacro_cite">Chiang et al. (<a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite>. Since the vicuna bench contains fewer complex questions (e.g., code and math), the judge can focus on the format rather than the result. We design an evaluation prompt prioritizing readability, refer to Tab. <a href="#A5.T16" title="Table 16 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>, and randomize response positions to eliminate bias.</p>
</div>
</section>
</section>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Results</h3>

<section id="S4.SS4.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">
<span id="S4.SS4.SSS0.Px1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> Improves General Alignment Ability.</h5>

<div id="S4.SS4.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px1.p1.1" class="ltx_p">Following <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib34" title="" class="ltx_ref">2023a</a>)</cite>, we conduct experiments on AlpacaEval, MT-Bench, and Vicuna-Bench to evaluate the general alignment ability.
From Tab. <a href="#S4.T3" title="Table 3 ‣ 4.3.2 Specific Alignment ‣ 4.3 Evaluation ‣ 4 Experiments ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we can see an increase in almost all three datasets and benchmarks on both the LLaMA-2-13B and Mistral-7B models, showing that <span id="S4.SS4.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> can significantly improve models’ response quality and conversation ability. Additionally, from the results of MT-Bench, we can see that <span id="S4.SS4.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_smallcaps">ReAlign</span> can improve the performance of the second turn of conversations on half the datasets even though it only rewrites the first turn of the instruction data.</p>
</div>
</section>
<section id="S4.SS4.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">
<span id="S4.SS4.SSS0.Px2.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> Can Boost Math Reasoning.</h5>

<div id="S4.SS4.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px2.p1.1" class="ltx_p">To evaluate the effect of <span id="S4.SS4.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> on math reasoning, we apply <span id="S4.SS4.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_smallcaps">ReAlign</span> to <span id="S4.SS4.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_typewriter">GSM8K</span> and <span id="S4.SS4.SSS0.Px2.p1.1.4" class="ltx_text ltx_font_typewriter">MATH</span> datasets. As shown in Tab. <a href="#S4.T4" title="Table 4 ‣ Math Reasoning. ‣ 4.3.2 Specific Alignment ‣ 4.3 Evaluation ‣ 4 Experiments ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, <span id="S4.SS4.SSS0.Px2.p1.1.5" class="ltx_text ltx_font_smallcaps">ReAlign</span> can dramatically boost the math reasoning on both datasets, even up to 9.86% on <span id="S4.SS4.SSS0.Px2.p1.1.6" class="ltx_text ltx_font_typewriter">GSM8K</span> using LLaMA-2-13B.
Remarkably, <span id="S4.SS4.SSS0.Px2.p1.1.7" class="ltx_text ltx_font_smallcaps">ReAlign</span> enhances generalization, demonstrated by cross-domain performance boosts. Specifically, training models using the <span id="S4.SS4.SSS0.Px2.p1.1.8" class="ltx_text ltx_font_typewriter">MATH</span> dataset yields notable improvements in the <span id="S4.SS4.SSS0.Px2.p1.1.9" class="ltx_text ltx_font_typewriter">GSM8K</span> test results, and vice versa. For instance, it has been observed that training on the <span id="S4.SS4.SSS0.Px2.p1.1.10" class="ltx_text ltx_font_typewriter">MATH</span> dataset can augment <span id="S4.SS4.SSS0.Px2.p1.1.11" class="ltx_text ltx_font_typewriter">GSM8K</span> performance by 10.69% based on LLaMA-2-13B.
One possible reason is that <span id="S4.SS4.SSS0.Px2.p1.1.12" class="ltx_text ltx_font_smallcaps">ReAlign</span> can provide more and clearer intermediate steps and explanations, thereby bolstering the reasoning ability of models.</p>
</div>
</section>
<section id="S4.SS4.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">
<span id="S4.SS4.SSS0.Px3.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> Can Enhance Factuality.</h5>

<div id="S4.SS4.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px3.p1.1" class="ltx_p">To evaluate the factuality, we employ <span id="S4.SS4.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> to <span id="S4.SS4.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_typewriter">Open-Platypus</span>, <span id="S4.SS4.SSS0.Px3.p1.1.3" class="ltx_text ltx_font_typewriter">No Robots</span>, and <span id="S4.SS4.SSS0.Px3.p1.1.4" class="ltx_text ltx_font_typewriter">Alpaca</span> datasets with LLaMA-2-13B, subsequently comparing the response to ground truth in NQ samples.
Fig. <a href="#S4.F4" title="Figure 4 ‣ ReAlign Can Enhance Factuality. ‣ 4.4 Results ‣ 4 Experiments ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows <span id="S4.SS4.SSS0.Px3.p1.1.5" class="ltx_text ltx_font_smallcaps">ReAlign</span> elevates the factuality, highlighting its efficacy. This improvement is probably due to the addition of retrieval augmentation.</p>
</div>
<figure id="S4.F4" class="ltx_figure"><img src="/html/2402.12219/assets/x5.png" id="S4.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="276" height="167" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>The results of the factuality score.</figcaption>
</figure>
<figure id="S4.F5" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F5.sf1" class="ltx_figure ltx_figure_panel ltx_align_center">
<div id="S4.F5.sf1.1" class="ltx_block ltx_minipage ltx_align_bottom" style="width:208.1pt;">
<img src="/html/2402.12219/assets/x6.png" id="S4.F5.sf1.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="177" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>GPT-4 Judgments</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F5.sf2" class="ltx_figure ltx_figure_panel ltx_align_center">
<div id="S4.F5.sf2.1" class="ltx_block ltx_minipage ltx_align_bottom" style="width:208.1pt;">
<img src="/html/2402.12219/assets/x7.png" id="S4.F5.sf2.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="165" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>Human Judgments</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>The readability win-rate of the original dataset + <span id="S4.F5.2.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> against the original dataset based on LLaMA-2-13B, judged by GPT-4 and human.</figcaption>
</figure>
</section>
<section id="S4.SS4.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">
<span id="S4.SS4.SSS0.Px4.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> Can Improve Readability.</h5>

<div id="S4.SS4.SSS0.Px4.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px4.p1.1" class="ltx_p">To evaluate the readability of the responses, we use a readability evaluation prompt (refer to Tab. <a href="#A5.T16" title="Table 16 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>) to guide GPT-4 and human to compare the model trained on the original dataset with the model trained with the addition of <span id="S4.SS4.SSS0.Px4.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>. As shown in Fig. <a href="#S4.F5" title="Figure 5 ‣ ReAlign Can Enhance Factuality. ‣ 4.4 Results ‣ 4 Experiments ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, we see that <span id="S4.SS4.SSS0.Px4.p1.1.2" class="ltx_text ltx_font_smallcaps">ReAlign</span> can improve the readability of three datasets, especially in the <span id="S4.SS4.SSS0.Px4.p1.1.3" class="ltx_text ltx_font_typewriter">Open-Platypus</span> dataset (i.e., 18.7% improvements in GPT-4 judgments). It demonstrates that designing different formats for different tasks and reformatting them can effectively improve readability. In addition, human tends to provide more ties for judgments compared to GPT-4. A possible reason is that <span id="S4.SS4.SSS0.Px4.p1.1.4" class="ltx_text ltx_font_smallcaps">ReAlign</span> can provide better structure, causing GPT-4 to be limited to surface formats ignoring content and deep structure. In contrast, humans can read more carefully not being limited to surface formats.</p>
</div>
<figure id="S4.T5" class="ltx_table">
<table id="S4.T5.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T5.1.1" class="ltx_tr">
<td id="S4.T5.1.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;">Dataset</td>
<td id="S4.T5.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;">Response Len.</td>
<td id="S4.T5.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;">
<span id="S4.T5.1.1.3.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> %</td>
</tr>
<tr id="S4.T5.1.2" class="ltx_tr">
<td id="S4.T5.1.2.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T5.1.2.1.1" class="ltx_text ltx_font_typewriter">Open-Platypus</span></td>
<td id="S4.T5.1.2.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">224.92</td>
<td id="S4.T5.1.2.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">-</td>
</tr>
<tr id="S4.T5.1.3" class="ltx_tr">
<td id="S4.T5.1.3.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;">+ <span id="S4.T5.1.3.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T5.1.3.2" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">206.91</td>
<td id="S4.T5.1.3.3" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">28.5%</td>
</tr>
<tr id="S4.T5.1.4" class="ltx_tr">
<td id="S4.T5.1.4.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T5.1.4.1.1" class="ltx_text ltx_font_typewriter">No Robots</span></td>
<td id="S4.T5.1.4.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">211.99</td>
<td id="S4.T5.1.4.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">-</td>
</tr>
<tr id="S4.T5.1.5" class="ltx_tr">
<td id="S4.T5.1.5.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;">+ <span id="S4.T5.1.5.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T5.1.5.2" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">211.54</td>
<td id="S4.T5.1.5.3" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">15.9%</td>
</tr>
<tr id="S4.T5.1.6" class="ltx_tr">
<td id="S4.T5.1.6.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T5.1.6.1.1" class="ltx_text ltx_font_typewriter">Alpaca</span></td>
<td id="S4.T5.1.6.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">65.51</td>
<td id="S4.T5.1.6.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">-</td>
</tr>
<tr id="S4.T5.1.7" class="ltx_tr">
<td id="S4.T5.1.7.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;">+ <span id="S4.T5.1.7.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T5.1.7.2" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">72.38</td>
<td id="S4.T5.1.7.3" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">29.9%</td>
</tr>
<tr id="S4.T5.1.8" class="ltx_tr">
<td id="S4.T5.1.8.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T5.1.8.1.1" class="ltx_text ltx_font_typewriter">GSM8K</span></td>
<td id="S4.T5.1.8.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">130.59</td>
<td id="S4.T5.1.8.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">-</td>
</tr>
<tr id="S4.T5.1.9" class="ltx_tr">
<td id="S4.T5.1.9.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;">+ <span id="S4.T5.1.9.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T5.1.9.2" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">327.65</td>
<td id="S4.T5.1.9.3" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">100%</td>
</tr>
<tr id="S4.T5.1.10" class="ltx_tr">
<td id="S4.T5.1.10.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T5.1.10.1.1" class="ltx_text ltx_font_typewriter">MATH</span></td>
<td id="S4.T5.1.10.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">243.73</td>
<td id="S4.T5.1.10.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">-</td>
</tr>
<tr id="S4.T5.1.11" class="ltx_tr">
<td id="S4.T5.1.11.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;">+ <span id="S4.T5.1.11.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T5.1.11.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;">375.35</td>
<td id="S4.T5.1.11.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;">100%</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>The datasets analysis includes original datasets and them + <span id="S4.T5.4.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>. Response Len. is the average number of tokens of the responses. <span id="S4.T5.5.2" class="ltx_text ltx_font_smallcaps">ReAlign</span> % denotes the percentage of successful reformatting after the adaptive rewriting.</figcaption>
</figure>
<figure id="S4.F6" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F6.sf1" class="ltx_figure ltx_figure_panel ltx_align_center">
<div id="S4.F6.sf1.1" class="ltx_block ltx_minipage ltx_align_bottom" style="width:208.1pt;">
<img src="/html/2402.12219/assets/x8.png" id="S4.F6.sf1.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="281" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>BBH</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F6.sf2" class="ltx_figure ltx_figure_panel ltx_align_center">
<div id="S4.F6.sf2.1" class="ltx_block ltx_minipage ltx_align_bottom" style="width:208.1pt;">
<img src="/html/2402.12219/assets/x9.png" id="S4.F6.sf2.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="292" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>AGIEval</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>The results of the knowledge abilities, including the Big Bench Hard (BBH) (3-shot), and AGIEval (zero-shot). We evaluate the abilities across the <span id="S4.F6.4.1" class="ltx_text ltx_font_typewriter">Open-Platypus</span>, <span id="S4.F6.5.2" class="ltx_text ltx_font_typewriter">No Robots</span>, and <span id="S4.F6.6.3" class="ltx_text ltx_font_typewriter">Alpaca</span> datasets, based on LLaMA-2-13B.</figcaption>
</figure>
<figure id="S4.T6" class="ltx_table">
<table id="S4.T6.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T6.1.1" class="ltx_tr">
<td id="S4.T6.1.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;">Dataset</td>
<td id="S4.T6.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;">General Align.</td>
<td id="S4.T6.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;">Know. Ab.</td>
<td id="S4.T6.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;">FS</td>
</tr>
<tr id="S4.T6.1.2" class="ltx_tr">
<td id="S4.T6.1.2.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T6.1.2.1.1" class="ltx_text ltx_font_typewriter">Open-Platypus</span></td>
<td id="S4.T6.1.2.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">6.18</td>
<td id="S4.T6.1.2.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">39.65</td>
<td id="S4.T6.1.2.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">5.1</td>
</tr>
<tr id="S4.T6.1.3" class="ltx_tr">
<td id="S4.T6.1.3.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;">+ <span id="S4.T6.1.3.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</td>
<td id="S4.T6.1.3.2" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T6.1.3.2.1" class="ltx_text ltx_font_bold">6.24</span></td>
<td id="S4.T6.1.3.3" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T6.1.3.3.1" class="ltx_text ltx_font_bold">41.35</span></td>
<td id="S4.T6.1.3.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">5.5</td>
</tr>
<tr id="S4.T6.1.4" class="ltx_tr">
<td id="S4.T6.1.4.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">W/o RAG</td>
<td id="S4.T6.1.4.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">6.18</td>
<td id="S4.T6.1.4.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">40.6</td>
<td id="S4.T6.1.4.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">5.3</td>
</tr>
<tr id="S4.T6.1.5" class="ltx_tr">
<td id="S4.T6.1.5.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;">W/o Adaption</td>
<td id="S4.T6.1.5.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;">6.17</td>
<td id="S4.T6.1.5.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;">39.8</td>
<td id="S4.T6.1.5.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T6.1.5.4.1" class="ltx_text ltx_font_bold">5.6</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Ablation study results show that removing retrieval augmentation is indicated by ”W/o RAG” and removing adaptive rewriting by ”W/o Adaption” in <span id="S4.T6.4.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>. “General Align.” and “Know. Ab.” denotes general alignment ability and Knowledge Ability, which are the average results. FS denotes Factuality Score. <span id="S4.T6.5.2" class="ltx_text ltx_font_bold">Bold</span> denotes the best.</figcaption>
</figure>
</section>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span>Analysis</h3>

<section id="S4.SS5.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.5.1 </span>Datasets Analysis</h4>

<div id="S4.SS5.SSS1.p1" class="ltx_para">
<p id="S4.SS5.SSS1.p1.1" class="ltx_p">First, we compare the change in the length of responses (i.e., the number of tokens) between the original datasets and the addition of <span id="S4.SS5.SSS1.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>, finding that <span id="S4.SS5.SSS1.p1.1.2" class="ltx_text ltx_font_typewriter">Open-Platypus</span> becomes shorter and <span id="S4.SS5.SSS1.p1.1.3" class="ltx_text ltx_font_typewriter">No Robots</span> does not change much, while <span id="S4.SS5.SSS1.p1.1.4" class="ltx_text ltx_font_typewriter">Alpaca</span>, <span id="S4.SS5.SSS1.p1.1.5" class="ltx_text ltx_font_typewriter">GSM8K</span>, and <span id="S4.SS5.SSS1.p1.1.6" class="ltx_text ltx_font_typewriter">MATH</span> become longer (see Tab. <a href="#S4.T5" title="Table 5 ‣ ReAlign Can Improve Readability. ‣ 4.4 Results ‣ 4 Experiments ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>).
Second, we calculate the percentage of responses for which the adaptive rewriting method selects rewrite by edit distance (the results are shown in Tab <a href="#S4.T5" title="Table 5 ‣ ReAlign Can Improve Readability. ‣ 4.4 Results ‣ 4 Experiments ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>). Specifically, we compute the edit distance (including substitution, deletion, and insertion) on a word basis, then divide the edit distance by the length of the longest of the original and rewritten responses to obtain the edit rate, and finally record those with an edit rate greater than 0.2 as rewritten, and the rest as unrewritten. For <span id="S4.SS5.SSS1.p1.1.7" class="ltx_text ltx_font_typewriter">GSM8K</span> and <span id="S4.SS5.SSS1.p1.1.8" class="ltx_text ltx_font_typewriter">MATH</span> datasets, all data are ReAligned as adaptive rewriting was not applied to them.</p>
</div>
</section>
<section id="S4.SS5.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.5.2 </span>Alignment Tax</h4>

<div id="S4.SS5.SSS2.p1" class="ltx_para">
<p id="S4.SS5.SSS2.p1.1" class="ltx_p">When the model is fine-tuned on the <span id="S4.SS5.SSS2.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> dataset, a question worth exploring is whether there is a drop in knowledge ability even as alignment ability improves.
To evaluate the knowledge ability, we follow <cite class="ltx_cite ltx_citemacro_cite">Mitra et al. (<a href="#bib.bib23" title="" class="ltx_ref">2023</a>)</cite> to employ the following benchmarks: <span id="S4.SS5.SSS2.p1.1.2" class="ltx_text ltx_font_bold">Big Bench Hard (BBH)</span> <cite class="ltx_cite ltx_citemacro_cite">Suzgun et al. (<a href="#bib.bib31" title="" class="ltx_ref">2022</a>)</cite> and <span id="S4.SS5.SSS2.p1.1.3" class="ltx_text ltx_font_bold">AGIEval</span> <cite class="ltx_cite ltx_citemacro_cite">Zhong et al. (<a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite>, which is multiple choices knowledge-intensive QA task.
As shown in Fig. <a href="#S4.F6" title="Figure 6 ‣ ReAlign Can Improve Readability. ‣ 4.4 Results ‣ 4 Experiments ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, we can see that <span id="S4.SS5.SSS2.p1.1.4" class="ltx_text ltx_font_smallcaps">ReAlign</span> has little effect on the knowledge-based tasks, indicating that our approach does not impair the knowledge in the original dataset.
It is worth noting that in some cases <span id="S4.SS5.SSS2.p1.1.5" class="ltx_text ltx_font_smallcaps">ReAlign</span> will also provide a significant boost to knowledge, such as <span id="S4.SS5.SSS2.p1.1.6" class="ltx_text ltx_font_typewriter">Open-Platypus</span> on AGIEval. Possible reasons are that a well-defined format can facilitate the accuracy of the knowledge-based tasks <cite class="ltx_cite ltx_citemacro_cite">Wei et al. (<a href="#bib.bib38" title="" class="ltx_ref">2022b</a>)</cite> and that retrieving external information can augment knowledge.</p>
</div>
<figure id="S4.F7" class="ltx_figure"><img src="/html/2402.12219/assets/x10.png" id="S4.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="323" height="222" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>The scaling trends in <span id="S4.F7.3.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> data percentage, including general alignment ability and knowledge ability. We conduct the experiment in the <span id="S4.F7.4.2" class="ltx_text ltx_font_typewriter">Alpaca</span> dataset based on LLaMA-2-13B.</figcaption>
</figure>
</section>
<section id="S4.SS5.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.5.3 </span>Ablation Studies</h4>

<div id="S4.SS5.SSS3.p1" class="ltx_para">
<p id="S4.SS5.SSS3.p1.1" class="ltx_p">We rewrite two variants of the <span id="S4.SS5.SSS3.p1.1.1" class="ltx_text ltx_font_typewriter">Open-Platypus</span> dataset and train them based on LLaMA-2-13B for ablation studies:</p>
</div>
<div id="S4.SS5.SSS3.p2" class="ltx_para">
<ol id="S4.I1" class="ltx_enumerate">
<li id="S4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">(1)</span> 
<div id="S4.I1.i1.p1" class="ltx_para">
<p id="S4.I1.i1.p1.1" class="ltx_p"><span id="S4.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">W/o Retrieval Augmentation</span>: We remove the retrieval augmentation from <span id="S4.I1.i1.p1.1.2" class="ltx_text ltx_font_smallcaps">ReAlign</span> and rewrite all tasks without evidences. As shown in Tab. <a href="#S4.T6" title="Table 6 ‣ ReAlign Can Improve Readability. ‣ 4.4 Results ‣ 4 Experiments ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, the general alignment ability, knowledge ability, and factuality score (FS) are reduced, indicating the effectiveness of retrieval augmentation. Interestingly, the FS metrics are higher without RAG than in the original dataset, suggesting that <span id="S4.I1.i1.p1.1.3" class="ltx_text ltx_font_smallcaps">ReAlign</span> also has the potential to improve the factuality.</p>
</div>
</li>
<li id="S4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">(2)</span> 
<div id="S4.I1.i2.p1" class="ltx_para">
<p id="S4.I1.i2.p1.1" class="ltx_p"><span id="S4.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">W/o Adaption</span>: We remove the adaptive rewriting from <span id="S4.I1.i2.p1.1.2" class="ltx_text ltx_font_smallcaps">ReAlign</span> and use force rewriting. Tab. <a href="#S4.T6" title="Table 6 ‣ ReAlign Can Improve Readability. ‣ 4.4 Results ‣ 4 Experiments ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> shows the general alignment and knowledge ability decrease. This may be because forced rewriting, while making the responses more structured, ignores the question’s requirements, weakening the instruction-following ability. In addition, FS has increased, probably because forced rewriting leads to a larger amount of <span id="S4.I1.i2.p1.1.3" class="ltx_text ltx_font_smallcaps">ReAlign</span> data, introducing more retrieved knowledge and boosting factuality.</p>
</div>
</li>
</ol>
</div>
</section>
<section id="S4.SS5.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.5.4 </span>The Scaling Law of <span id="S4.SS5.SSS4.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>
</h4>

<div id="S4.SS5.SSS4.p1" class="ltx_para">
<p id="S4.SS5.SSS4.p1.5" class="ltx_p">We experiment to explore the impact of the number of <span id="S4.SS5.SSS4.p1.5.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>. Specifically, we randomly sample a <math id="S4.SS5.SSS4.p1.1.m1.1" class="ltx_Math" alttext="k\%" display="inline"><semantics id="S4.SS5.SSS4.p1.1.m1.1a"><mrow id="S4.SS5.SSS4.p1.1.m1.1.1" xref="S4.SS5.SSS4.p1.1.m1.1.1.cmml"><mi id="S4.SS5.SSS4.p1.1.m1.1.1.2" xref="S4.SS5.SSS4.p1.1.m1.1.1.2.cmml">k</mi><mo id="S4.SS5.SSS4.p1.1.m1.1.1.1" xref="S4.SS5.SSS4.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.SSS4.p1.1.m1.1b"><apply id="S4.SS5.SSS4.p1.1.m1.1.1.cmml" xref="S4.SS5.SSS4.p1.1.m1.1.1"><csymbol cd="latexml" id="S4.SS5.SSS4.p1.1.m1.1.1.1.cmml" xref="S4.SS5.SSS4.p1.1.m1.1.1.1">percent</csymbol><ci id="S4.SS5.SSS4.p1.1.m1.1.1.2.cmml" xref="S4.SS5.SSS4.p1.1.m1.1.1.2">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.SSS4.p1.1.m1.1c">k\%</annotation></semantics></math> (<math id="S4.SS5.SSS4.p1.2.m2.5" class="ltx_Math" alttext="k=0,5,10,20,\text{Full}" display="inline"><semantics id="S4.SS5.SSS4.p1.2.m2.5a"><mrow id="S4.SS5.SSS4.p1.2.m2.5.6" xref="S4.SS5.SSS4.p1.2.m2.5.6.cmml"><mi id="S4.SS5.SSS4.p1.2.m2.5.6.2" xref="S4.SS5.SSS4.p1.2.m2.5.6.2.cmml">k</mi><mo id="S4.SS5.SSS4.p1.2.m2.5.6.1" xref="S4.SS5.SSS4.p1.2.m2.5.6.1.cmml">=</mo><mrow id="S4.SS5.SSS4.p1.2.m2.5.6.3.2" xref="S4.SS5.SSS4.p1.2.m2.5.6.3.1.cmml"><mn id="S4.SS5.SSS4.p1.2.m2.1.1" xref="S4.SS5.SSS4.p1.2.m2.1.1.cmml">0</mn><mo id="S4.SS5.SSS4.p1.2.m2.5.6.3.2.1" xref="S4.SS5.SSS4.p1.2.m2.5.6.3.1.cmml">,</mo><mn id="S4.SS5.SSS4.p1.2.m2.2.2" xref="S4.SS5.SSS4.p1.2.m2.2.2.cmml">5</mn><mo id="S4.SS5.SSS4.p1.2.m2.5.6.3.2.2" xref="S4.SS5.SSS4.p1.2.m2.5.6.3.1.cmml">,</mo><mn id="S4.SS5.SSS4.p1.2.m2.3.3" xref="S4.SS5.SSS4.p1.2.m2.3.3.cmml">10</mn><mo id="S4.SS5.SSS4.p1.2.m2.5.6.3.2.3" xref="S4.SS5.SSS4.p1.2.m2.5.6.3.1.cmml">,</mo><mn id="S4.SS5.SSS4.p1.2.m2.4.4" xref="S4.SS5.SSS4.p1.2.m2.4.4.cmml">20</mn><mo id="S4.SS5.SSS4.p1.2.m2.5.6.3.2.4" xref="S4.SS5.SSS4.p1.2.m2.5.6.3.1.cmml">,</mo><mtext id="S4.SS5.SSS4.p1.2.m2.5.5" xref="S4.SS5.SSS4.p1.2.m2.5.5a.cmml">Full</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.SSS4.p1.2.m2.5b"><apply id="S4.SS5.SSS4.p1.2.m2.5.6.cmml" xref="S4.SS5.SSS4.p1.2.m2.5.6"><eq id="S4.SS5.SSS4.p1.2.m2.5.6.1.cmml" xref="S4.SS5.SSS4.p1.2.m2.5.6.1"></eq><ci id="S4.SS5.SSS4.p1.2.m2.5.6.2.cmml" xref="S4.SS5.SSS4.p1.2.m2.5.6.2">𝑘</ci><list id="S4.SS5.SSS4.p1.2.m2.5.6.3.1.cmml" xref="S4.SS5.SSS4.p1.2.m2.5.6.3.2"><cn type="integer" id="S4.SS5.SSS4.p1.2.m2.1.1.cmml" xref="S4.SS5.SSS4.p1.2.m2.1.1">0</cn><cn type="integer" id="S4.SS5.SSS4.p1.2.m2.2.2.cmml" xref="S4.SS5.SSS4.p1.2.m2.2.2">5</cn><cn type="integer" id="S4.SS5.SSS4.p1.2.m2.3.3.cmml" xref="S4.SS5.SSS4.p1.2.m2.3.3">10</cn><cn type="integer" id="S4.SS5.SSS4.p1.2.m2.4.4.cmml" xref="S4.SS5.SSS4.p1.2.m2.4.4">20</cn><ci id="S4.SS5.SSS4.p1.2.m2.5.5a.cmml" xref="S4.SS5.SSS4.p1.2.m2.5.5"><mtext id="S4.SS5.SSS4.p1.2.m2.5.5.cmml" xref="S4.SS5.SSS4.p1.2.m2.5.5">Full</mtext></ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.SSS4.p1.2.m2.5c">k=0,5,10,20,\text{Full}</annotation></semantics></math>, with <span id="S4.SS5.SSS4.p1.5.2" class="ltx_text ltx_markedasmath">Full</span> being <math id="S4.SS5.SSS4.p1.4.m4.1" class="ltx_Math" alttext="29.9\%" display="inline"><semantics id="S4.SS5.SSS4.p1.4.m4.1a"><mrow id="S4.SS5.SSS4.p1.4.m4.1.1" xref="S4.SS5.SSS4.p1.4.m4.1.1.cmml"><mn id="S4.SS5.SSS4.p1.4.m4.1.1.2" xref="S4.SS5.SSS4.p1.4.m4.1.1.2.cmml">29.9</mn><mo id="S4.SS5.SSS4.p1.4.m4.1.1.1" xref="S4.SS5.SSS4.p1.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.SSS4.p1.4.m4.1b"><apply id="S4.SS5.SSS4.p1.4.m4.1.1.cmml" xref="S4.SS5.SSS4.p1.4.m4.1.1"><csymbol cd="latexml" id="S4.SS5.SSS4.p1.4.m4.1.1.1.cmml" xref="S4.SS5.SSS4.p1.4.m4.1.1.1">percent</csymbol><cn type="float" id="S4.SS5.SSS4.p1.4.m4.1.1.2.cmml" xref="S4.SS5.SSS4.p1.4.m4.1.1.2">29.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.SSS4.p1.4.m4.1c">29.9\%</annotation></semantics></math>) of <span id="S4.SS5.SSS4.p1.5.3" class="ltx_text ltx_font_smallcaps">ReAlign</span> <span id="S4.SS5.SSS4.p1.5.4" class="ltx_text ltx_font_typewriter">Alpaca</span> data, and fill in the remainder with original responses. The original Alpaca dataset corresponds to <math id="S4.SS5.SSS4.p1.5.m5.1" class="ltx_Math" alttext="0\%" display="inline"><semantics id="S4.SS5.SSS4.p1.5.m5.1a"><mrow id="S4.SS5.SSS4.p1.5.m5.1.1" xref="S4.SS5.SSS4.p1.5.m5.1.1.cmml"><mn id="S4.SS5.SSS4.p1.5.m5.1.1.2" xref="S4.SS5.SSS4.p1.5.m5.1.1.2.cmml">0</mn><mo id="S4.SS5.SSS4.p1.5.m5.1.1.1" xref="S4.SS5.SSS4.p1.5.m5.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.SSS4.p1.5.m5.1b"><apply id="S4.SS5.SSS4.p1.5.m5.1.1.cmml" xref="S4.SS5.SSS4.p1.5.m5.1.1"><csymbol cd="latexml" id="S4.SS5.SSS4.p1.5.m5.1.1.1.cmml" xref="S4.SS5.SSS4.p1.5.m5.1.1.1">percent</csymbol><cn type="integer" id="S4.SS5.SSS4.p1.5.m5.1.1.2.cmml" xref="S4.SS5.SSS4.p1.5.m5.1.1.2">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.SSS4.p1.5.m5.1c">0\%</annotation></semantics></math>.
Interestingly, we find that only 5% of the <span id="S4.SS5.SSS4.p1.5.5" class="ltx_text ltx_font_smallcaps">ReAlign</span> data yields a 67% boost in general alignment ability compared to the entire <span id="S4.SS5.SSS4.p1.5.6" class="ltx_text ltx_font_smallcaps">ReAlign</span> data (see Fig. <a href="#S4.F7" title="Figure 7 ‣ 4.5.2 Alignment Tax ‣ 4.5 Analysis ‣ 4 Experiments ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>).
This suggests that only a small amount of data is required to learn style and format, to expose the knowledge and capabilities that were already acquired during pretraining <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite>. Additionally, the knowledge capability continues to improve as the amount of <span id="S4.SS5.SSS4.p1.5.7" class="ltx_text ltx_font_smallcaps">ReAlign</span> data improves.</p>
</div>
</section>
<section id="S4.SS5.SSS5" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.5.5 </span>Case Study</h4>

<div id="S4.SS5.SSS5.p1" class="ltx_para">
<p id="S4.SS5.SSS5.p1.1" class="ltx_p">We show a case from the MT-Bench test set in Tab. <a href="#A4.T7" title="Table 7 ‣ Appendix D The Description of Auto-J ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>. This example shows that the response given by the <span id="S4.SS5.SSS5.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> model has a better format.</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">In this work, we propose <span id="S5.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span>, a simple and effective method for alignment, which automatically improves the quality of the existing instruction datasets while minimizing labor costs and hallucinations. We create five new high-quality datasets from <span id="S5.p1.1.2" class="ltx_text ltx_font_typewriter">Open-Platypus</span> <cite class="ltx_cite ltx_citemacro_cite">Lee et al. (<a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>, <span id="S5.p1.1.3" class="ltx_text ltx_font_typewriter">No Robots</span> <cite class="ltx_cite ltx_citemacro_cite">Rajani et al. (<a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite>, <span id="S5.p1.1.4" class="ltx_text ltx_font_typewriter">Alpaca</span> <cite class="ltx_cite ltx_citemacro_cite">Taori et al. (<a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite>, <span id="S5.p1.1.5" class="ltx_text ltx_font_typewriter">GSM8K</span> <cite class="ltx_cite ltx_citemacro_cite">Cobbe et al. (<a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite>, and <span id="S5.p1.1.6" class="ltx_text ltx_font_typewriter">MATH</span> <cite class="ltx_cite ltx_citemacro_cite">Hendrycks et al. (<a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite> and high-quality manual-written natural language formats. Experiments demonstrate that <span id="S5.p1.1.7" class="ltx_text ltx_font_smallcaps">ReAlign</span> significantly boosts general alignment ability, math reasoning, factuality, and readability without impairing knowledge ability. Last but not least, we release the code and data to facilitate future research.</p>
</div>
</section>
<section id="Sx1" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">Limitations</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">First, our approach relies on the ability to reformatting models, which is currently less effective in open-source models (e.g., LLaMA2 <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite>) but more costly in closed-source models (e.g., GPT-4 <cite class="ltx_cite ltx_citemacro_cite">OpenAI (<a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>).
Second, the task categories we define cannot cover all tasks in reality, as real questions may be more complex and involve multiple tasks. Therefore, it is necessary to define more tasks and formats for a wide range of diverse and regional scenarios.
Last, applying <span id="Sx1.p1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> only to single-turn conversations has the potential to hurt the alignment ability of the second-turn conversations, hence extending <span id="Sx1.p1.1.2" class="ltx_text ltx_font_smallcaps">ReAlign</span> to multi-turn conversation would also be valuable.</p>
</div>
</section>
<section id="Sx2" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">Ethics Statement</h2>

<div id="Sx2.p1" class="ltx_para">
<p id="Sx2.p1.1" class="ltx_p">We take ethical considerations very seriously. In this paper, both the datasets and models are publicly available and have been widely adopted by researchers. We ensure that the findings and conclusions of this paper are reported accurately and objectively.</p>
</div>
</section>
<section id="Sx3" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">Acknowledgements</h2>

<div id="Sx3.p1" class="ltx_para">
<p id="Sx3.p1.1" class="ltx_p">We thank the GAIR members for reviewing our paper and giving valuable feedback. We appreciate the authors in <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib34" title="" class="ltx_ref">2023a</a>)</cite> for providing the training codebase and the helpfulness.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography" lang="en">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html" title="" class="ltx_ref ltx_href">Language models are few-shot learners</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cao et al. (2023)</span>
<span class="ltx_bibblock">
Yihan Cao, Yanbin Kang, and Lichao Sun. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2307.06290" title="" class="ltx_ref ltx_href">Instruction mining: High-quality instruction data selection for large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2023)</span>
<span class="ltx_bibblock">
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2307.08701" title="" class="ltx_ref ltx_href">Alpagasus: Training a better alpaca with fewer data</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chern et al. (2023)</span>
<span class="ltx_bibblock">
I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2307.13528" title="" class="ltx_ref ltx_href">Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.13528</em>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chiang et al. (2023)</span>
<span class="ltx_bibblock">
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://lmsys.org/blog/2023-03-30-vicuna/" title="" class="ltx_ref ltx_href">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. (2021)</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2110.14168" title="" class="ltx_ref ltx_href">Training verifiers to solve math word problems</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2110.14168</em>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Granziol et al. (2022)</span>
<span class="ltx_bibblock">
Diego Granziol, Stefan Zohren, and Stephen Roberts. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://jmlr.org/papers/v23/20-1258.html" title="" class="ltx_ref ltx_href">Learning rates as a function of batch size: A random matrix theory approach to neural network training</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">J. Mach. Learn. Res.</em>, 23:173:1–173:65.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gudibande et al. (2023)</span>
<span class="ltx_bibblock">
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2305.15717" title="" class="ltx_ref ltx_href">The false promise of imitating proprietary llms</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.

</span>
<span class="ltx_bibblock">Measuring mathematical problem solving with the math dataset.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Honovich et al. (2023)</span>
<span class="ltx_bibblock">
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.acl-long.806" title="" class="ltx_ref ltx_href">Unnatural instructions: Tuning language models with (almost) no human labor</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">ACL</em>, pages 14409–14428, Toronto, Canada. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ji et al. (2023)</span>
<span class="ltx_bibblock">
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/3571730" title="" class="ltx_ref ltx_href">Survey of hallucination in natural language generation</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">ACM Comput. Surv.</em>, 55(12):248:1–248:38.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2023)</span>
<span class="ltx_bibblock">
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2310.06825" title="" class="ltx_ref ltx_href">Mistral 7b</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.06825</em>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kwiatkowski et al. (2019)</span>
<span class="ltx_bibblock">
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1162/tacl_a_00276" title="" class="ltx_ref ltx_href">Natural questions: A benchmark for question answering research</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>, 7:453–466.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Köpf et al. (2023)</span>
<span class="ltx_bibblock">
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2304.07327" title="" class="ltx_ref ltx_href">Openassistant conversations – democratizing large language model alignment</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2023)</span>
<span class="ltx_bibblock">
Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2308.07317" title="" class="ltx_ref ltx_href">Platypus: Quick, cheap, and powerful refinement of llms</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2024)</span>
<span class="ltx_bibblock">
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, hai zhao, and Pengfei Liu. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=gtkFw6sZGS" title="" class="ltx_ref ltx_href">Generative judge for evaluating alignment</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">The Twelfth International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023a)</span>
<span class="ltx_bibblock">
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2308.12032" title="" class="ltx_ref ltx_href">From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023b)</span>
<span class="ltx_bibblock">
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. 2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2308.06259" title="" class="ltx_ref ltx_href">Self-alignment with instruction backtranslation</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023c)</span>
<span class="ltx_bibblock">
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023c.

</span>
<span class="ltx_bibblock">Alpacaeval: An automatic evaluator of instruction-following models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/tatsu-lab/alpaca_eval" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tatsu-lab/alpaca_eval</a>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023)</span>
<span class="ltx_bibblock">
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2312.15685" title="" class="ltx_ref ltx_href">What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.15685</em>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. (2023)</span>
<span class="ltx_bibblock">
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2308.07074" title="" class="ltx_ref ltx_href">#instag: Instruction tagging for analyzing supervised fine-tuning of large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mishra et al. (2022)</span>
<span class="ltx_bibblock">
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.acl-long.244" title="" class="ltx_ref ltx_href">Cross-task generalization via natural language crowdsourcing instructions</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">ACL</em>, pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mitra et al. (2023)</span>
<span class="ltx_bibblock">
Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2311.11045" title="" class="ltx_ref ltx_href">Orca 2: Teaching small language models how to reason</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.11045</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2023)</span>
<span class="ltx_bibblock">
OpenAI. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2303.08774" title="" class="ltx_ref ltx_href">Gpt-4 technical report</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ouyang et al. (2022)</span>
<span class="ltx_bibblock">
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html" title="" class="ltx_ref ltx_href">Training language models to follow instructions with human feedback</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peng et al. (2023)</span>
<span class="ltx_bibblock">
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2304.03277" title="" class="ltx_ref ltx_href">Instruction tuning with gpt-4</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Petroni et al. (2021)</span>
<span class="ltx_bibblock">
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.naacl-main.200" title="" class="ltx_ref ltx_href">KILT: a benchmark for knowledge intensive language tasks</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em>, pages 2523–2544, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rajani et al. (2023)</span>
<span class="ltx_bibblock">
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. 2023.

</span>
<span class="ltx_bibblock">No robots.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://huggingface.co/datasets/HuggingFaceH4/no_robots" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/HuggingFaceH4/no_robots</a>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sanh et al. (2022)</span>
<span class="ltx_bibblock">
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2110.08207" title="" class="ltx_ref ltx_href">Multitask prompted training enables zero-shot task generalization</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. (2024)</span>
<span class="ltx_bibblock">
Shichao Sun, Junlong Li, Weizhe Yuan, Ruifeng Yuan, Wenjie Li, and Pengfei Liu. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2401.04518" title="" class="ltx_ref ltx_href">The critique of critique</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.04518</em>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Suzgun et al. (2022)</span>
<span class="ltx_bibblock">
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2210.09261" title="" class="ltx_ref ltx_href">Challenging big-bench tasks and whether chain-of-thought can solve them</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.09261</em>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Taori et al. (2023)</span>
<span class="ltx_bibblock">
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.

</span>
<span class="ltx_bibblock">Stanford alpaca: An instruction-following llama model.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/tatsu-lab/stanford_alpaca" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tatsu-lab/stanford_alpaca</a>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas
Scialom. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2307.09288" title="" class="ltx_ref ltx_href">Llama 2: Open foundation and fine-tuned chat models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023a)</span>
<span class="ltx_bibblock">
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2309.11235" title="" class="ltx_ref ltx_href">Openchat: Advancing open-source language models with mixed-quality data</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.11235</em>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023b)</span>
<span class="ltx_bibblock">
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.acl-long.754" title="" class="ltx_ref ltx_href">Self-instruct: Aligning language models with self-generated instructions</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">ACL</em>, pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022)</span>
<span class="ltx_bibblock">
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2204.07705" title="" class="ltx_ref ltx_href">Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022a)</span>
<span class="ltx_bibblock">
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2109.01652" title="" class="ltx_ref ltx_href">Finetuned language models are zero-shot learners</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022b)</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html" title="" class="ltx_ref ltx_href">Chain-of-thought prompting elicits reasoning in large language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2023)</span>
<span class="ltx_bibblock">
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2304.12244" title="" class="ltx_ref ltx_href">Wizardlm: Empowering large language models to follow complex instructions</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yuan et al. (2024)</span>
<span class="ltx_bibblock">
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2401.10020" title="" class="ltx_ref ltx_href">Self-rewarding language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.10020</em>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. (2023)</span>
<span class="ltx_bibblock">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2306.05685" title="" class="ltx_ref ltx_href">Judging llm-as-a-judge with mt-bench and chatbot arena</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2023)</span>
<span class="ltx_bibblock">
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2304.06364" title="" class="ltx_ref ltx_href">Agieval: A human-centric benchmark for evaluating foundation models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.06364</em>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2023)</span>
<span class="ltx_bibblock">
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2305.11206" title="" class="ltx_ref ltx_href">Lima: Less is more for alignment</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">arXiv Preprint</em>.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Implementation Details</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.14" class="ltx_p">For retrieval augmentation, we select the top-<math id="A1.p1.1.m1.1" class="ltx_Math" alttext="5" display="inline"><semantics id="A1.p1.1.m1.1a"><mn id="A1.p1.1.m1.1.1" xref="A1.p1.1.m1.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="A1.p1.1.m1.1b"><cn type="integer" id="A1.p1.1.m1.1.1.cmml" xref="A1.p1.1.m1.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.1.m1.1c">5</annotation></semantics></math> evidence from the retrieval results. For reformatting, We guide gpt-3.5-turbo-1106 to reformat the responses. We set the temperature as <math id="A1.p1.2.m2.1" class="ltx_Math" alttext="0.3" display="inline"><semantics id="A1.p1.2.m2.1a"><mn id="A1.p1.2.m2.1.1" xref="A1.p1.2.m2.1.1.cmml">0.3</mn><annotation-xml encoding="MathML-Content" id="A1.p1.2.m2.1b"><cn type="float" id="A1.p1.2.m2.1.1.cmml" xref="A1.p1.2.m2.1.1">0.3</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.2.m2.1c">0.3</annotation></semantics></math>, the top-p as <math id="A1.p1.3.m3.1" class="ltx_Math" alttext="0.1" display="inline"><semantics id="A1.p1.3.m3.1a"><mn id="A1.p1.3.m3.1.1" xref="A1.p1.3.m3.1.1.cmml">0.1</mn><annotation-xml encoding="MathML-Content" id="A1.p1.3.m3.1b"><cn type="float" id="A1.p1.3.m3.1.1.cmml" xref="A1.p1.3.m3.1.1">0.1</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.3.m3.1c">0.1</annotation></semantics></math>, and the target length as <math id="A1.p1.4.m4.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="A1.p1.4.m4.1a"><mn id="A1.p1.4.m4.1.1" xref="A1.p1.4.m4.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="A1.p1.4.m4.1b"><cn type="integer" id="A1.p1.4.m4.1.1.cmml" xref="A1.p1.4.m4.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.4.m4.1c">2048</annotation></semantics></math>. Additionally, we generate two rewrite results at a time and choose the longest one, hence <math id="A1.p1.5.m5.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A1.p1.5.m5.1a"><mi id="A1.p1.5.m5.1.1" xref="A1.p1.5.m5.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A1.p1.5.m5.1b"><ci id="A1.p1.5.m5.1.1.cmml" xref="A1.p1.5.m5.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.5.m5.1c">n</annotation></semantics></math> is set to <math id="A1.p1.6.m6.1" class="ltx_Math" alttext="2" display="inline"><semantics id="A1.p1.6.m6.1a"><mn id="A1.p1.6.m6.1.1" xref="A1.p1.6.m6.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="A1.p1.6.m6.1b"><cn type="integer" id="A1.p1.6.m6.1.1.cmml" xref="A1.p1.6.m6.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.6.m6.1c">2</annotation></semantics></math>.
For training, we fine-tune the models based on the LLaMA-2-13B <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite> and Mistral-7B <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a href="#bib.bib12" title="" class="ltx_ref">2023</a>)</cite> for 5 epochs on the <span id="A1.p1.14.1" class="ltx_text ltx_font_typewriter">Open-Platypus</span> dataset, 20 epochs on the <span id="A1.p1.14.2" class="ltx_text ltx_font_typewriter">No Robots</span> and <span id="A1.p1.14.3" class="ltx_text ltx_font_typewriter">Alpaca</span> datasets, and 3 epochs on the <span id="A1.p1.14.4" class="ltx_text ltx_font_typewriter">GSM8K</span> and <span id="A1.p1.14.5" class="ltx_text ltx_font_typewriter">MATH</span> datasets, using the AdamW optimizer with a sequence length of 4,096 tokens. The batch size is 64 for the <span id="A1.p1.14.6" class="ltx_text ltx_font_typewriter">Open-Platypus</span>, <span id="A1.p1.14.7" class="ltx_text ltx_font_typewriter">No Robots</span>, and <span id="A1.p1.14.8" class="ltx_text ltx_font_typewriter">Alpaca</span> datasets, and 128 for the <span id="A1.p1.14.9" class="ltx_text ltx_font_typewriter">GSM8K</span> and <span id="A1.p1.14.10" class="ltx_text ltx_font_typewriter">MATH</span> datasets. The AdamW optimizer’s hyperparameters are set as follows: <math id="A1.p1.7.m7.1" class="ltx_Math" alttext="\beta_{1}=0.9" display="inline"><semantics id="A1.p1.7.m7.1a"><mrow id="A1.p1.7.m7.1.1" xref="A1.p1.7.m7.1.1.cmml"><msub id="A1.p1.7.m7.1.1.2" xref="A1.p1.7.m7.1.1.2.cmml"><mi id="A1.p1.7.m7.1.1.2.2" xref="A1.p1.7.m7.1.1.2.2.cmml">β</mi><mn id="A1.p1.7.m7.1.1.2.3" xref="A1.p1.7.m7.1.1.2.3.cmml">1</mn></msub><mo id="A1.p1.7.m7.1.1.1" xref="A1.p1.7.m7.1.1.1.cmml">=</mo><mn id="A1.p1.7.m7.1.1.3" xref="A1.p1.7.m7.1.1.3.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.7.m7.1b"><apply id="A1.p1.7.m7.1.1.cmml" xref="A1.p1.7.m7.1.1"><eq id="A1.p1.7.m7.1.1.1.cmml" xref="A1.p1.7.m7.1.1.1"></eq><apply id="A1.p1.7.m7.1.1.2.cmml" xref="A1.p1.7.m7.1.1.2"><csymbol cd="ambiguous" id="A1.p1.7.m7.1.1.2.1.cmml" xref="A1.p1.7.m7.1.1.2">subscript</csymbol><ci id="A1.p1.7.m7.1.1.2.2.cmml" xref="A1.p1.7.m7.1.1.2.2">𝛽</ci><cn type="integer" id="A1.p1.7.m7.1.1.2.3.cmml" xref="A1.p1.7.m7.1.1.2.3">1</cn></apply><cn type="float" id="A1.p1.7.m7.1.1.3.cmml" xref="A1.p1.7.m7.1.1.3">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.7.m7.1c">\beta_{1}=0.9</annotation></semantics></math>, <math id="A1.p1.8.m8.1" class="ltx_Math" alttext="\beta_{2}=0.95" display="inline"><semantics id="A1.p1.8.m8.1a"><mrow id="A1.p1.8.m8.1.1" xref="A1.p1.8.m8.1.1.cmml"><msub id="A1.p1.8.m8.1.1.2" xref="A1.p1.8.m8.1.1.2.cmml"><mi id="A1.p1.8.m8.1.1.2.2" xref="A1.p1.8.m8.1.1.2.2.cmml">β</mi><mn id="A1.p1.8.m8.1.1.2.3" xref="A1.p1.8.m8.1.1.2.3.cmml">2</mn></msub><mo id="A1.p1.8.m8.1.1.1" xref="A1.p1.8.m8.1.1.1.cmml">=</mo><mn id="A1.p1.8.m8.1.1.3" xref="A1.p1.8.m8.1.1.3.cmml">0.95</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.8.m8.1b"><apply id="A1.p1.8.m8.1.1.cmml" xref="A1.p1.8.m8.1.1"><eq id="A1.p1.8.m8.1.1.1.cmml" xref="A1.p1.8.m8.1.1.1"></eq><apply id="A1.p1.8.m8.1.1.2.cmml" xref="A1.p1.8.m8.1.1.2"><csymbol cd="ambiguous" id="A1.p1.8.m8.1.1.2.1.cmml" xref="A1.p1.8.m8.1.1.2">subscript</csymbol><ci id="A1.p1.8.m8.1.1.2.2.cmml" xref="A1.p1.8.m8.1.1.2.2">𝛽</ci><cn type="integer" id="A1.p1.8.m8.1.1.2.3.cmml" xref="A1.p1.8.m8.1.1.2.3">2</cn></apply><cn type="float" id="A1.p1.8.m8.1.1.3.cmml" xref="A1.p1.8.m8.1.1.3">0.95</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.8.m8.1c">\beta_{2}=0.95</annotation></semantics></math>, <math id="A1.p1.9.m9.1" class="ltx_Math" alttext="\epsilon=10^{-5}" display="inline"><semantics id="A1.p1.9.m9.1a"><mrow id="A1.p1.9.m9.1.1" xref="A1.p1.9.m9.1.1.cmml"><mi id="A1.p1.9.m9.1.1.2" xref="A1.p1.9.m9.1.1.2.cmml">ϵ</mi><mo id="A1.p1.9.m9.1.1.1" xref="A1.p1.9.m9.1.1.1.cmml">=</mo><msup id="A1.p1.9.m9.1.1.3" xref="A1.p1.9.m9.1.1.3.cmml"><mn id="A1.p1.9.m9.1.1.3.2" xref="A1.p1.9.m9.1.1.3.2.cmml">10</mn><mrow id="A1.p1.9.m9.1.1.3.3" xref="A1.p1.9.m9.1.1.3.3.cmml"><mo id="A1.p1.9.m9.1.1.3.3a" xref="A1.p1.9.m9.1.1.3.3.cmml">−</mo><mn id="A1.p1.9.m9.1.1.3.3.2" xref="A1.p1.9.m9.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.9.m9.1b"><apply id="A1.p1.9.m9.1.1.cmml" xref="A1.p1.9.m9.1.1"><eq id="A1.p1.9.m9.1.1.1.cmml" xref="A1.p1.9.m9.1.1.1"></eq><ci id="A1.p1.9.m9.1.1.2.cmml" xref="A1.p1.9.m9.1.1.2">italic-ϵ</ci><apply id="A1.p1.9.m9.1.1.3.cmml" xref="A1.p1.9.m9.1.1.3"><csymbol cd="ambiguous" id="A1.p1.9.m9.1.1.3.1.cmml" xref="A1.p1.9.m9.1.1.3">superscript</csymbol><cn type="integer" id="A1.p1.9.m9.1.1.3.2.cmml" xref="A1.p1.9.m9.1.1.3.2">10</cn><apply id="A1.p1.9.m9.1.1.3.3.cmml" xref="A1.p1.9.m9.1.1.3.3"><minus id="A1.p1.9.m9.1.1.3.3.1.cmml" xref="A1.p1.9.m9.1.1.3.3"></minus><cn type="integer" id="A1.p1.9.m9.1.1.3.3.2.cmml" xref="A1.p1.9.m9.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.9.m9.1c">\epsilon=10^{-5}</annotation></semantics></math>, and weight decay of <math id="A1.p1.10.m10.1" class="ltx_Math" alttext="0.1" display="inline"><semantics id="A1.p1.10.m10.1a"><mn id="A1.p1.10.m10.1.1" xref="A1.p1.10.m10.1.1.cmml">0.1</mn><annotation-xml encoding="MathML-Content" id="A1.p1.10.m10.1b"><cn type="float" id="A1.p1.10.m10.1.1.cmml" xref="A1.p1.10.m10.1.1">0.1</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.10.m10.1c">0.1</annotation></semantics></math>. We employ a cosine learning rate schedule with a maximum learning rate of <math id="A1.p1.11.m11.1" class="ltx_Math" alttext="5.7\times 10^{-5}" display="inline"><semantics id="A1.p1.11.m11.1a"><mrow id="A1.p1.11.m11.1.1" xref="A1.p1.11.m11.1.1.cmml"><mn id="A1.p1.11.m11.1.1.2" xref="A1.p1.11.m11.1.1.2.cmml">5.7</mn><mo lspace="0.222em" rspace="0.222em" id="A1.p1.11.m11.1.1.1" xref="A1.p1.11.m11.1.1.1.cmml">×</mo><msup id="A1.p1.11.m11.1.1.3" xref="A1.p1.11.m11.1.1.3.cmml"><mn id="A1.p1.11.m11.1.1.3.2" xref="A1.p1.11.m11.1.1.3.2.cmml">10</mn><mrow id="A1.p1.11.m11.1.1.3.3" xref="A1.p1.11.m11.1.1.3.3.cmml"><mo id="A1.p1.11.m11.1.1.3.3a" xref="A1.p1.11.m11.1.1.3.3.cmml">−</mo><mn id="A1.p1.11.m11.1.1.3.3.2" xref="A1.p1.11.m11.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.11.m11.1b"><apply id="A1.p1.11.m11.1.1.cmml" xref="A1.p1.11.m11.1.1"><times id="A1.p1.11.m11.1.1.1.cmml" xref="A1.p1.11.m11.1.1.1"></times><cn type="float" id="A1.p1.11.m11.1.1.2.cmml" xref="A1.p1.11.m11.1.1.2">5.7</cn><apply id="A1.p1.11.m11.1.1.3.cmml" xref="A1.p1.11.m11.1.1.3"><csymbol cd="ambiguous" id="A1.p1.11.m11.1.1.3.1.cmml" xref="A1.p1.11.m11.1.1.3">superscript</csymbol><cn type="integer" id="A1.p1.11.m11.1.1.3.2.cmml" xref="A1.p1.11.m11.1.1.3.2">10</cn><apply id="A1.p1.11.m11.1.1.3.3.cmml" xref="A1.p1.11.m11.1.1.3.3"><minus id="A1.p1.11.m11.1.1.3.3.1.cmml" xref="A1.p1.11.m11.1.1.3.3"></minus><cn type="integer" id="A1.p1.11.m11.1.1.3.3.2.cmml" xref="A1.p1.11.m11.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.11.m11.1c">5.7\times 10^{-5}</annotation></semantics></math> for the <span id="A1.p1.14.11" class="ltx_text ltx_font_typewriter">Open-Platypus</span> dataset, <math id="A1.p1.12.m12.1" class="ltx_Math" alttext="6.25\times 10^{-5}" display="inline"><semantics id="A1.p1.12.m12.1a"><mrow id="A1.p1.12.m12.1.1" xref="A1.p1.12.m12.1.1.cmml"><mn id="A1.p1.12.m12.1.1.2" xref="A1.p1.12.m12.1.1.2.cmml">6.25</mn><mo lspace="0.222em" rspace="0.222em" id="A1.p1.12.m12.1.1.1" xref="A1.p1.12.m12.1.1.1.cmml">×</mo><msup id="A1.p1.12.m12.1.1.3" xref="A1.p1.12.m12.1.1.3.cmml"><mn id="A1.p1.12.m12.1.1.3.2" xref="A1.p1.12.m12.1.1.3.2.cmml">10</mn><mrow id="A1.p1.12.m12.1.1.3.3" xref="A1.p1.12.m12.1.1.3.3.cmml"><mo id="A1.p1.12.m12.1.1.3.3a" xref="A1.p1.12.m12.1.1.3.3.cmml">−</mo><mn id="A1.p1.12.m12.1.1.3.3.2" xref="A1.p1.12.m12.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.12.m12.1b"><apply id="A1.p1.12.m12.1.1.cmml" xref="A1.p1.12.m12.1.1"><times id="A1.p1.12.m12.1.1.1.cmml" xref="A1.p1.12.m12.1.1.1"></times><cn type="float" id="A1.p1.12.m12.1.1.2.cmml" xref="A1.p1.12.m12.1.1.2">6.25</cn><apply id="A1.p1.12.m12.1.1.3.cmml" xref="A1.p1.12.m12.1.1.3"><csymbol cd="ambiguous" id="A1.p1.12.m12.1.1.3.1.cmml" xref="A1.p1.12.m12.1.1.3">superscript</csymbol><cn type="integer" id="A1.p1.12.m12.1.1.3.2.cmml" xref="A1.p1.12.m12.1.1.3.2">10</cn><apply id="A1.p1.12.m12.1.1.3.3.cmml" xref="A1.p1.12.m12.1.1.3.3"><minus id="A1.p1.12.m12.1.1.3.3.1.cmml" xref="A1.p1.12.m12.1.1.3.3"></minus><cn type="integer" id="A1.p1.12.m12.1.1.3.3.2.cmml" xref="A1.p1.12.m12.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.12.m12.1c">6.25\times 10^{-5}</annotation></semantics></math> for the <span id="A1.p1.14.12" class="ltx_text ltx_font_typewriter">No Robots</span> dataset, <math id="A1.p1.13.m13.1" class="ltx_Math" alttext="6.55\times 10^{-5}" display="inline"><semantics id="A1.p1.13.m13.1a"><mrow id="A1.p1.13.m13.1.1" xref="A1.p1.13.m13.1.1.cmml"><mn id="A1.p1.13.m13.1.1.2" xref="A1.p1.13.m13.1.1.2.cmml">6.55</mn><mo lspace="0.222em" rspace="0.222em" id="A1.p1.13.m13.1.1.1" xref="A1.p1.13.m13.1.1.1.cmml">×</mo><msup id="A1.p1.13.m13.1.1.3" xref="A1.p1.13.m13.1.1.3.cmml"><mn id="A1.p1.13.m13.1.1.3.2" xref="A1.p1.13.m13.1.1.3.2.cmml">10</mn><mrow id="A1.p1.13.m13.1.1.3.3" xref="A1.p1.13.m13.1.1.3.3.cmml"><mo id="A1.p1.13.m13.1.1.3.3a" xref="A1.p1.13.m13.1.1.3.3.cmml">−</mo><mn id="A1.p1.13.m13.1.1.3.3.2" xref="A1.p1.13.m13.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.13.m13.1b"><apply id="A1.p1.13.m13.1.1.cmml" xref="A1.p1.13.m13.1.1"><times id="A1.p1.13.m13.1.1.1.cmml" xref="A1.p1.13.m13.1.1.1"></times><cn type="float" id="A1.p1.13.m13.1.1.2.cmml" xref="A1.p1.13.m13.1.1.2">6.55</cn><apply id="A1.p1.13.m13.1.1.3.cmml" xref="A1.p1.13.m13.1.1.3"><csymbol cd="ambiguous" id="A1.p1.13.m13.1.1.3.1.cmml" xref="A1.p1.13.m13.1.1.3">superscript</csymbol><cn type="integer" id="A1.p1.13.m13.1.1.3.2.cmml" xref="A1.p1.13.m13.1.1.3.2">10</cn><apply id="A1.p1.13.m13.1.1.3.3.cmml" xref="A1.p1.13.m13.1.1.3.3"><minus id="A1.p1.13.m13.1.1.3.3.1.cmml" xref="A1.p1.13.m13.1.1.3.3"></minus><cn type="integer" id="A1.p1.13.m13.1.1.3.3.2.cmml" xref="A1.p1.13.m13.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.13.m13.1c">6.55\times 10^{-5}</annotation></semantics></math> for the <span id="A1.p1.14.13" class="ltx_text ltx_font_typewriter">Alpaca</span> dataset, and <math id="A1.p1.14.m14.1" class="ltx_Math" alttext="1\times 10^{-5}" display="inline"><semantics id="A1.p1.14.m14.1a"><mrow id="A1.p1.14.m14.1.1" xref="A1.p1.14.m14.1.1.cmml"><mn id="A1.p1.14.m14.1.1.2" xref="A1.p1.14.m14.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A1.p1.14.m14.1.1.1" xref="A1.p1.14.m14.1.1.1.cmml">×</mo><msup id="A1.p1.14.m14.1.1.3" xref="A1.p1.14.m14.1.1.3.cmml"><mn id="A1.p1.14.m14.1.1.3.2" xref="A1.p1.14.m14.1.1.3.2.cmml">10</mn><mrow id="A1.p1.14.m14.1.1.3.3" xref="A1.p1.14.m14.1.1.3.3.cmml"><mo id="A1.p1.14.m14.1.1.3.3a" xref="A1.p1.14.m14.1.1.3.3.cmml">−</mo><mn id="A1.p1.14.m14.1.1.3.3.2" xref="A1.p1.14.m14.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.14.m14.1b"><apply id="A1.p1.14.m14.1.1.cmml" xref="A1.p1.14.m14.1.1"><times id="A1.p1.14.m14.1.1.1.cmml" xref="A1.p1.14.m14.1.1.1"></times><cn type="integer" id="A1.p1.14.m14.1.1.2.cmml" xref="A1.p1.14.m14.1.1.2">1</cn><apply id="A1.p1.14.m14.1.1.3.cmml" xref="A1.p1.14.m14.1.1.3"><csymbol cd="ambiguous" id="A1.p1.14.m14.1.1.3.1.cmml" xref="A1.p1.14.m14.1.1.3">superscript</csymbol><cn type="integer" id="A1.p1.14.m14.1.1.3.2.cmml" xref="A1.p1.14.m14.1.1.3.2">10</cn><apply id="A1.p1.14.m14.1.1.3.3.cmml" xref="A1.p1.14.m14.1.1.3.3"><minus id="A1.p1.14.m14.1.1.3.3.1.cmml" xref="A1.p1.14.m14.1.1.3.3"></minus><cn type="integer" id="A1.p1.14.m14.1.1.3.3.2.cmml" xref="A1.p1.14.m14.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.14.m14.1c">1\times 10^{-5}</annotation></semantics></math> for the <span id="A1.p1.14.14" class="ltx_text ltx_font_typewriter">GSM8K</span> and <span id="A1.p1.14.15" class="ltx_text ltx_font_typewriter">MATH</span> datasets, which decays to 10% of the maximum value. Following <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib34" title="" class="ltx_ref">2023a</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Granziol et al. (<a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite>, the learning rate is scaled proportionally to the square root of the batch size. All models are trained on 8 NVIDIA A100 80G GPUs.</p>
</div>
</section>
<section id="A2" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Task Description</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">The task descriptions mentioned in §<a href="#S3.SS1" title="3.1 Criteria Definition ‣ 3 ReAlign ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a> and whether they are retrieved and rewritten are exhibited in Tab. <a href="#A4.T8" title="Table 8 ‣ Appendix D The Description of Auto-J ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>.</p>
</div>
</section>
<section id="A3" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Training Details of Task Classifier</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">In this section, we describe the training details of the task classifier mentioned in §<a href="#S3.SS1" title="3.1 Criteria Definition ‣ 3 ReAlign ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>.</p>
</div>
<div id="A3.p2" class="ltx_para">
<p id="A3.p2.1" class="ltx_p">In real scenarios, user questions can be quite long and complex (with more than 1k words), while traditional BERT-like models only have a context length of 512 tokens, urging us to train a large language model for classification.
Following <cite class="ltx_cite ltx_citemacro_citet">Li et al. (<a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite>, we convert the classification task into a generation task, which directly generates the task name given a question with the prompt as shown in Tab. <a href="#A4.T10" title="Table 10 ‣ Appendix D The Description of Auto-J ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>. Specifically, we manually label about 33 questions for each kind of task from <span id="A3.p2.1.1" class="ltx_text ltx_font_typewriter">LIMA</span> <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite>, <span id="A3.p2.1.2" class="ltx_text ltx_font_typewriter">ShareGPT</span> <cite class="ltx_cite ltx_citemacro_cite">Zheng et al. (<a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite>, and <span id="A3.p2.1.3" class="ltx_text ltx_font_typewriter">Alpaca</span> <cite class="ltx_cite ltx_citemacro_cite">Taori et al. (<a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite> datasets. For tasks with less training data, we use ChatGPT to generate a portion of the questions.
Then, we divide questions in a 9:1 train/test split (Tab. <a href="#A4.T9" title="Table 9 ‣ Appendix D The Description of Auto-J ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>). We train the task classifier from LLaMA-2-13B <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite>, and set the max sequence length as 2,048, epochs as 20, and batch size as 16. We set the initial learning rate to 2e-5 and consine decaying to 0 by the end of training with warmup steps as 10. The optimizer is AdamW with <math id="A3.p2.1.m1.2" class="ltx_Math" alttext="\beta_{1}=0.9,\beta_{2}=0.95" display="inline"><semantics id="A3.p2.1.m1.2a"><mrow id="A3.p2.1.m1.2.2.2" xref="A3.p2.1.m1.2.2.3.cmml"><mrow id="A3.p2.1.m1.1.1.1.1" xref="A3.p2.1.m1.1.1.1.1.cmml"><msub id="A3.p2.1.m1.1.1.1.1.2" xref="A3.p2.1.m1.1.1.1.1.2.cmml"><mi id="A3.p2.1.m1.1.1.1.1.2.2" xref="A3.p2.1.m1.1.1.1.1.2.2.cmml">β</mi><mn id="A3.p2.1.m1.1.1.1.1.2.3" xref="A3.p2.1.m1.1.1.1.1.2.3.cmml">1</mn></msub><mo id="A3.p2.1.m1.1.1.1.1.1" xref="A3.p2.1.m1.1.1.1.1.1.cmml">=</mo><mn id="A3.p2.1.m1.1.1.1.1.3" xref="A3.p2.1.m1.1.1.1.1.3.cmml">0.9</mn></mrow><mo id="A3.p2.1.m1.2.2.2.3" xref="A3.p2.1.m1.2.2.3a.cmml">,</mo><mrow id="A3.p2.1.m1.2.2.2.2" xref="A3.p2.1.m1.2.2.2.2.cmml"><msub id="A3.p2.1.m1.2.2.2.2.2" xref="A3.p2.1.m1.2.2.2.2.2.cmml"><mi id="A3.p2.1.m1.2.2.2.2.2.2" xref="A3.p2.1.m1.2.2.2.2.2.2.cmml">β</mi><mn id="A3.p2.1.m1.2.2.2.2.2.3" xref="A3.p2.1.m1.2.2.2.2.2.3.cmml">2</mn></msub><mo id="A3.p2.1.m1.2.2.2.2.1" xref="A3.p2.1.m1.2.2.2.2.1.cmml">=</mo><mn id="A3.p2.1.m1.2.2.2.2.3" xref="A3.p2.1.m1.2.2.2.2.3.cmml">0.95</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.p2.1.m1.2b"><apply id="A3.p2.1.m1.2.2.3.cmml" xref="A3.p2.1.m1.2.2.2"><csymbol cd="ambiguous" id="A3.p2.1.m1.2.2.3a.cmml" xref="A3.p2.1.m1.2.2.2.3">formulae-sequence</csymbol><apply id="A3.p2.1.m1.1.1.1.1.cmml" xref="A3.p2.1.m1.1.1.1.1"><eq id="A3.p2.1.m1.1.1.1.1.1.cmml" xref="A3.p2.1.m1.1.1.1.1.1"></eq><apply id="A3.p2.1.m1.1.1.1.1.2.cmml" xref="A3.p2.1.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="A3.p2.1.m1.1.1.1.1.2.1.cmml" xref="A3.p2.1.m1.1.1.1.1.2">subscript</csymbol><ci id="A3.p2.1.m1.1.1.1.1.2.2.cmml" xref="A3.p2.1.m1.1.1.1.1.2.2">𝛽</ci><cn type="integer" id="A3.p2.1.m1.1.1.1.1.2.3.cmml" xref="A3.p2.1.m1.1.1.1.1.2.3">1</cn></apply><cn type="float" id="A3.p2.1.m1.1.1.1.1.3.cmml" xref="A3.p2.1.m1.1.1.1.1.3">0.9</cn></apply><apply id="A3.p2.1.m1.2.2.2.2.cmml" xref="A3.p2.1.m1.2.2.2.2"><eq id="A3.p2.1.m1.2.2.2.2.1.cmml" xref="A3.p2.1.m1.2.2.2.2.1"></eq><apply id="A3.p2.1.m1.2.2.2.2.2.cmml" xref="A3.p2.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="A3.p2.1.m1.2.2.2.2.2.1.cmml" xref="A3.p2.1.m1.2.2.2.2.2">subscript</csymbol><ci id="A3.p2.1.m1.2.2.2.2.2.2.cmml" xref="A3.p2.1.m1.2.2.2.2.2.2">𝛽</ci><cn type="integer" id="A3.p2.1.m1.2.2.2.2.2.3.cmml" xref="A3.p2.1.m1.2.2.2.2.2.3">2</cn></apply><cn type="float" id="A3.p2.1.m1.2.2.2.2.3.cmml" xref="A3.p2.1.m1.2.2.2.2.3">0.95</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.1.m1.2c">\beta_{1}=0.9,\beta_{2}=0.95</annotation></semantics></math>. The loss is only calculated on the output end as well. The accuracy and F1 of the final task classifier on the test set are 78.32% and 81.59%, respectively.</p>
</div>
</section>
<section id="A4" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>The Description of Auto-J</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">Auto-J <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite> is an open-source generative judge designed to evaluate LLMs based on their alignment with human preferences, which is the best critique model besides GPT-4 <cite class="ltx_cite ltx_citemacro_cite">Sun et al. (<a href="#bib.bib30" title="" class="ltx_ref">2024</a>)</cite>. Auto-J stands out due to its generality, being trained on real-world user queries and responses from various LLMs across 58 scenarios. It offers flexibility by enabling both pairwise comparison and single-response evaluation through prompt adjustments. Additionally, Auto-J enhances reliability and encourages human participation in the evaluation process by offering detailed natural language critiques, improving interpretability.</p>
</div>
<figure id="A4.T7" class="ltx_table">
<table id="A4.T7.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A4.T7.1.1" class="ltx_tr">
<td id="A4.T7.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A4.T7.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T7.1.1.1.1.1" class="ltx_p" style="width:620.0pt;"><span id="A4.T7.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Query</span>: Benjamin went to a bookstore and purchased a variety of books. He bought <math id="A4.T7.1.1.1.1.1.m1.1" class="ltx_Math" alttext="5" display="inline"><semantics id="A4.T7.1.1.1.1.1.m1.1a"><mn id="A4.T7.1.1.1.1.1.m1.1.1" xref="A4.T7.1.1.1.1.1.m1.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="A4.T7.1.1.1.1.1.m1.1b"><cn type="integer" id="A4.T7.1.1.1.1.1.m1.1.1.cmml" xref="A4.T7.1.1.1.1.1.m1.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="A4.T7.1.1.1.1.1.m1.1c">5</annotation></semantics></math> copies of a sci-fi novel, each priced at $20, 3 copies of a history book priced at $30 each, and 2 copies of a philosophy book for $45 each.
What was the total cost of his purchases?</span>
</span>
</td>
</tr>
<tr id="A4.T7.1.2" class="ltx_tr">
<td id="A4.T7.1.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A4.T7.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T7.1.2.1.1.1" class="ltx_p" style="width:620.0pt;"><span id="A4.T7.1.2.1.1.1.1" class="ltx_text ltx_font_bold">Response from Original Model</span>: To find the total cost of Benjamin’s purchases, I need to add up the prices of all the books he bought. I can use the information given in the problem to set up an equation:</span>
<span id="A4.T7.1.2.1.1.2" class="ltx_p"><span id="A4.T7.1.2.1.1.2.1" class="ltx_text" style="color:#000000;background-color:#FBE5D6;">Total cost = Price of sci-fi novel * Number of copies + Price of history book * Number of copies + Price of philosophy book * Number of copies</span></span>
<span id="A4.T7.1.2.1.1.3" class="ltx_p"><span id="A4.T7.1.2.1.1.3.1" class="ltx_text" style="color:#000000;background-color:#FBE5D6;">Plugging in the values from the problem, I get:</span></span>
<span id="A4.T7.1.2.1.1.4" class="ltx_p"><span id="A4.T7.1.2.1.1.4.1" class="ltx_text" style="color:#000000;background-color:#FBE5D6;">Total cost = $20 * 5 + $30 * 3 + $45 * 2</span></span>
<span id="A4.T7.1.2.1.1.5" class="ltx_p"><span id="A4.T7.1.2.1.1.5.1" class="ltx_text" style="color:#000000;background-color:#FBE5D6;">Simplifying, I get:</span></span>
<span id="A4.T7.1.2.1.1.6" class="ltx_p"><span id="A4.T7.1.2.1.1.6.1" class="ltx_text" style="color:#000000;background-color:#FBE5D6;">Total cost = $100 + $90 + $90</span></span>
<span id="A4.T7.1.2.1.1.7" class="ltx_p"><span id="A4.T7.1.2.1.1.7.1" class="ltx_text" style="color:#000000;background-color:#FBE5D6;">Adding, I get:</span></span>
<span id="A4.T7.1.2.1.1.8" class="ltx_p"><span id="A4.T7.1.2.1.1.8.1" class="ltx_text" style="color:#000000;background-color:#FBE5D6;">Total cost = $280</span></span>
<span id="A4.T7.1.2.1.1.9" class="ltx_p">So, the total cost of Benjamin’s purchases was $280.</span>
</span>
</td>
</tr>
<tr id="A4.T7.1.3" class="ltx_tr">
<td id="A4.T7.1.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A4.T7.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T7.1.3.1.1.1" class="ltx_p" style="width:620.0pt;"><span id="A4.T7.1.3.1.1.1.1" class="ltx_text ltx_font_bold">Response from <span id="A4.T7.1.3.1.1.1.1.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> Model</span>:
To find the total cost of Benjamin’s purchases, we need to add up the prices of the books he bought.</span>
</span>
</td>
</tr>
<tr id="A4.T7.1.4" class="ltx_tr">
<td id="A4.T7.1.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A4.T7.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T7.1.4.1.1.1" class="ltx_p" style="width:620.0pt;"><span id="A4.T7.1.4.1.1.1.1" class="ltx_text" style="color:#000000;background-color:#E2F0D9;">Step-by-step solution:</span></span>
<span id="A4.T7.1.4.1.1.2" class="ltx_p"><span id="A4.T7.1.4.1.1.2.1" class="ltx_text" style="color:#000000;background-color:#E2F0D9;">1. Calculate the cost of the sci-fi novels: 5 copies at $20 each = $100</span></span>
<span id="A4.T7.1.4.1.1.3" class="ltx_p"><span id="A4.T7.1.4.1.1.3.1" class="ltx_text" style="color:#000000;background-color:#E2F0D9;">2. Calculate the cost of the history books: 3 copies at $30 each = $90</span></span>
<span id="A4.T7.1.4.1.1.4" class="ltx_p"><span id="A4.T7.1.4.1.1.4.1" class="ltx_text" style="color:#000000;background-color:#E2F0D9;">3. Calculate the cost of the philosophy books: 2 copies at $45 each = $90</span></span>
</span>
</td>
</tr>
<tr id="A4.T7.1.5" class="ltx_tr">
<td id="A4.T7.1.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A4.T7.1.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T7.1.5.1.1.1" class="ltx_p" style="width:620.0pt;"><span id="A4.T7.1.5.1.1.1.1" class="ltx_text" style="color:#000000;background-color:#E2F0D9;">Detailed explanation:</span></span>
<span id="A4.T7.1.5.1.1.2" class="ltx_p"><span id="A4.T7.1.5.1.1.2.1" class="ltx_text" style="color:#000000;background-color:#E2F0D9;">The total cost of Benjamin’s purchases is the sum of the costs of the sci-fi novels, the history books, and the philosophy books: $100 + $90 + $90 = $280.</span></span>
</span>
</td>
</tr>
<tr id="A4.T7.1.6" class="ltx_tr">
<td id="A4.T7.1.6.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A4.T7.1.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T7.1.6.1.1.1" class="ltx_p" style="width:620.0pt;"><span id="A4.T7.1.6.1.1.1.1" class="ltx_text" style="color:#000000;background-color:#E2F0D9;">Correct result and conclusion:</span></span>
<span id="A4.T7.1.6.1.1.2" class="ltx_p">The total cost of Benjamin’s purchases was $280.</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>An example of the response from a model trained on the original <span id="A4.T7.9.1" class="ltx_text ltx_font_typewriter">Open-Platypus</span> dataset (denoted as Original Model) and the response from a model trained on the <span id="A4.T7.10.2" class="ltx_text ltx_font_smallcaps">ReAlign</span> <span id="A4.T7.11.3" class="ltx_text ltx_font_typewriter">Open-Platypus</span> dataset (denoted as <span id="A4.T7.12.4" class="ltx_text ltx_font_smallcaps">ReAlign</span> Model). We highlight the content in the better format given by the <span id="A4.T7.13.5" class="ltx_text ltx_font_smallcaps">ReAlign</span> Model in 
<span id="A4.T7.14.6" class="ltx_text" style="color:#000000;background-color:#E2F0D9;">green</span> compared with the content given by the Original Model in 
<span id="A4.T7.15.7" class="ltx_text" style="color:#000000;background-color:#FBE5D6;">red</span>.</figcaption>
</figure>
<figure id="A4.T8" class="ltx_table">
<table id="A4.T8.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A4.T8.1.1" class="ltx_tr">
<td id="A4.T8.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="A4.T8.1.1.1.1" class="ltx_text ltx_font_bold">Task name</span></td>
<td id="A4.T8.1.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A4.T8.1.1.2.1" class="ltx_text ltx_font_bold">Description</span></td>
<td id="A4.T8.1.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="A4.T8.1.1.3.1" class="ltx_text ltx_font_bold">Retrieval</span></td>
<td id="A4.T8.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="A4.T8.1.1.4.1" class="ltx_text ltx_font_bold">Rewriting</span></td>
</tr>
<tr id="A4.T8.1.2" class="ltx_tr">
<td id="A4.T8.1.2.1" class="ltx_td ltx_align_center ltx_border_t" colspan="4"><span id="A4.T8.1.2.1.1" class="ltx_text ltx_font_italic">Generation</span></td>
</tr>
<tr id="A4.T8.1.3" class="ltx_tr">
<td id="A4.T8.1.3.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">question generation</td>
<td id="A4.T8.1.3.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Write some questions based on the given description.</td>
<td id="A4.T8.1.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">✗</td>
<td id="A4.T8.1.3.4" class="ltx_td ltx_align_center ltx_border_t">✓</td>
</tr>
<tr id="A4.T8.1.4" class="ltx_tr">
<td id="A4.T8.1.4.1" class="ltx_td ltx_align_center ltx_border_r">story generation</td>
<td id="A4.T8.1.4.2" class="ltx_td ltx_align_left ltx_border_r">Write a story based on the given description.</td>
<td id="A4.T8.1.4.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.4.4" class="ltx_td ltx_align_center">✗</td>
</tr>
<tr id="A4.T8.1.5" class="ltx_tr">
<td id="A4.T8.1.5.1" class="ltx_td ltx_align_center ltx_border_r">poem generation</td>
<td id="A4.T8.1.5.2" class="ltx_td ltx_align_left ltx_border_r">Write a poem based on the given description.</td>
<td id="A4.T8.1.5.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.5.4" class="ltx_td ltx_align_center">✗</td>
</tr>
<tr id="A4.T8.1.6" class="ltx_tr">
<td id="A4.T8.1.6.1" class="ltx_td ltx_align_center ltx_border_r">email generation</td>
<td id="A4.T8.1.6.2" class="ltx_td ltx_align_left ltx_border_r">Write an email based on the given description.</td>
<td id="A4.T8.1.6.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.6.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.7" class="ltx_tr">
<td id="A4.T8.1.7.1" class="ltx_td ltx_align_center ltx_border_r">data generation</td>
<td id="A4.T8.1.7.2" class="ltx_td ltx_align_left ltx_border_r">Generate data based on the given description.</td>
<td id="A4.T8.1.7.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.7.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.8" class="ltx_tr">
<td id="A4.T8.1.8.1" class="ltx_td ltx_align_center ltx_border_r">text-to-text translation</td>
<td id="A4.T8.1.8.2" class="ltx_td ltx_align_left ltx_border_r">Translate the given text into another language.</td>
<td id="A4.T8.1.8.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.8.4" class="ltx_td ltx_align_center">✗</td>
</tr>
<tr id="A4.T8.1.9" class="ltx_tr">
<td id="A4.T8.1.9.1" class="ltx_td ltx_align_center ltx_border_t" colspan="4"><span id="A4.T8.1.9.1.1" class="ltx_text ltx_font_italic">Brainstorming</span></td>
</tr>
<tr id="A4.T8.1.10" class="ltx_tr">
<td id="A4.T8.1.10.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">advice giving</td>
<td id="A4.T8.1.10.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Respond well to users when they seek advice.</td>
<td id="A4.T8.1.10.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">✗</td>
<td id="A4.T8.1.10.4" class="ltx_td ltx_align_center ltx_border_t">✗</td>
</tr>
<tr id="A4.T8.1.11" class="ltx_tr">
<td id="A4.T8.1.11.1" class="ltx_td ltx_align_center ltx_border_r">recommendations</td>
<td id="A4.T8.1.11.2" class="ltx_td ltx_align_left ltx_border_r">Give recommendations to users.</td>
<td id="A4.T8.1.11.3" class="ltx_td ltx_align_center ltx_border_r">✓</td>
<td id="A4.T8.1.11.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.12" class="ltx_tr">
<td id="A4.T8.1.12.1" class="ltx_td ltx_align_center ltx_border_r">how-to generation</td>
<td id="A4.T8.1.12.2" class="ltx_td ltx_align_left ltx_border_r">Give relevant and complete answer when users ask ‘how to do‘ something.</td>
<td id="A4.T8.1.12.3" class="ltx_td ltx_align_center ltx_border_r">✓</td>
<td id="A4.T8.1.12.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.13" class="ltx_tr">
<td id="A4.T8.1.13.1" class="ltx_td ltx_align_center ltx_border_r">planning</td>
<td id="A4.T8.1.13.2" class="ltx_td ltx_align_left ltx_border_r">Write a plan for an event or activity.</td>
<td id="A4.T8.1.13.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.13.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.14" class="ltx_tr">
<td id="A4.T8.1.14.1" class="ltx_td ltx_align_center ltx_border_t" colspan="4"><span id="A4.T8.1.14.1.1" class="ltx_text ltx_font_italic">Code</span></td>
</tr>
<tr id="A4.T8.1.15" class="ltx_tr">
<td id="A4.T8.1.15.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">code correction</td>
<td id="A4.T8.1.15.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Correct the potential errors in a piece of code.</td>
<td id="A4.T8.1.15.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">✗</td>
<td id="A4.T8.1.15.4" class="ltx_td ltx_align_center ltx_border_t">✓</td>
</tr>
<tr id="A4.T8.1.16" class="ltx_tr">
<td id="A4.T8.1.16.1" class="ltx_td ltx_align_center ltx_border_r">code simplification</td>
<td id="A4.T8.1.16.2" class="ltx_td ltx_align_left ltx_border_r">Rewrite a piece of code to make it more concise and easy to understand.</td>
<td id="A4.T8.1.16.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.16.4" class="ltx_td ltx_align_center">✗</td>
</tr>
<tr id="A4.T8.1.17" class="ltx_tr">
<td id="A4.T8.1.17.1" class="ltx_td ltx_align_center ltx_border_r">explain code</td>
<td id="A4.T8.1.17.2" class="ltx_td ltx_align_left ltx_border_r">Write an explanation for a piece of code.</td>
<td id="A4.T8.1.17.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.17.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.18" class="ltx_tr">
<td id="A4.T8.1.18.1" class="ltx_td ltx_align_center ltx_border_r">text-to-code translation</td>
<td id="A4.T8.1.18.2" class="ltx_td ltx_align_left ltx_border_r">Write a piece of code based on the given description.</td>
<td id="A4.T8.1.18.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.18.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.19" class="ltx_tr">
<td id="A4.T8.1.19.1" class="ltx_td ltx_align_center ltx_border_r">code-to-code translation</td>
<td id="A4.T8.1.19.2" class="ltx_td ltx_align_left ltx_border_r">Convert the given code into another programming language.</td>
<td id="A4.T8.1.19.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.19.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.20" class="ltx_tr">
<td id="A4.T8.1.20.1" class="ltx_td ltx_align_center ltx_border_r">language learning questions</td>
<td id="A4.T8.1.20.2" class="ltx_td ltx_align_left ltx_border_r">Write an answer for the given question about programming language learning.</td>
<td id="A4.T8.1.20.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.20.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.21" class="ltx_tr">
<td id="A4.T8.1.21.1" class="ltx_td ltx_align_center ltx_border_r">code language classification</td>
<td id="A4.T8.1.21.2" class="ltx_td ltx_align_left ltx_border_r">Classify the programming language for the given code.</td>
<td id="A4.T8.1.21.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.21.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.22" class="ltx_tr">
<td id="A4.T8.1.22.1" class="ltx_td ltx_align_center ltx_border_r">code-to-text-translation</td>
<td id="A4.T8.1.22.2" class="ltx_td ltx_align_left ltx_border_r">Write a document for the given code.</td>
<td id="A4.T8.1.22.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.22.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.23" class="ltx_tr">
<td id="A4.T8.1.23.1" class="ltx_td ltx_align_center ltx_border_t" colspan="4"><span id="A4.T8.1.23.1.1" class="ltx_text ltx_font_italic">Rewriting</span></td>
</tr>
<tr id="A4.T8.1.24" class="ltx_tr">
<td id="A4.T8.1.24.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">instructional rewriting</td>
<td id="A4.T8.1.24.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Rewrite a given text with a specific instruction.</td>
<td id="A4.T8.1.24.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">✗</td>
<td id="A4.T8.1.24.4" class="ltx_td ltx_align_center ltx_border_t">✓</td>
</tr>
<tr id="A4.T8.1.25" class="ltx_tr">
<td id="A4.T8.1.25.1" class="ltx_td ltx_align_center ltx_border_r">language polishing</td>
<td id="A4.T8.1.25.2" class="ltx_td ltx_align_left ltx_border_r">Polish a piece of text to make it more fluent, natural, and readable.</td>
<td id="A4.T8.1.25.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.25.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.26" class="ltx_tr">
<td id="A4.T8.1.26.1" class="ltx_td ltx_align_center ltx_border_r">paraphrasing</td>
<td id="A4.T8.1.26.2" class="ltx_td ltx_align_left ltx_border_r">Paraphrase a given text.</td>
<td id="A4.T8.1.26.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.26.4" class="ltx_td ltx_align_center">✗</td>
</tr>
<tr id="A4.T8.1.27" class="ltx_tr">
<td id="A4.T8.1.27.1" class="ltx_td ltx_align_center ltx_border_r">text correction</td>
<td id="A4.T8.1.27.2" class="ltx_td ltx_align_left ltx_border_r">Correct the potential errors in a piece of text.</td>
<td id="A4.T8.1.27.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.27.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.28" class="ltx_tr">
<td id="A4.T8.1.28.1" class="ltx_td ltx_align_center ltx_border_t" colspan="4"><span id="A4.T8.1.28.1.1" class="ltx_text ltx_font_italic">Extraction</span></td>
</tr>
<tr id="A4.T8.1.29" class="ltx_tr">
<td id="A4.T8.1.29.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">information extraction</td>
<td id="A4.T8.1.29.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Extract one or multiple user-specified categories of information from a piece of text attached in the user’s query.</td>
<td id="A4.T8.1.29.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">✗</td>
<td id="A4.T8.1.29.4" class="ltx_td ltx_align_center ltx_border_t">✓</td>
</tr>
<tr id="A4.T8.1.30" class="ltx_tr">
<td id="A4.T8.1.30.1" class="ltx_td ltx_align_center ltx_border_r">keywords extraction</td>
<td id="A4.T8.1.30.2" class="ltx_td ltx_align_left ltx_border_r">Extract the keywords from a piece of text.</td>
<td id="A4.T8.1.30.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.30.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.31" class="ltx_tr">
<td id="A4.T8.1.31.1" class="ltx_td ltx_align_center ltx_border_r">table extraction</td>
<td id="A4.T8.1.31.2" class="ltx_td ltx_align_left ltx_border_r">Generate a table include the key information from a piece of text attached in the user’s query.</td>
<td id="A4.T8.1.31.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.31.4" class="ltx_td ltx_align_center">✗</td>
</tr>
<tr id="A4.T8.1.32" class="ltx_tr">
<td id="A4.T8.1.32.1" class="ltx_td ltx_align_center ltx_border_t" colspan="4"><span id="A4.T8.1.32.1.1" class="ltx_text ltx_font_italic">Summarization</span></td>
</tr>
<tr id="A4.T8.1.33" class="ltx_tr">
<td id="A4.T8.1.33.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">title generation</td>
<td id="A4.T8.1.33.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Generate a title for the given text or based on a description of the work.</td>
<td id="A4.T8.1.33.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">✗</td>
<td id="A4.T8.1.33.4" class="ltx_td ltx_align_center ltx_border_t">✗</td>
</tr>
<tr id="A4.T8.1.34" class="ltx_tr">
<td id="A4.T8.1.34.1" class="ltx_td ltx_align_center ltx_border_r">text summarization</td>
<td id="A4.T8.1.34.2" class="ltx_td ltx_align_left ltx_border_r">Write a summary for a piece of text.</td>
<td id="A4.T8.1.34.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.34.4" class="ltx_td ltx_align_center">✗</td>
</tr>
<tr id="A4.T8.1.35" class="ltx_tr">
<td id="A4.T8.1.35.1" class="ltx_td ltx_align_center ltx_border_r">note summarization</td>
<td id="A4.T8.1.35.2" class="ltx_td ltx_align_left ltx_border_r">Write a note to summarize a piece of text.</td>
<td id="A4.T8.1.35.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.35.4" class="ltx_td ltx_align_center">✗</td>
</tr>
<tr id="A4.T8.1.36" class="ltx_tr">
<td id="A4.T8.1.36.1" class="ltx_td ltx_align_center ltx_border_t" colspan="4"><span id="A4.T8.1.36.1.1" class="ltx_text ltx_font_italic">Conversation</span></td>
</tr>
<tr id="A4.T8.1.37" class="ltx_tr">
<td id="A4.T8.1.37.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">open qa</td>
<td id="A4.T8.1.37.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">The user’s query is an open domain question with no attached passage or article.</td>
<td id="A4.T8.1.37.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">✓</td>
<td id="A4.T8.1.37.4" class="ltx_td ltx_align_center ltx_border_t">✓</td>
</tr>
<tr id="A4.T8.1.38" class="ltx_tr">
<td id="A4.T8.1.38.1" class="ltx_td ltx_align_center ltx_border_r">closed qa</td>
<td id="A4.T8.1.38.2" class="ltx_td ltx_align_left ltx_border_r">Answer the questions that can be directly answered by the attached passage.</td>
<td id="A4.T8.1.38.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.38.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.39" class="ltx_tr">
<td id="A4.T8.1.39.1" class="ltx_td ltx_align_center ltx_border_r">fact verification</td>
<td id="A4.T8.1.39.2" class="ltx_td ltx_align_left ltx_border_r">Verify if the given fact is true or false.</td>
<td id="A4.T8.1.39.3" class="ltx_td ltx_align_center ltx_border_r">✓</td>
<td id="A4.T8.1.39.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.40" class="ltx_tr">
<td id="A4.T8.1.40.1" class="ltx_td ltx_align_center ltx_border_r">value judgment</td>
<td id="A4.T8.1.40.2" class="ltx_td ltx_align_left ltx_border_r">Provide a value judgment on a given topic or statement.</td>
<td id="A4.T8.1.40.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.40.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.41" class="ltx_tr">
<td id="A4.T8.1.41.1" class="ltx_td ltx_align_center ltx_border_r">roleplay</td>
<td id="A4.T8.1.41.2" class="ltx_td ltx_align_left ltx_border_r">Pretend to be a specific person, character, profession or identity, and complete the required task on this basis.</td>
<td id="A4.T8.1.41.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.41.4" class="ltx_td ltx_align_center">✗</td>
</tr>
<tr id="A4.T8.1.42" class="ltx_tr">
<td id="A4.T8.1.42.1" class="ltx_td ltx_align_center ltx_border_r">explain answer</td>
<td id="A4.T8.1.42.2" class="ltx_td ltx_align_left ltx_border_r">Explain something the user wants to know.</td>
<td id="A4.T8.1.42.3" class="ltx_td ltx_align_center ltx_border_r">✓</td>
<td id="A4.T8.1.42.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.43" class="ltx_tr">
<td id="A4.T8.1.43.1" class="ltx_td ltx_align_center ltx_border_t" colspan="4"><span id="A4.T8.1.43.1.1" class="ltx_text ltx_font_italic">Specialized Educational Dialog</span></td>
</tr>
<tr id="A4.T8.1.44" class="ltx_tr">
<td id="A4.T8.1.44.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">natural language tutor</td>
<td id="A4.T8.1.44.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Write an answer for the given question about natural language learning.</td>
<td id="A4.T8.1.44.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">✗</td>
<td id="A4.T8.1.44.4" class="ltx_td ltx_align_center ltx_border_t">✓</td>
</tr>
<tr id="A4.T8.1.45" class="ltx_tr">
<td id="A4.T8.1.45.1" class="ltx_td ltx_align_center ltx_border_r">exam problem tutor</td>
<td id="A4.T8.1.45.2" class="ltx_td ltx_align_left ltx_border_r">Solve an exam question (like fill-in-the-blank, multiple choice, problem solving, etc) with no math involved.</td>
<td id="A4.T8.1.45.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.45.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.46" class="ltx_tr">
<td id="A4.T8.1.46.1" class="ltx_td ltx_align_center ltx_border_r">ai tutor</td>
<td id="A4.T8.1.46.2" class="ltx_td ltx_align_left ltx_border_r">Write an answer for the given question about machine learning, artificial intelligence or language model.</td>
<td id="A4.T8.1.46.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.46.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.47" class="ltx_tr">
<td id="A4.T8.1.47.1" class="ltx_td ltx_align_center ltx_border_r">math puzzles</td>
<td id="A4.T8.1.47.2" class="ltx_td ltx_align_left ltx_border_r">Write an answer with the step-by-step reasoning process for a math question.</td>
<td id="A4.T8.1.47.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.47.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.48" class="ltx_tr">
<td id="A4.T8.1.48.1" class="ltx_td ltx_align_center ltx_border_r">fill in the blank</td>
<td id="A4.T8.1.48.2" class="ltx_td ltx_align_left ltx_border_r">Complete the missing parts with the most appropriate words to make the text coherent and meaningful.</td>
<td id="A4.T8.1.48.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.48.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.49" class="ltx_tr">
<td id="A4.T8.1.49.1" class="ltx_td ltx_align_center ltx_border_t" colspan="4"><span id="A4.T8.1.49.1.1" class="ltx_text ltx_font_italic">Classification</span></td>
</tr>
<tr id="A4.T8.1.50" class="ltx_tr">
<td id="A4.T8.1.50.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">general classification</td>
<td id="A4.T8.1.50.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Classify one or multiple objects given by the user into the specified categories.</td>
<td id="A4.T8.1.50.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">✗</td>
<td id="A4.T8.1.50.4" class="ltx_td ltx_align_center ltx_border_t">✓</td>
</tr>
<tr id="A4.T8.1.51" class="ltx_tr">
<td id="A4.T8.1.51.1" class="ltx_td ltx_align_center ltx_border_r">ordering</td>
<td id="A4.T8.1.51.2" class="ltx_td ltx_align_left ltx_border_r">Sort some things, according to some criteria.</td>
<td id="A4.T8.1.51.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.51.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.52" class="ltx_tr">
<td id="A4.T8.1.52.1" class="ltx_td ltx_align_center ltx_border_r">sentiment analysis</td>
<td id="A4.T8.1.52.2" class="ltx_td ltx_align_left ltx_border_r">Identify and categorize the subjective opinions, attitudes, and feelings of the writer towards a particular subject.</td>
<td id="A4.T8.1.52.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.52.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.53" class="ltx_tr">
<td id="A4.T8.1.53.1" class="ltx_td ltx_align_center ltx_border_r">language classification</td>
<td id="A4.T8.1.53.2" class="ltx_td ltx_align_left ltx_border_r">Classify the language for the given text.</td>
<td id="A4.T8.1.53.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.53.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.54" class="ltx_tr">
<td id="A4.T8.1.54.1" class="ltx_td ltx_align_center ltx_border_r">topic classification</td>
<td id="A4.T8.1.54.2" class="ltx_td ltx_align_left ltx_border_r">Extract the high-level topics or themes from a given text, i.e., what kind of topics are discussed in the text.</td>
<td id="A4.T8.1.54.3" class="ltx_td ltx_align_center ltx_border_r">✗</td>
<td id="A4.T8.1.54.4" class="ltx_td ltx_align_center">✓</td>
</tr>
<tr id="A4.T8.1.55" class="ltx_tr">
<td id="A4.T8.1.55.1" class="ltx_td ltx_align_center ltx_border_t" colspan="4"><span id="A4.T8.1.55.1.1" class="ltx_text ltx_font_italic">Others</span></td>
</tr>
<tr id="A4.T8.1.56" class="ltx_tr">
<td id="A4.T8.1.56.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">rejecting</td>
<td id="A4.T8.1.56.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Reject to respond when the query is beyond capacity or it violates general ethical and legal rules.</td>
<td id="A4.T8.1.56.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">✗</td>
<td id="A4.T8.1.56.4" class="ltx_td ltx_align_center ltx_border_t">✓</td>
</tr>
<tr id="A4.T8.1.57" class="ltx_tr">
<td id="A4.T8.1.57.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">others</td>
<td id="A4.T8.1.57.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">You must choose this if none of the other scenarios match the user’s query well.</td>
<td id="A4.T8.1.57.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">✗</td>
<td id="A4.T8.1.57.4" class="ltx_td ltx_align_center ltx_border_bb">✓</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 8: </span>Detailed description for each task.</figcaption>
</figure>
<figure id="A4.T9" class="ltx_table">
<table id="A4.T9.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A4.T9.1.1" class="ltx_tr">
<td id="A4.T9.1.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">task</td>
<td id="A4.T9.1.1.2" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">train</td>
<td id="A4.T9.1.1.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">test</td>
<td id="A4.T9.1.1.4" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">task</td>
<td id="A4.T9.1.1.5" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">train</td>
<td id="A4.T9.1.1.6" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">test</td>
<td id="A4.T9.1.1.7" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">task</td>
<td id="A4.T9.1.1.8" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">train</td>
<td id="A4.T9.1.1.9" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">test</td>
</tr>
<tr id="A4.T9.1.2" class="ltx_tr">
<td id="A4.T9.1.2.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">question_generation</td>
<td id="A4.T9.1.2.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.2.3" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.2.4" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">code_language_classification</td>
<td id="A4.T9.1.2.5" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.2.6" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.2.7" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">roleplay</td>
<td id="A4.T9.1.2.8" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.2.9" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
</tr>
<tr id="A4.T9.1.3" class="ltx_tr">
<td id="A4.T9.1.3.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">story_generation</td>
<td id="A4.T9.1.3.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.3.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">4</td>
<td id="A4.T9.1.3.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">code_to_text_translation</td>
<td id="A4.T9.1.3.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.3.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
<td id="A4.T9.1.3.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">explain_answer</td>
<td id="A4.T9.1.3.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.3.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">4</td>
</tr>
<tr id="A4.T9.1.4" class="ltx_tr">
<td id="A4.T9.1.4.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">poem_generation</td>
<td id="A4.T9.1.4.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.4.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
<td id="A4.T9.1.4.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">instructional_rewriting</td>
<td id="A4.T9.1.4.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.4.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">4</td>
<td id="A4.T9.1.4.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">natural_language_learning_tutor</td>
<td id="A4.T9.1.4.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.4.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
</tr>
<tr id="A4.T9.1.5" class="ltx_tr">
<td id="A4.T9.1.5.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">email_generation</td>
<td id="A4.T9.1.5.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.5.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
<td id="A4.T9.1.5.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">language_polishing</td>
<td id="A4.T9.1.5.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.5.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.5.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">exam_problem_solving_tutor</td>
<td id="A4.T9.1.5.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">31</td>
<td id="A4.T9.1.5.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
</tr>
<tr id="A4.T9.1.6" class="ltx_tr">
<td id="A4.T9.1.6.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">data_generation</td>
<td id="A4.T9.1.6.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.6.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
<td id="A4.T9.1.6.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">paraphrasing</td>
<td id="A4.T9.1.6.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.6.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.6.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">ml_ai_language_model_tutor</td>
<td id="A4.T9.1.6.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.6.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
</tr>
<tr id="A4.T9.1.7" class="ltx_tr">
<td id="A4.T9.1.7.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">text_to_text_translation</td>
<td id="A4.T9.1.7.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.7.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
<td id="A4.T9.1.7.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">text_correction</td>
<td id="A4.T9.1.7.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.7.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.7.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">math_puzzles</td>
<td id="A4.T9.1.7.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.7.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">6</td>
</tr>
<tr id="A4.T9.1.8" class="ltx_tr">
<td id="A4.T9.1.8.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">advice_giving</td>
<td id="A4.T9.1.8.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.8.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">4</td>
<td id="A4.T9.1.8.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">information_extraction</td>
<td id="A4.T9.1.8.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.8.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
<td id="A4.T9.1.8.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">fill_in_the_blank</td>
<td id="A4.T9.1.8.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.8.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
</tr>
<tr id="A4.T9.1.9" class="ltx_tr">
<td id="A4.T9.1.9.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">recommendations</td>
<td id="A4.T9.1.9.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.9.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.9.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">keywords_extraction</td>
<td id="A4.T9.1.9.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.9.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.9.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">general_classification</td>
<td id="A4.T9.1.9.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.9.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">4</td>
</tr>
<tr id="A4.T9.1.10" class="ltx_tr">
<td id="A4.T9.1.10.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">how_to_generation</td>
<td id="A4.T9.1.10.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.10.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
<td id="A4.T9.1.10.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">table_extraction</td>
<td id="A4.T9.1.10.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.10.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
<td id="A4.T9.1.10.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">ordering</td>
<td id="A4.T9.1.10.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.10.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
</tr>
<tr id="A4.T9.1.11" class="ltx_tr">
<td id="A4.T9.1.11.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">planning</td>
<td id="A4.T9.1.11.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.11.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.11.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">title_generation</td>
<td id="A4.T9.1.11.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.11.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.11.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">sentiment_analysis</td>
<td id="A4.T9.1.11.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.11.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
</tr>
<tr id="A4.T9.1.12" class="ltx_tr">
<td id="A4.T9.1.12.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">code_correction</td>
<td id="A4.T9.1.12.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.12.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">5</td>
<td id="A4.T9.1.12.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">text_summarization</td>
<td id="A4.T9.1.12.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.12.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">5</td>
<td id="A4.T9.1.12.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">language_classification</td>
<td id="A4.T9.1.12.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.12.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
</tr>
<tr id="A4.T9.1.13" class="ltx_tr">
<td id="A4.T9.1.13.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">code_simplification</td>
<td id="A4.T9.1.13.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.13.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.13.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">note_summarization</td>
<td id="A4.T9.1.13.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.13.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.13.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">topic_classification</td>
<td id="A4.T9.1.13.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.13.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
</tr>
<tr id="A4.T9.1.14" class="ltx_tr">
<td id="A4.T9.1.14.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">explain_code</td>
<td id="A4.T9.1.14.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.14.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.14.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">open_qa</td>
<td id="A4.T9.1.14.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.14.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">6</td>
<td id="A4.T9.1.14.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">rejecting</td>
<td id="A4.T9.1.14.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.14.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
</tr>
<tr id="A4.T9.1.15" class="ltx_tr">
<td id="A4.T9.1.15.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">text_to_code_translation</td>
<td id="A4.T9.1.15.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.15.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">4</td>
<td id="A4.T9.1.15.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">closed_qa</td>
<td id="A4.T9.1.15.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.15.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.15.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">others</td>
<td id="A4.T9.1.15.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">43</td>
<td id="A4.T9.1.15.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">8</td>
</tr>
<tr id="A4.T9.1.16" class="ltx_tr">
<td id="A4.T9.1.16.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">code_to_code_translation</td>
<td id="A4.T9.1.16.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.16.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">3</td>
<td id="A4.T9.1.16.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">fact_verification</td>
<td id="A4.T9.1.16.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.16.6" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.16.7" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">overall</td>
<td id="A4.T9.1.16.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">1395</td>
<td id="A4.T9.1.16.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">143</td>
</tr>
<tr id="A4.T9.1.17" class="ltx_tr">
<td id="A4.T9.1.17.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">language_learning_questions</td>
<td id="A4.T9.1.17.2" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">31</td>
<td id="A4.T9.1.17.3" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">5</td>
<td id="A4.T9.1.17.4" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">value_judgement</td>
<td id="A4.T9.1.17.5" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">30</td>
<td id="A4.T9.1.17.6" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">2</td>
<td id="A4.T9.1.17.7" class="ltx_td ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;"></td>
<td id="A4.T9.1.17.8" class="ltx_td ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;"></td>
<td id="A4.T9.1.17.9" class="ltx_td ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;"></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 9: </span>The task distribution in the training and test set for task classifier.</figcaption>
</figure>
<figure id="A4.T10" class="ltx_table">
<table id="A4.T10.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A4.T10.1.1" class="ltx_tr">
<td id="A4.T10.1.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A4.T10.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T10.1.1.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A4.T10.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Classification Prompt</span></span>
</span>
</td>
</tr>
<tr id="A4.T10.1.2" class="ltx_tr">
<td id="A4.T10.1.2.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A4.T10.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T10.1.2.1.1.1" class="ltx_p" style="width:433.6pt;">You will receive a user’s query. Additionally, you are given some pre-defined tasks below:</span>
</span>
</td>
</tr>
<tr id="A4.T10.1.3" class="ltx_tr">
<td id="A4.T10.1.3.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A4.T10.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T10.1.3.1.1.1" class="ltx_p" style="width:433.6pt;">[Existing tasks start]</span>
<span id="A4.T10.1.3.1.1.2" class="ltx_p">question_generation</span>
<span id="A4.T10.1.3.1.1.3" class="ltx_p">story_generation</span>
<span id="A4.T10.1.3.1.1.4" class="ltx_p">poem_generation</span>
<span id="A4.T10.1.3.1.1.5" class="ltx_p">email_generation</span>
<span id="A4.T10.1.3.1.1.6" class="ltx_p">data_generation</span>
<span id="A4.T10.1.3.1.1.7" class="ltx_p">advice_giving</span>
<span id="A4.T10.1.3.1.1.8" class="ltx_p">recommendations</span>
<span id="A4.T10.1.3.1.1.9" class="ltx_p">how_to_generation</span>
<span id="A4.T10.1.3.1.1.10" class="ltx_p">planning</span>
<span id="A4.T10.1.3.1.1.11" class="ltx_p">instructional_rewriting</span>
<span id="A4.T10.1.3.1.1.12" class="ltx_p">language_polishing</span>
<span id="A4.T10.1.3.1.1.13" class="ltx_p">paraphrasing</span>
<span id="A4.T10.1.3.1.1.14" class="ltx_p">text_correction</span>
<span id="A4.T10.1.3.1.1.15" class="ltx_p">code_correction</span>
<span id="A4.T10.1.3.1.1.16" class="ltx_p">code_simplification</span>
<span id="A4.T10.1.3.1.1.17" class="ltx_p">information_extraction</span>
<span id="A4.T10.1.3.1.1.18" class="ltx_p">keywords_extraction</span>
<span id="A4.T10.1.3.1.1.19" class="ltx_p">table_extraction</span>
<span id="A4.T10.1.3.1.1.20" class="ltx_p">title_generation</span>
<span id="A4.T10.1.3.1.1.21" class="ltx_p">text_summarization</span>
<span id="A4.T10.1.3.1.1.22" class="ltx_p">note_summarization</span>
<span id="A4.T10.1.3.1.1.23" class="ltx_p">explain_code</span>
<span id="A4.T10.1.3.1.1.24" class="ltx_p">explain_answer</span>
<span id="A4.T10.1.3.1.1.25" class="ltx_p">text_to_text_translation</span>
<span id="A4.T10.1.3.1.1.26" class="ltx_p">text_to_code_translation</span>
<span id="A4.T10.1.3.1.1.27" class="ltx_p">code_to_code_translation</span>
<span id="A4.T10.1.3.1.1.28" class="ltx_p">code_to_text_translation</span>
<span id="A4.T10.1.3.1.1.29" class="ltx_p">open_qa</span>
<span id="A4.T10.1.3.1.1.30" class="ltx_p">closed_qa</span>
<span id="A4.T10.1.3.1.1.31" class="ltx_p">fill_in_the_blank</span>
<span id="A4.T10.1.3.1.1.32" class="ltx_p">fact_verification</span>
<span id="A4.T10.1.3.1.1.33" class="ltx_p">math_puzzles</span>
<span id="A4.T10.1.3.1.1.34" class="ltx_p">language_learning_questions</span>
<span id="A4.T10.1.3.1.1.35" class="ltx_p">natural_language_learning_tutor</span>
<span id="A4.T10.1.3.1.1.36" class="ltx_p">exam_problem_solving_tutor</span>
<span id="A4.T10.1.3.1.1.37" class="ltx_p">ml_ai_language_model_tutor</span>
<span id="A4.T10.1.3.1.1.38" class="ltx_p">general_classification</span>
<span id="A4.T10.1.3.1.1.39" class="ltx_p">ordering</span>
<span id="A4.T10.1.3.1.1.40" class="ltx_p">sentiment_analysis</span>
<span id="A4.T10.1.3.1.1.41" class="ltx_p">code_language_classification</span>
<span id="A4.T10.1.3.1.1.42" class="ltx_p">language_classification</span>
<span id="A4.T10.1.3.1.1.43" class="ltx_p">topic_classification</span>
<span id="A4.T10.1.3.1.1.44" class="ltx_p">value_judgement</span>
<span id="A4.T10.1.3.1.1.45" class="ltx_p">rejecting</span>
<span id="A4.T10.1.3.1.1.46" class="ltx_p">roleplay</span>
<span id="A4.T10.1.3.1.1.47" class="ltx_p">default</span>
<span id="A4.T10.1.3.1.1.48" class="ltx_p">[Existing tasks end]</span>
</span>
</td>
</tr>
<tr id="A4.T10.1.4" class="ltx_tr">
<td id="A4.T10.1.4.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A4.T10.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T10.1.4.1.1.1" class="ltx_p" style="width:433.6pt;">Your objective is to choose the most appropriate task that can reflect the high-level intention of this query. You should first clearly give out your choice. Your choice should exactly match one of the task names provided above, without any modification. Do not include the task description in your choice.</span>
</span>
</td>
</tr>
<tr id="A4.T10.1.5" class="ltx_tr">
<td id="A4.T10.1.5.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A4.T10.1.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T10.1.5.1.1.1" class="ltx_p" style="width:433.6pt;">Your output should be just the task name.</span>
</span>
</td>
</tr>
<tr id="A4.T10.1.6" class="ltx_tr">
<td id="A4.T10.1.6.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A4.T10.1.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T10.1.6.1.1.1" class="ltx_p" style="width:433.6pt;"></span>
<span id="A4.T10.1.6.1.1.2" class="ltx_p">User’s query is below:</span>
<span id="A4.T10.1.6.1.1.3" class="ltx_p">[User’s query start]</span>
<span id="A4.T10.1.6.1.1.4" class="ltx_p">{input}</span>
<span id="A4.T10.1.6.1.1.5" class="ltx_p">[User’s query end]</span>
</span>
</td>
</tr>
<tr id="A4.T10.1.7" class="ltx_tr">
<td id="A4.T10.1.7.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A4.T10.1.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T10.1.7.1.1.1" class="ltx_p" style="width:433.6pt;">Task name:</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 10: </span>The classification prompt for the task classifier in the training and inference phase.</figcaption>
</figure>
</section>
<section id="A5" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Prompts</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.1" class="ltx_p">The reformatting prompts for non-knowledge-intensive tasks and knowledge-intensive tasks are shown in Tab. <a href="#A5.T11" title="Table 11 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> and Tab. <a href="#A5.T12" title="Table 12 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>. The complete version of Fig. <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> and Tab. <a href="#S2.T2" title="Table 2 ‣ 2.3 Instruction Tuning ‣ 2 Related Work ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> are shown in Tab. <a href="#A5.T13" title="Table 13 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a> and Tab. <a href="#A5.T14" title="Table 14 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>, respectively. The example of a knowledge-intensive task is shown in Tab. <a href="#A5.T15" title="Table 15 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>. The evaluation prompts for readability and factuality are shown in Tab. <a href="#A5.T16" title="Table 16 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a> and Tab. <a href="#A5.T17" title="Table 17 ‣ Appendix E Prompts ‣ Reformatted Alignment" class="ltx_ref"><span class="ltx_text ltx_ref_tag">17</span></a>, respectively.</p>
</div>
<figure id="A5.T11" class="ltx_table">
<table id="A5.T11.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A5.T11.1.1" class="ltx_tr">
<td id="A5.T11.1.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T11.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.1.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T11.1.1.1.1.1.1" class="ltx_text ltx_font_bold">System Prompt</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.2" class="ltx_tr">
<td id="A5.T11.1.2.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T11.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.2.1.1.1" class="ltx_p" style="width:433.6pt;">Please act as a rewriter to modify the format of the AI assistant’s response to the user’s question presented below.</span>
</span>
</td>
</tr>
<tr id="A5.T11.1.3" class="ltx_tr">
<td id="A5.T11.1.3.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T11.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.3.1.1.1" class="ltx_p" style="width:433.6pt;">Please follow the instructions below:</span>
</span>
</td>
</tr>
<tr id="A5.T11.1.4" class="ltx_tr">
<td id="A5.T11.1.4.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T11.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.4.1.1.1" class="ltx_p" style="width:433.6pt;">1. Please first determine whether the given format meets the requirements of the user’s question, if it does not, then copy the AI assistant’s response, if it does, then modify the response’s format following the provided format.</span>
<span id="A5.T11.1.4.1.1.2" class="ltx_p">2. Your task is limited to altering the format while keeping the original meaning and information intact.</span>
<span id="A5.T11.1.4.1.1.3" class="ltx_p">3. Please make sure that the revised response can answer the user’s question correctly.</span>
<span id="A5.T11.1.4.1.1.4" class="ltx_p">4. Please make sure that the revised response is fluent and has no additional subheadings.</span>
</span>
</td>
</tr>
<tr id="A5.T11.1.5" class="ltx_tr">
<td id="A5.T11.1.5.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T11.1.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.5.1.1.1" class="ltx_p" style="width:433.6pt;">Please first write ”Reasoning: ¡reason¿” to provide a brief reasoning you used to modify, and then write ”Revised response: ¡response¿” to output your final revised response without any additional information, ensuring its fluency.</span>
</span>
</td>
</tr>
<tr id="A5.T11.1.6" class="ltx_tr">
<td id="A5.T11.1.6.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T11.1.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.6.1.1.1" class="ltx_p" style="width:433.6pt;">Do not output any additional subheadings.</span>
</span>
</td>
</tr>
<tr id="A5.T11.1.7" class="ltx_tr">
<td id="A5.T11.1.7.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T11.1.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.7.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T11.1.7.1.1.1.1" class="ltx_text ltx_font_bold">User Prompt</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.8" class="ltx_tr">
<td id="A5.T11.1.8.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T11.1.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.8.1.1.1" class="ltx_p" style="width:433.6pt;">Below is a user’s question, the AI assistant’s response, and the provided format.</span>
</span>
</td>
</tr>
<tr id="A5.T11.1.9" class="ltx_tr">
<td id="A5.T11.1.9.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T11.1.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.9.1.1.1" class="ltx_p" style="width:433.6pt;">[Question start]</span>
<span id="A5.T11.1.9.1.1.2" class="ltx_p">{question}</span>
<span id="A5.T11.1.9.1.1.3" class="ltx_p">[Question end]</span>
</span>
</td>
</tr>
<tr id="A5.T11.1.10" class="ltx_tr">
<td id="A5.T11.1.10.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T11.1.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.10.1.1.1" class="ltx_p" style="width:433.6pt;">[Response start]</span>
<span id="A5.T11.1.10.1.1.2" class="ltx_p">{response}</span>
<span id="A5.T11.1.10.1.1.3" class="ltx_p">[Response end]</span>
</span>
</td>
</tr>
<tr id="A5.T11.1.11" class="ltx_tr">
<td id="A5.T11.1.11.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T11.1.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.11.1.1.1" class="ltx_p" style="width:433.6pt;">[Format start]</span>
<span id="A5.T11.1.11.1.1.2" class="ltx_p">{format}</span>
<span id="A5.T11.1.11.1.1.3" class="ltx_p">[Format end]</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 11: </span>Reformatting prompts for non-knowledge-intensive tasks.</figcaption>
</figure>
<figure id="A5.T12" class="ltx_table">
<table id="A5.T12.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A5.T12.1.1" class="ltx_tr">
<td id="A5.T12.1.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T12.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.1.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T12.1.1.1.1.1.1" class="ltx_text ltx_font_bold">System Prompt</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.2" class="ltx_tr">
<td id="A5.T12.1.2.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T12.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.2.1.1.1" class="ltx_p" style="width:433.6pt;">Please act as a rewriter to modify the format of the AI assistant’s response to the user’s question presented below.</span>
</span>
</td>
</tr>
<tr id="A5.T12.1.3" class="ltx_tr">
<td id="A5.T12.1.3.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T12.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.3.1.1.1" class="ltx_p" style="width:433.6pt;">Please follow the instructions below:</span>
</span>
</td>
</tr>
<tr id="A5.T12.1.4" class="ltx_tr">
<td id="A5.T12.1.4.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T12.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.4.1.1.1" class="ltx_p" style="width:433.6pt;">1. Please first determine whether the given format meets the requirements of the user’s question, if it does not, then copy the AI assistant’s response, if it does, then modify the response’s format following the provided format.</span>
<span id="A5.T12.1.4.1.1.2" class="ltx_p">2. Your task is limited to altering the format while keeping the original meaning and information intact.</span>
<span id="A5.T12.1.4.1.1.3" class="ltx_p">3. Please make sure that the revised response can answer the user’s question correctly.</span>
<span id="A5.T12.1.4.1.1.4" class="ltx_p">4. Please make sure that the revised response is fluent and has no additional subheadings.</span>
<span id="A5.T12.1.4.1.1.5" class="ltx_p">5. Evidence is the useful information. You should decide for yourself which parts of the evidence to help rewriting the response.</span>
</span>
</td>
</tr>
<tr id="A5.T12.1.5" class="ltx_tr">
<td id="A5.T12.1.5.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T12.1.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.5.1.1.1" class="ltx_p" style="width:433.6pt;">Please first write ”Reasoning: ¡reason¿” to provide a brief reasoning you used to modify, and then write ”Revised response: ¡response¿” to output your final revised response without any additional information, ensuring its fluency.</span>
</span>
</td>
</tr>
<tr id="A5.T12.1.6" class="ltx_tr">
<td id="A5.T12.1.6.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T12.1.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.6.1.1.1" class="ltx_p" style="width:433.6pt;">Do not output any additional subheadings.</span>
</span>
</td>
</tr>
<tr id="A5.T12.1.7" class="ltx_tr">
<td id="A5.T12.1.7.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T12.1.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.7.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T12.1.7.1.1.1.1" class="ltx_text ltx_font_bold">User Prompt</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.8" class="ltx_tr">
<td id="A5.T12.1.8.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T12.1.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.8.1.1.1" class="ltx_p" style="width:433.6pt;">Below is a user’s question, the AI assistant’s response, the provided format, and the evidences.</span>
</span>
</td>
</tr>
<tr id="A5.T12.1.9" class="ltx_tr">
<td id="A5.T12.1.9.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T12.1.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.9.1.1.1" class="ltx_p" style="width:433.6pt;">[Question start]</span>
<span id="A5.T12.1.9.1.1.2" class="ltx_p">{question}</span>
<span id="A5.T12.1.9.1.1.3" class="ltx_p">[Question end]</span>
</span>
</td>
</tr>
<tr id="A5.T12.1.10" class="ltx_tr">
<td id="A5.T12.1.10.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T12.1.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.10.1.1.1" class="ltx_p" style="width:433.6pt;">[Response start]</span>
<span id="A5.T12.1.10.1.1.2" class="ltx_p">{response}</span>
<span id="A5.T12.1.10.1.1.3" class="ltx_p">[Response end]</span>
</span>
</td>
</tr>
<tr id="A5.T12.1.11" class="ltx_tr">
<td id="A5.T12.1.11.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T12.1.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.11.1.1.1" class="ltx_p" style="width:433.6pt;">[Format start]</span>
<span id="A5.T12.1.11.1.1.2" class="ltx_p">{format}</span>
<span id="A5.T12.1.11.1.1.3" class="ltx_p">[Format end]</span>
</span>
</td>
</tr>
<tr id="A5.T12.1.12" class="ltx_tr">
<td id="A5.T12.1.12.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T12.1.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.12.1.1.1" class="ltx_p" style="width:433.6pt;">[Evidence start]</span>
<span id="A5.T12.1.12.1.1.2" class="ltx_p">{evidence}</span>
<span id="A5.T12.1.12.1.1.3" class="ltx_p">[Evidence end]</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 12: </span>Reformatting prompts for knowledge-intensive tasks.</figcaption>
</figure>
<figure id="A5.T13" class="ltx_table">
<table id="A5.T13.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A5.T13.6.7" class="ltx_tr">
<td id="A5.T13.6.7.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T13.6.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.7.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T13.6.7.1.1.1.1" class="ltx_text ltx_font_bold">Query</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.6.8" class="ltx_tr">
<td id="A5.T13.6.8.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T13.6.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.8.1.1.1" class="ltx_p" style="width:433.6pt;">How many positive three-digit integers less than 500 have at least two digits that are the same?</span>
</span>
</td>
</tr>
<tr id="A5.T13.6.9" class="ltx_tr">
<td id="A5.T13.6.9.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T13.6.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.9.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T13.6.9.1.1.1.1" class="ltx_text ltx_font_bold">Criteria</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.6.10" class="ltx_tr">
<td id="A5.T13.6.10.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T13.6.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.10.1.1.1" class="ltx_p" style="width:433.6pt;">Task Name: math puzzles</span>
<span id="A5.T13.6.10.1.1.2" class="ltx_p">Format: This is a math question. First, analyze the question and give a brief analysis in the first paragraph. Then, use a list to present the step-by-step solution. Next, give another list to output a detailed explanation. Last, give the correct result and a conclusion.</span>
</span>
</td>
</tr>
<tr id="A5.T13.6.11" class="ltx_tr">
<td id="A5.T13.6.11.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T13.6.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.11.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T13.6.11.1.1.1.1" class="ltx_text ltx_font_bold">Original Response</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.1" class="ltx_tr">
<td id="A5.T13.1.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T13.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.1.1.1.1" class="ltx_p" style="width:433.6pt;">To solve this problem, I will count how many three-digit numbers less than 500 have at least two digits that are the same, and subtract that from the total number of three-digit numbers less than 500. The total number of three-digit numbers less than 500 is <math id="A5.T13.1.1.1.1.1.m1.1" class="ltx_Math" alttext="4\times 10\times 10=400" display="inline"><semantics id="A5.T13.1.1.1.1.1.m1.1a"><mrow id="A5.T13.1.1.1.1.1.m1.1.1" xref="A5.T13.1.1.1.1.1.m1.1.1.cmml"><mrow id="A5.T13.1.1.1.1.1.m1.1.1.2" xref="A5.T13.1.1.1.1.1.m1.1.1.2.cmml"><mn id="A5.T13.1.1.1.1.1.m1.1.1.2.2" xref="A5.T13.1.1.1.1.1.m1.1.1.2.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="A5.T13.1.1.1.1.1.m1.1.1.2.1" xref="A5.T13.1.1.1.1.1.m1.1.1.2.1.cmml">×</mo><mn id="A5.T13.1.1.1.1.1.m1.1.1.2.3" xref="A5.T13.1.1.1.1.1.m1.1.1.2.3.cmml">10</mn><mo lspace="0.222em" rspace="0.222em" id="A5.T13.1.1.1.1.1.m1.1.1.2.1a" xref="A5.T13.1.1.1.1.1.m1.1.1.2.1.cmml">×</mo><mn id="A5.T13.1.1.1.1.1.m1.1.1.2.4" xref="A5.T13.1.1.1.1.1.m1.1.1.2.4.cmml">10</mn></mrow><mo id="A5.T13.1.1.1.1.1.m1.1.1.1" xref="A5.T13.1.1.1.1.1.m1.1.1.1.cmml">=</mo><mn id="A5.T13.1.1.1.1.1.m1.1.1.3" xref="A5.T13.1.1.1.1.1.m1.1.1.3.cmml">400</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.T13.1.1.1.1.1.m1.1b"><apply id="A5.T13.1.1.1.1.1.m1.1.1.cmml" xref="A5.T13.1.1.1.1.1.m1.1.1"><eq id="A5.T13.1.1.1.1.1.m1.1.1.1.cmml" xref="A5.T13.1.1.1.1.1.m1.1.1.1"></eq><apply id="A5.T13.1.1.1.1.1.m1.1.1.2.cmml" xref="A5.T13.1.1.1.1.1.m1.1.1.2"><times id="A5.T13.1.1.1.1.1.m1.1.1.2.1.cmml" xref="A5.T13.1.1.1.1.1.m1.1.1.2.1"></times><cn type="integer" id="A5.T13.1.1.1.1.1.m1.1.1.2.2.cmml" xref="A5.T13.1.1.1.1.1.m1.1.1.2.2">4</cn><cn type="integer" id="A5.T13.1.1.1.1.1.m1.1.1.2.3.cmml" xref="A5.T13.1.1.1.1.1.m1.1.1.2.3">10</cn><cn type="integer" id="A5.T13.1.1.1.1.1.m1.1.1.2.4.cmml" xref="A5.T13.1.1.1.1.1.m1.1.1.2.4">10</cn></apply><cn type="integer" id="A5.T13.1.1.1.1.1.m1.1.1.3.cmml" xref="A5.T13.1.1.1.1.1.m1.1.1.3">400</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T13.1.1.1.1.1.m1.1c">4\times 10\times 10=400</annotation></semantics></math>, since the first digit can be 1, 2, 3, or 4, and the second and third digits can be anything from 0 to 9. Now I will count how many of these 400 numbers have no digits that are the same. To have no digits that are the same, the first digit can be any of the 4 choices, but the second digit has to be different from the first, so it has 9 choices. The third digit has to be different from both the first and the second, so it has 8 choices.</span>
</span>
</td>
</tr>
<tr id="A5.T13.3.3" class="ltx_tr">
<td id="A5.T13.3.3.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T13.3.3.2.2" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.3.3.2.2.2" class="ltx_p" style="width:433.6pt;">[* on another note] I am assuming that the second digit can be 0, since the problem does not say otherwise. Therefore, the number of three-digit numbers less than 500 that have no digits that are the same is <math id="A5.T13.2.2.1.1.1.m1.1" class="ltx_Math" alttext="4\times 9\times 8=288" display="inline"><semantics id="A5.T13.2.2.1.1.1.m1.1a"><mrow id="A5.T13.2.2.1.1.1.m1.1.1" xref="A5.T13.2.2.1.1.1.m1.1.1.cmml"><mrow id="A5.T13.2.2.1.1.1.m1.1.1.2" xref="A5.T13.2.2.1.1.1.m1.1.1.2.cmml"><mn id="A5.T13.2.2.1.1.1.m1.1.1.2.2" xref="A5.T13.2.2.1.1.1.m1.1.1.2.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="A5.T13.2.2.1.1.1.m1.1.1.2.1" xref="A5.T13.2.2.1.1.1.m1.1.1.2.1.cmml">×</mo><mn id="A5.T13.2.2.1.1.1.m1.1.1.2.3" xref="A5.T13.2.2.1.1.1.m1.1.1.2.3.cmml">9</mn><mo lspace="0.222em" rspace="0.222em" id="A5.T13.2.2.1.1.1.m1.1.1.2.1a" xref="A5.T13.2.2.1.1.1.m1.1.1.2.1.cmml">×</mo><mn id="A5.T13.2.2.1.1.1.m1.1.1.2.4" xref="A5.T13.2.2.1.1.1.m1.1.1.2.4.cmml">8</mn></mrow><mo id="A5.T13.2.2.1.1.1.m1.1.1.1" xref="A5.T13.2.2.1.1.1.m1.1.1.1.cmml">=</mo><mn id="A5.T13.2.2.1.1.1.m1.1.1.3" xref="A5.T13.2.2.1.1.1.m1.1.1.3.cmml">288</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.T13.2.2.1.1.1.m1.1b"><apply id="A5.T13.2.2.1.1.1.m1.1.1.cmml" xref="A5.T13.2.2.1.1.1.m1.1.1"><eq id="A5.T13.2.2.1.1.1.m1.1.1.1.cmml" xref="A5.T13.2.2.1.1.1.m1.1.1.1"></eq><apply id="A5.T13.2.2.1.1.1.m1.1.1.2.cmml" xref="A5.T13.2.2.1.1.1.m1.1.1.2"><times id="A5.T13.2.2.1.1.1.m1.1.1.2.1.cmml" xref="A5.T13.2.2.1.1.1.m1.1.1.2.1"></times><cn type="integer" id="A5.T13.2.2.1.1.1.m1.1.1.2.2.cmml" xref="A5.T13.2.2.1.1.1.m1.1.1.2.2">4</cn><cn type="integer" id="A5.T13.2.2.1.1.1.m1.1.1.2.3.cmml" xref="A5.T13.2.2.1.1.1.m1.1.1.2.3">9</cn><cn type="integer" id="A5.T13.2.2.1.1.1.m1.1.1.2.4.cmml" xref="A5.T13.2.2.1.1.1.m1.1.1.2.4">8</cn></apply><cn type="integer" id="A5.T13.2.2.1.1.1.m1.1.1.3.cmml" xref="A5.T13.2.2.1.1.1.m1.1.1.3">288</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T13.2.2.1.1.1.m1.1c">4\times 9\times 8=288</annotation></semantics></math>. This means that the number of three-digit numbers less than 500 that have at least two digits that are the same is <math id="A5.T13.3.3.2.2.2.m2.1" class="ltx_Math" alttext="400-288=112" display="inline"><semantics id="A5.T13.3.3.2.2.2.m2.1a"><mrow id="A5.T13.3.3.2.2.2.m2.1.1" xref="A5.T13.3.3.2.2.2.m2.1.1.cmml"><mrow id="A5.T13.3.3.2.2.2.m2.1.1.2" xref="A5.T13.3.3.2.2.2.m2.1.1.2.cmml"><mn id="A5.T13.3.3.2.2.2.m2.1.1.2.2" xref="A5.T13.3.3.2.2.2.m2.1.1.2.2.cmml">400</mn><mo id="A5.T13.3.3.2.2.2.m2.1.1.2.1" xref="A5.T13.3.3.2.2.2.m2.1.1.2.1.cmml">−</mo><mn id="A5.T13.3.3.2.2.2.m2.1.1.2.3" xref="A5.T13.3.3.2.2.2.m2.1.1.2.3.cmml">288</mn></mrow><mo id="A5.T13.3.3.2.2.2.m2.1.1.1" xref="A5.T13.3.3.2.2.2.m2.1.1.1.cmml">=</mo><mn id="A5.T13.3.3.2.2.2.m2.1.1.3" xref="A5.T13.3.3.2.2.2.m2.1.1.3.cmml">112</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.T13.3.3.2.2.2.m2.1b"><apply id="A5.T13.3.3.2.2.2.m2.1.1.cmml" xref="A5.T13.3.3.2.2.2.m2.1.1"><eq id="A5.T13.3.3.2.2.2.m2.1.1.1.cmml" xref="A5.T13.3.3.2.2.2.m2.1.1.1"></eq><apply id="A5.T13.3.3.2.2.2.m2.1.1.2.cmml" xref="A5.T13.3.3.2.2.2.m2.1.1.2"><minus id="A5.T13.3.3.2.2.2.m2.1.1.2.1.cmml" xref="A5.T13.3.3.2.2.2.m2.1.1.2.1"></minus><cn type="integer" id="A5.T13.3.3.2.2.2.m2.1.1.2.2.cmml" xref="A5.T13.3.3.2.2.2.m2.1.1.2.2">400</cn><cn type="integer" id="A5.T13.3.3.2.2.2.m2.1.1.2.3.cmml" xref="A5.T13.3.3.2.2.2.m2.1.1.2.3">288</cn></apply><cn type="integer" id="A5.T13.3.3.2.2.2.m2.1.1.3.cmml" xref="A5.T13.3.3.2.2.2.m2.1.1.3">112</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T13.3.3.2.2.2.m2.1c">400-288=112</annotation></semantics></math>.</span>
</span>
</td>
</tr>
<tr id="A5.T13.6.12" class="ltx_tr">
<td id="A5.T13.6.12.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T13.6.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.12.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T13.6.12.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">ReAlign<span id="A5.T13.6.12.1.1.1.1.1" class="ltx_text ltx_font_upright"> Response (Ours)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.T13.6.13" class="ltx_tr">
<td id="A5.T13.6.13.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T13.6.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.13.1.1.1" class="ltx_p" style="width:433.6pt;">Analysis:</span>
<span id="A5.T13.6.13.1.1.2" class="ltx_p">To solve the problem, we need to count how many three-digit numbers less than 500 have at least two digits that are the same.</span>
</span>
</td>
</tr>
<tr id="A5.T13.6.6" class="ltx_tr">
<td id="A5.T13.6.6.3" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T13.6.6.3.3" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.6.3.3.4" class="ltx_p" style="width:433.6pt;">Solution:</span>
<span id="A5.T13.4.4.1.1.1" class="ltx_p">1. Calculate the total number of three-digit numbers less than 500: <math id="A5.T13.4.4.1.1.1.m1.1" class="ltx_Math" alttext="4\times 10\times 10=400" display="inline"><semantics id="A5.T13.4.4.1.1.1.m1.1a"><mrow id="A5.T13.4.4.1.1.1.m1.1.1" xref="A5.T13.4.4.1.1.1.m1.1.1.cmml"><mrow id="A5.T13.4.4.1.1.1.m1.1.1.2" xref="A5.T13.4.4.1.1.1.m1.1.1.2.cmml"><mn id="A5.T13.4.4.1.1.1.m1.1.1.2.2" xref="A5.T13.4.4.1.1.1.m1.1.1.2.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="A5.T13.4.4.1.1.1.m1.1.1.2.1" xref="A5.T13.4.4.1.1.1.m1.1.1.2.1.cmml">×</mo><mn id="A5.T13.4.4.1.1.1.m1.1.1.2.3" xref="A5.T13.4.4.1.1.1.m1.1.1.2.3.cmml">10</mn><mo lspace="0.222em" rspace="0.222em" id="A5.T13.4.4.1.1.1.m1.1.1.2.1a" xref="A5.T13.4.4.1.1.1.m1.1.1.2.1.cmml">×</mo><mn id="A5.T13.4.4.1.1.1.m1.1.1.2.4" xref="A5.T13.4.4.1.1.1.m1.1.1.2.4.cmml">10</mn></mrow><mo id="A5.T13.4.4.1.1.1.m1.1.1.1" xref="A5.T13.4.4.1.1.1.m1.1.1.1.cmml">=</mo><mn id="A5.T13.4.4.1.1.1.m1.1.1.3" xref="A5.T13.4.4.1.1.1.m1.1.1.3.cmml">400</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.T13.4.4.1.1.1.m1.1b"><apply id="A5.T13.4.4.1.1.1.m1.1.1.cmml" xref="A5.T13.4.4.1.1.1.m1.1.1"><eq id="A5.T13.4.4.1.1.1.m1.1.1.1.cmml" xref="A5.T13.4.4.1.1.1.m1.1.1.1"></eq><apply id="A5.T13.4.4.1.1.1.m1.1.1.2.cmml" xref="A5.T13.4.4.1.1.1.m1.1.1.2"><times id="A5.T13.4.4.1.1.1.m1.1.1.2.1.cmml" xref="A5.T13.4.4.1.1.1.m1.1.1.2.1"></times><cn type="integer" id="A5.T13.4.4.1.1.1.m1.1.1.2.2.cmml" xref="A5.T13.4.4.1.1.1.m1.1.1.2.2">4</cn><cn type="integer" id="A5.T13.4.4.1.1.1.m1.1.1.2.3.cmml" xref="A5.T13.4.4.1.1.1.m1.1.1.2.3">10</cn><cn type="integer" id="A5.T13.4.4.1.1.1.m1.1.1.2.4.cmml" xref="A5.T13.4.4.1.1.1.m1.1.1.2.4">10</cn></apply><cn type="integer" id="A5.T13.4.4.1.1.1.m1.1.1.3.cmml" xref="A5.T13.4.4.1.1.1.m1.1.1.3">400</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T13.4.4.1.1.1.m1.1c">4\times 10\times 10=400</annotation></semantics></math>.</span>
<span id="A5.T13.5.5.2.2.2" class="ltx_p">2. Count the number of three-digit numbers less than 500 that have no digits that are the same: <math id="A5.T13.5.5.2.2.2.m1.1" class="ltx_Math" alttext="4\times 9\times 8=288" display="inline"><semantics id="A5.T13.5.5.2.2.2.m1.1a"><mrow id="A5.T13.5.5.2.2.2.m1.1.1" xref="A5.T13.5.5.2.2.2.m1.1.1.cmml"><mrow id="A5.T13.5.5.2.2.2.m1.1.1.2" xref="A5.T13.5.5.2.2.2.m1.1.1.2.cmml"><mn id="A5.T13.5.5.2.2.2.m1.1.1.2.2" xref="A5.T13.5.5.2.2.2.m1.1.1.2.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="A5.T13.5.5.2.2.2.m1.1.1.2.1" xref="A5.T13.5.5.2.2.2.m1.1.1.2.1.cmml">×</mo><mn id="A5.T13.5.5.2.2.2.m1.1.1.2.3" xref="A5.T13.5.5.2.2.2.m1.1.1.2.3.cmml">9</mn><mo lspace="0.222em" rspace="0.222em" id="A5.T13.5.5.2.2.2.m1.1.1.2.1a" xref="A5.T13.5.5.2.2.2.m1.1.1.2.1.cmml">×</mo><mn id="A5.T13.5.5.2.2.2.m1.1.1.2.4" xref="A5.T13.5.5.2.2.2.m1.1.1.2.4.cmml">8</mn></mrow><mo id="A5.T13.5.5.2.2.2.m1.1.1.1" xref="A5.T13.5.5.2.2.2.m1.1.1.1.cmml">=</mo><mn id="A5.T13.5.5.2.2.2.m1.1.1.3" xref="A5.T13.5.5.2.2.2.m1.1.1.3.cmml">288</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.T13.5.5.2.2.2.m1.1b"><apply id="A5.T13.5.5.2.2.2.m1.1.1.cmml" xref="A5.T13.5.5.2.2.2.m1.1.1"><eq id="A5.T13.5.5.2.2.2.m1.1.1.1.cmml" xref="A5.T13.5.5.2.2.2.m1.1.1.1"></eq><apply id="A5.T13.5.5.2.2.2.m1.1.1.2.cmml" xref="A5.T13.5.5.2.2.2.m1.1.1.2"><times id="A5.T13.5.5.2.2.2.m1.1.1.2.1.cmml" xref="A5.T13.5.5.2.2.2.m1.1.1.2.1"></times><cn type="integer" id="A5.T13.5.5.2.2.2.m1.1.1.2.2.cmml" xref="A5.T13.5.5.2.2.2.m1.1.1.2.2">4</cn><cn type="integer" id="A5.T13.5.5.2.2.2.m1.1.1.2.3.cmml" xref="A5.T13.5.5.2.2.2.m1.1.1.2.3">9</cn><cn type="integer" id="A5.T13.5.5.2.2.2.m1.1.1.2.4.cmml" xref="A5.T13.5.5.2.2.2.m1.1.1.2.4">8</cn></apply><cn type="integer" id="A5.T13.5.5.2.2.2.m1.1.1.3.cmml" xref="A5.T13.5.5.2.2.2.m1.1.1.3">288</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T13.5.5.2.2.2.m1.1c">4\times 9\times 8=288</annotation></semantics></math>.</span>
<span id="A5.T13.6.6.3.3.3" class="ltx_p">3. Subtract the count of numbers with no same digits from the total to find the number of three-digit numbers less than 500 that have at least two digits that are the same: <math id="A5.T13.6.6.3.3.3.m1.1" class="ltx_Math" alttext="400-288=112" display="inline"><semantics id="A5.T13.6.6.3.3.3.m1.1a"><mrow id="A5.T13.6.6.3.3.3.m1.1.1" xref="A5.T13.6.6.3.3.3.m1.1.1.cmml"><mrow id="A5.T13.6.6.3.3.3.m1.1.1.2" xref="A5.T13.6.6.3.3.3.m1.1.1.2.cmml"><mn id="A5.T13.6.6.3.3.3.m1.1.1.2.2" xref="A5.T13.6.6.3.3.3.m1.1.1.2.2.cmml">400</mn><mo id="A5.T13.6.6.3.3.3.m1.1.1.2.1" xref="A5.T13.6.6.3.3.3.m1.1.1.2.1.cmml">−</mo><mn id="A5.T13.6.6.3.3.3.m1.1.1.2.3" xref="A5.T13.6.6.3.3.3.m1.1.1.2.3.cmml">288</mn></mrow><mo id="A5.T13.6.6.3.3.3.m1.1.1.1" xref="A5.T13.6.6.3.3.3.m1.1.1.1.cmml">=</mo><mn id="A5.T13.6.6.3.3.3.m1.1.1.3" xref="A5.T13.6.6.3.3.3.m1.1.1.3.cmml">112</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.T13.6.6.3.3.3.m1.1b"><apply id="A5.T13.6.6.3.3.3.m1.1.1.cmml" xref="A5.T13.6.6.3.3.3.m1.1.1"><eq id="A5.T13.6.6.3.3.3.m1.1.1.1.cmml" xref="A5.T13.6.6.3.3.3.m1.1.1.1"></eq><apply id="A5.T13.6.6.3.3.3.m1.1.1.2.cmml" xref="A5.T13.6.6.3.3.3.m1.1.1.2"><minus id="A5.T13.6.6.3.3.3.m1.1.1.2.1.cmml" xref="A5.T13.6.6.3.3.3.m1.1.1.2.1"></minus><cn type="integer" id="A5.T13.6.6.3.3.3.m1.1.1.2.2.cmml" xref="A5.T13.6.6.3.3.3.m1.1.1.2.2">400</cn><cn type="integer" id="A5.T13.6.6.3.3.3.m1.1.1.2.3.cmml" xref="A5.T13.6.6.3.3.3.m1.1.1.2.3">288</cn></apply><cn type="integer" id="A5.T13.6.6.3.3.3.m1.1.1.3.cmml" xref="A5.T13.6.6.3.3.3.m1.1.1.3">112</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T13.6.6.3.3.3.m1.1c">400-288=112</annotation></semantics></math>.</span>
</span>
</td>
</tr>
<tr id="A5.T13.6.14" class="ltx_tr">
<td id="A5.T13.6.14.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T13.6.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.14.1.1.1" class="ltx_p" style="width:433.6pt;">Explanation:</span>
<span id="A5.T13.6.14.1.1.2" class="ltx_p">- The first digit can be 1, 2, 3, or 4, and the second and third digits can be anything from 0 to 9.</span>
<span id="A5.T13.6.14.1.1.3" class="ltx_p">- To have no digits that are the same, the first digit can be any of the 4 choices, but the second digit has to be different from the first, so it has 9 choices. The third digit has to be different from both the first and the second, so it has 8 choices.</span>
</span>
</td>
</tr>
<tr id="A5.T13.6.15" class="ltx_tr">
<td id="A5.T13.6.15.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T13.6.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.15.1.1.1" class="ltx_p" style="width:433.6pt;">Result:</span>
<span id="A5.T13.6.15.1.1.2" class="ltx_p">There are 112 three-digit numbers less than 500 that have at least two digits that are the same.</span>
</span>
</td>
</tr>
<tr id="A5.T13.6.16" class="ltx_tr">
<td id="A5.T13.6.16.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T13.6.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.6.16.1.1.1" class="ltx_p" style="width:433.6pt;">Conclusion:</span>
<span id="A5.T13.6.16.1.1.2" class="ltx_p">There are 112 positive three-digit integers less than 500 that have at least two digits that are the same.</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 13: </span>A complete example of a query, including the original response and the <span id="A5.T13.8.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> response.</figcaption>
</figure>
<figure id="A5.T14" class="ltx_table">
<table id="A5.T14.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A5.T14.1.1" class="ltx_tr">
<td id="A5.T14.1.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T14.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T14.1.1.1.1.1" class="ltx_p" style="width:433.6pt;">It is an email-writing task. Here is a general guideline for creating a well-structured and professional email:</span>
</span>
</td>
</tr>
<tr id="A5.T14.1.2" class="ltx_tr">
<td id="A5.T14.1.2.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T14.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T14.1.2.1.1.1" class="ltx_p" style="width:433.6pt;">    1. Subject Line: Write a clear and concise subject line that accurately summarizes the content of your email. This helps the recipient understand the purpose of the email at a glance.</span>
<span id="A5.T14.1.2.1.1.2" class="ltx_p">2. Salutation: Begin your email with a formal salutation such as ”Dear [Recipient’s Name],” or use a more casual salutation if you have an informal relationship with the recipient.</span>
<span id="A5.T14.1.2.1.1.3" class="ltx_p">3. Introduction: Start your email with a brief introduction, stating who you are and the reason for writing the email. Be clear and to the point, and avoid unnecessary details.</span>
<span id="A5.T14.1.2.1.1.4" class="ltx_p">4. Body: This is the main content of your email. Organize your thoughts into paragraphs or bullet points to make them easier to read. Keep your sentences concise and focused. Use proper grammar, punctuation, and spelling to maintain professionalism. If you need to discuss multiple topics, consider using headings or numbered points to separate them.</span>
<span id="A5.T14.1.2.1.1.5" class="ltx_p">5. Politeness and Tone: Maintain a polite and respectful tone throughout your email. Be mindful of the recipient’s perspective and use appropriate language. Avoid using excessive capitalization, exclamation marks, or emoticons, as they can come across as unprofessional.</span>
<span id="A5.T14.1.2.1.1.6" class="ltx_p">6. Closing: Conclude your email with a closing remark, such as ”Thank you,” or ”Best regards,” followed by your name. If you expect a response or need specific action, you can mention it in this section as well.</span>
<span id="A5.T14.1.2.1.1.7" class="ltx_p">7. Signature: Include your full name, job title, and contact information (e.g., phone number, email address) in your email signature. This helps the recipient easily identify and contact you if needed.</span>
<span id="A5.T14.1.2.1.1.8" class="ltx_p">8. Attachments: If you need to include attachments, mention them in the email body and make sure they are relevant to the email’s purpose. Consider compressing large files or using cloud storage services if the attachments are too large to be sent via email.</span>
<span id="A5.T14.1.2.1.1.9" class="ltx_p">9. Proofread: Before sending the email, proofread it for any grammatical or spelling errors. Make sure the email conveys your message clearly and effectively.</span>
</span>
</td>
</tr>
<tr id="A5.T14.1.3" class="ltx_tr">
<td id="A5.T14.1.3.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T14.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T14.1.3.1.1.1" class="ltx_p" style="width:433.6pt;">The best emails are short, direct, professional, and scannable for the recipient. Follow formal business email structure unless you have an established casual rapport with the recipient.</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 14: </span>A complete format for “email generation” task.</figcaption>
</figure>
<figure id="A5.T15" class="ltx_table">
<table id="A5.T15.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A5.T15.1.1" class="ltx_tr">
<td id="A5.T15.1.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T15.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T15.1.1.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T15.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Query</span></span>
</span>
</td>
</tr>
<tr id="A5.T15.1.2" class="ltx_tr">
<td id="A5.T15.1.2.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T15.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T15.1.2.1.1.1" class="ltx_p" style="width:433.6pt;">Is the given statement true or false?</span>
<span id="A5.T15.1.2.1.1.2" class="ltx_p">A robotic vacuum can clean both carpets and hard floors.</span>
</span>
</td>
</tr>
<tr id="A5.T15.1.3" class="ltx_tr">
<td id="A5.T15.1.3.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T15.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T15.1.3.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T15.1.3.1.1.1.1" class="ltx_text ltx_font_bold">Criteria</span></span>
</span>
</td>
</tr>
<tr id="A5.T15.1.4" class="ltx_tr">
<td id="A5.T15.1.4.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T15.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T15.1.4.1.1.1" class="ltx_p" style="width:433.6pt;">Task Name: fact verification</span>
<span id="A5.T15.1.4.1.1.2" class="ltx_p">Format: This is a fact-verification task. First, give the answer. Then, give an explanation.</span>
</span>
</td>
</tr>
<tr id="A5.T15.1.5" class="ltx_tr">
<td id="A5.T15.1.5.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T15.1.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T15.1.5.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T15.1.5.1.1.1.1" class="ltx_text ltx_font_bold">Retrieved Evidence</span></span>
</span>
</td>
</tr>
<tr id="A5.T15.1.6" class="ltx_tr">
<td id="A5.T15.1.6.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T15.1.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T15.1.6.1.1.1" class="ltx_p" style="width:433.6pt;">1. Robot vacuums run fully autonomously and automatically. They self-navigate on any flooring type, from hardwoods to carpets, and some models can even be programmed to have a cleaning schedule and to sync to smartphones and smart home devices.</span>
<span id="A5.T15.1.6.1.1.2" class="ltx_p">2. iRobot roomba vacuum works on laminate flooring. In fact they work better on hard floor as compared to carpet since their suction and beater bar is not …</span>
<span id="A5.T15.1.6.1.1.3" class="ltx_p">3. Our top pick among the best vacuums for hardwood floors is the iRobot Roomba 694. iRobot is known as a top choice when it comes to robotic …</span>
<span id="A5.T15.1.6.1.1.4" class="ltx_p">4. Because they lack the power to deep clean, though, robot vacuums are best for flooring other than carpet, such as tile, laminate or hardwood, …</span>
</span>
</td>
</tr>
<tr id="A5.T15.1.7" class="ltx_tr">
<td id="A5.T15.1.7.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T15.1.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T15.1.7.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T15.1.7.1.1.1.1" class="ltx_text ltx_font_bold">Original Response</span></span>
</span>
</td>
</tr>
<tr id="A5.T15.1.8" class="ltx_tr">
<td id="A5.T15.1.8.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T15.1.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T15.1.8.1.1.1" class="ltx_p" style="width:433.6pt;">True.</span>
</span>
</td>
</tr>
<tr id="A5.T15.1.9" class="ltx_tr">
<td id="A5.T15.1.9.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T15.1.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T15.1.9.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T15.1.9.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">ReAlign<span id="A5.T15.1.9.1.1.1.1.1" class="ltx_text ltx_font_upright"> Response (Ours)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.T15.1.10" class="ltx_tr">
<td id="A5.T15.1.10.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T15.1.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T15.1.10.1.1.1" class="ltx_p" style="width:433.6pt;">True.</span>
<span id="A5.T15.1.10.1.1.2" class="ltx_p"><span id="A5.T15.1.10.1.1.2.1" class="ltx_text" style="color:#000000;background-color:#E2F0D9;">Robot vacuums can clean both carpets and hard floors. They are designed to self-navigate on any flooring type, from hardwoods to carpets, and some models can even be programmed to have a cleaning schedule and to sync to smartphones and smart home devices. Additionally, some specific models like the iRobot Roomba 694 are known as top choices for cleaning hardwood floors. However, it is important to note that robot vacuums may lack the power to deep clean carpets compared to hard floors.</span></span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 15: </span>An example of a knowledge-intensive task, including the query, the criteria, the retrieved evidence, the original response, and the <span id="A5.T15.4.1" class="ltx_text ltx_font_smallcaps">ReAlign</span> response. We highlight the content that is more factual and knowledge in 
<span id="A5.T15.5.2" class="ltx_text" style="color:#000000;background-color:#E2F0D9;">green</span>.</figcaption>
</figure>
<figure id="A5.T16" class="ltx_table">
<table id="A5.T16.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A5.T16.1.1" class="ltx_tr">
<td id="A5.T16.1.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T16.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T16.1.1.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T16.1.1.1.1.1.1" class="ltx_text ltx_font_bold">System Prompt</span></span>
</span>
</td>
</tr>
<tr id="A5.T16.1.2" class="ltx_tr">
<td id="A5.T16.1.2.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T16.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T16.1.2.1.1.1" class="ltx_p" style="width:433.6pt;">Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user’s instructions and answers the user’s question better. Your evaluation should consider the readability including clarity, format, and ease of understanding. Well-organized, grammatically correct response is better. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: “[[A]]” if assistant A is better, “[[B]]” if assistant B is better, and “[[C]]” for a tie.</span>
</span>
</td>
</tr>
<tr id="A5.T16.1.3" class="ltx_tr">
<td id="A5.T16.1.3.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T16.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T16.1.3.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T16.1.3.1.1.1.1" class="ltx_text ltx_font_bold">User Prompt</span></span>
</span>
</td>
</tr>
<tr id="A5.T16.1.4" class="ltx_tr">
<td id="A5.T16.1.4.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T16.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T16.1.4.1.1.1" class="ltx_p" style="width:433.6pt;">[User Question]</span>
<span id="A5.T16.1.4.1.1.2" class="ltx_p">{question}</span>
</span>
</td>
</tr>
<tr id="A5.T16.1.5" class="ltx_tr">
<td id="A5.T16.1.5.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T16.1.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T16.1.5.1.1.1" class="ltx_p" style="width:433.6pt;">[The Start of Assistant A’s Answer]</span>
<span id="A5.T16.1.5.1.1.2" class="ltx_p">{answer_a}</span>
<span id="A5.T16.1.5.1.1.3" class="ltx_p">[The End of Assistant A’s Answer]</span>
</span>
</td>
</tr>
<tr id="A5.T16.1.6" class="ltx_tr">
<td id="A5.T16.1.6.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T16.1.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T16.1.6.1.1.1" class="ltx_p" style="width:433.6pt;">[The Start of Assistant B’s Answer]</span>
<span id="A5.T16.1.6.1.1.2" class="ltx_p">{answer_b}</span>
<span id="A5.T16.1.6.1.1.3" class="ltx_p">[The End of Assistant B’s Answer]</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 16: </span>Evaluation prompt for readability.</figcaption>
</figure>
<figure id="A5.T17" class="ltx_table">
<table id="A5.T17.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A5.T17.1.1" class="ltx_tr">
<td id="A5.T17.1.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T17.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T17.1.1.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T17.1.1.1.1.1.1" class="ltx_text ltx_font_bold">System Prompt</span></span>
</span>
</td>
</tr>
<tr id="A5.T17.1.2" class="ltx_tr">
<td id="A5.T17.1.2.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T17.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T17.1.2.1.1.1" class="ltx_p" style="width:433.6pt;">Please act as an impartial judge and evaluate the factuality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider correctness. You will be given a reference answer and the assistant’s answer. Begin your evaluation by comparing the assistant’s answer with the reference answer. Identify and correct any mistakes. Be as objective as possible. After providing your explanation, you must rate the response on a scale of 1 to 10 by strictly following this format: “[[rating]]”, for example: “Rating: [[5]]”.</span>
</span>
</td>
</tr>
<tr id="A5.T17.1.3" class="ltx_tr">
<td id="A5.T17.1.3.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.T17.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T17.1.3.1.1.1" class="ltx_p" style="width:433.6pt;"><span id="A5.T17.1.3.1.1.1.1" class="ltx_text ltx_font_bold">User Prompt</span></span>
</span>
</td>
</tr>
<tr id="A5.T17.1.4" class="ltx_tr">
<td id="A5.T17.1.4.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T17.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T17.1.4.1.1.1" class="ltx_p" style="width:433.6pt;">[Question]</span>
<span id="A5.T17.1.4.1.1.2" class="ltx_p">{question}</span>
</span>
</td>
</tr>
<tr id="A5.T17.1.5" class="ltx_tr">
<td id="A5.T17.1.5.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A5.T17.1.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T17.1.5.1.1.1" class="ltx_p" style="width:433.6pt;">[The Start of Reference Answer]</span>
<span id="A5.T17.1.5.1.1.2" class="ltx_p">{ref_answer}</span>
<span id="A5.T17.1.5.1.1.3" class="ltx_p">[The End of Reference Answer]</span>
</span>
</td>
</tr>
<tr id="A5.T17.1.6" class="ltx_tr">
<td id="A5.T17.1.6.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T17.1.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T17.1.6.1.1.1" class="ltx_p" style="width:433.6pt;">[The Start of Assistant’s Answer]</span>
<span id="A5.T17.1.6.1.1.2" class="ltx_p">{answer}</span>
<span id="A5.T17.1.6.1.1.3" class="ltx_p">[The End of Assistant’s Answer]</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 17: </span>Evaluation prompt for factuality.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2402.12218" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2402.12219" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2402.12219">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2402.12219" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2402.12220" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Mar  5 17:45:32 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2402.12219
The quality of finetuning data is crucial for aligning large language models (LLMs)  with human values.
Current methods to improve data quality are either labor-intensive or prone to factual errors caused by LLM hallucinations.
This paper explores elevating the quality of existing instruction data to better align with human values, introducing a simple and effective approach namedReAlign, whichreformatsthe responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence.
This approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques.
Experimentally,ReAlignsignificantly boosts the general alignment ability, math reasoning, factuality, and readability of the LLMs. Encouragingly,withoutintroducing any additional data or advanced training techniques, and merely by reformatting the response, LLaMA-2-13B’s mathematical reasoning ability onGSM8Kcan be improvedfrom 46.77% to 56.63%in accuracy.
Additionally, a mere 5% ofReAligndata yields a 67% boost in general alignment ability measured by the Alpaca dataset.
This work highlights the need for further research into thescienceandmechanistic interpretabilityof LLMs. We have made the associated code and data publicly accessible to support future studies athttps://github.com/GAIR-NLP/ReAlign.
abstract
Reformatted Alignment
Abstract
2402.12219
1IntroductionAlignment has been witnessed to be an effective technique for aligning large language models (LLMs)  to human values and human intent, which usually requires fine-tuning on a large amount of synthetic data derived from LLMsor human-annotated instruction data.Recent studies, notably byhighlight the critical role of instruction data quality in this process.
Numerous workshave contributed to enhancing instruction quality by focusing on the diversity and complexity of input queries as well as the quality of responses. These efforts can be divided into two primary approaches.
The first approach, advocated byand, involves the manual creation of high-quality data. Although this method creates complex queries and factually correct and highly readable responses, it is labor-intensive and challenging to scale.
The second approach revolves around the automated extraction of high-quality instructions from existing datasets due to their extensive availability.
However, this method inherits the limitations associated with distilled data, such as containing factually incorrect contentand the format and style of the generated response are often determined by distilled LLMs’ preference.In this paper, instead of focusing on the creation of instruction data from scratch, we investigate how existing instruction data can be made higher quality and better aligned with human values.
We propose a simple and effective method, namedReAlign, which is orthogonal to the above existing approaches.
Specifically,ReAlignnecessitates a base instruction dataset, which can be sourced from extensive existing supervised datasets (e.g.,GSM8K) , or publicly available instruction data compiled through various methods (e.g., Self-Instruct, Evol-Instruct, and Self-Alignment) .
TheReAlignprocess unfolds in three main steps.
The first step involvescriteria definition(§3.1) , where humans define their preferences (e.g., the preferred format of responses)  in various scenarios in the form of natural language.
In this paper, we meticulously define criteria for 46 distinct scenarios.
The second step,retrieval augmentation(§3.2) , broadens the knowledge base for knowledge-intensive tasks like open-domain QA and fact verification. This is achieved by incorporating additional information, thereby improving the factuality and informativeness of responses.
The final step,reformatting(§3.3) , aims to re-align the responses with the pre-established criteria and the collated evidence, guaranteeing outputs that are both structured and substantiated.
As demonstrated in Fig.2, the realigned response provides a better format and a clearer chain of thoughts.The underlyingphilosophyofReAlignis to re-coordinate the roles of humans and LLMs in the alignment process, leveraging their complementary strengths – humans articulate their preferences, and LLMs, in turn, reconstruct instructions based on their generative power (e.g., instruction-following ability) , without directly using distilled LLM knowledge.
Through this collaborative synergy, we expect the generated instruction data to be not only more contextually precise but also more closely aligned with human preferences.We operationalize this idea on five types of existing instruction data, where three are general datasets (i.e.,Open-Platypus,No Robots, andAlpaca)  and two are mathematical datasets (i.e.,GSM8KandMATH) .
The performance ofReAlignhas been validated across various well-established benchmarks, including AlpacaEval, MT-Bench, and Vicuna-Benchfor general alignment, as well asGSM8KandMATHfor mathematical reasoning. Additionally, it has also been evaluated for factuality and readability, demonstrating its proficiency.
In particular,ReAlignsignificantly boosts math reasoning, even up to 9.86% onGSM8Kfor LLaMA-2-13B.
Notably, we find that only 5% of theReAligndata yields a 67% boost in general alignment ability compared to the fullReAligndata based on theAlpacadataset, indicating that only a small amount of data is required to learn style and format.Figure 2:ReAlignrealigns the original response with the pre-defined criteria to be a better format. The original response is from theOpen-Platypusdataset. The complete version is shown in Tab.13.
introduction
Reformatted Alignment
1Introduction
2402.12219
2Related Work2.1Instruction CreationCreating instructional data significantly improves LLMs’ alignment abilities.
High-quality instruction generation traditionally depends on human annotation for tasks like query writing, response drafting, and preference indication. This approach produces premium open-source datasets (e.g.,Open-PlatypusandOpenAssistant)  and supports advanced LLMs (e.g., LIMAand LLaMA-2) , but it’s hard to scale due to high labor costs and the need for domain-specific expertise.
Many studies have explored using LLMs (e.g., GPT-3and GPT-4)  to generate instruction data. Techniques like unnatural instructionsand self-instructutilize GPT-3’s in-context learning with seed data to generate instructions, while evol-instructgenerates more complex and varied instructions through ChatGPT. Recently, training with self-generated data has yielded excellent results, achieving self-alignment. While it can be easily scaled up, this approach inherits the drawbacks of LLMs (e.g., factual errors) .
Our proposed method contrasts by offering automatic data quality improvement with minimal labor and reduced factual errors.2.2Instruction SelectionAfter the discovery of “quality is all you need”, instruction selection has been paid attention to, aiming at selecting a small number of the highest-quality samples from a large amount of instruction data as a training dataset.evaluates the dataset’s quality by utilizing the evaluation dataset loss to fit the natural language indicators of the dataset.proposes to use ChatGPT directly to score the data, whileproposes to score the data using the trained model directly to save costs.proposes to tag samples within SFT datasets based on semantics and intentions and define instruction diversity and complexity regarding tags to rank data.introduces a self-guided approach that utilizes a new indicator, Instruction-Following Difficulty (IDF) , to score data by identifying gaps in a model’s responses versus its autonomous generation capability.trains two scorers to evaluate the complexity of the instruction and the quality of the response, respectively, and then uses the embedding distance to determine the diversity to select high-quality data.
However, the above works usually mine from distilled datasets because the large scale of distilled datasets is available, thereby inheriting the drawbacks of distilled data and suffering from the hallucination of LLMs.2.3Instruction TuningInstruction tuning aims to reinforce the model’s instruction-following capabilities and align LLMs to human values.
Early instruction tuning was designed to improve cross-task generalization capabilities, in which they usually scale up the quantity and the diversity of tasks. Recent works no longer explicitly define tasks, but extend to more generalized capabilities, especially for scenarios of real-world questions.
Differently, our work utilizes the future of the task to design a better format for it, which further improves the quality of the data.GroupTasksGenerationquestion generation; story generation; poem generation; email generation; data generation; text-to-text translationBrainstormingadvice giving; recommendations; how-to generation; planningCodecode correction; code simplification; explain code; text-to-code translation; code-to-code translation;language learning questions; code language classification; code-to-text-translationRewritinginstructional rewriting; language polishing; paraphrasing; text correctionExtractioninformation extraction; keywords extraction; table extractionSummarizationtitle generation; text summarization; note summarizationConversationopen qa; closed qa; fact verification; value judgment; roleplay; explain answerEducation.natural language tutor; exam problem tutor; ai tutor; math puzzles; fill in the blankClassificationgeneral classification; ordering; sentiment analysis; language classification; topic classificationOthersrejecting; othersTable 1:The category of tasks. “Education.” denotes Specialized Educational Dialog.Email GenerationIt is an email-writing task. Here is a general guideline for creating a well-structured and professional email:1. Subject Line:Write a clear and concise subject line that accurately summarizes the content of your email …2. Salutation:Begin your email with a formal salutation such as ”Dear [Recipient’s Name],” …3. Introduction:Start your email with a brief introduction …4. Body:This is the main content of your email …5. Politeness and Tone:Maintain a polite and respectful tone throughout your email …6. Closing:Conclude your email with a closing remark, such as ”Thank you,” or ”Best regards,” followed by your name …7. Signature:Include your full name, job title, and contact information (e.g., phone number, email address)  …8. Attachments:If you need to include attachments, mention them …9. Proofread:Before sending the email, proofread it for any grammatical or spelling errors …The best emails are short, direct, professional, and scannable for the recipient. Follow a formal business email structure unless you havean established casual rapport with the recipient.Table 2:An example of the format for the “email generation” task.Figure 3:An overview of ourReAlignincluding three steps. KILT denotes Knowledge Intensive Language Tasks.
body
Reformatted Alignment
2Related Work
2402.12219
3ReAlignGiven a base instruction dataset, whereandare the input query and response respectively,ReAlignaims to improve the quality of responses by three steps as shown in Fig.3:
(1)  Criteria Definition: defining the criteria including tasks and formats for each task, (2)  Retrieval Augmentation: retrieving relevant external information for the knowledge-intensive tasks, and (3)  Reformatting: reformatting the original response based on the guidance consisting of hand-written format and the retrieved information. An overview of our method is shown in Fig.3.3.1Criteria DefinitionThe predefined criteria consist of the tasks and the corresponding formats:Tasks.Clearly defining tasks is crucial to subsequently devising tailored formats, as the optimal format varies across distinct tasks.
In this paper, we followto define 46 different tasks, categorized into 10 major groups, as shown in Tab.1. The detailed description for each task is shown in Tab.8, §B. We also train a task classifier, detailed in §C.Format.Due to the distinct formatting requisites associated with diverse tasks, we meticulously devised tailored formatsfor each task based on the task definition and description, encompassing considerations such as organizational structure, section content, and output modality.
This format, which has been carefully designed for different tasks, is more readable than the generic format.
Each format has a task name and a detailed format description. We show an example of a format for “email generation” in Tab.2(The complete version is shown in Tab.14) .In this step, we input queryto the task classifier(detailed in §C)  to acquire the category:and then obtain the corresponding format.3.2Retrieval AugmentationKnowledge-intensive language tasks (KILT) , such as open-domain QA and fact verification, usually require large and external knowledge sources as the evidence to ensure the factuality.
Thus, we followto choose five knowledge-intensive tasks and use the queryto retrieve relevant information as our evidence. The tasks for retrieval augmentation are shown in Tab.8. Specifically, we followand use the Google Search API as our retrieverprovided by Serper111https://serper.dev/to retrieve the most relevant search snippets included in the API’s answer. We then parse the response to obtain different types of snippets such as answer boxes, knowledge graphs, and organic search results. Finally, we choose the top-snippets and filter them as our evidence:We show an example of a knowledge-intensive language task in Tab.15, demonstrating that retrieval augmentation enables the response more factual and informative.3.3Reformatting3.3.1RewritingIn this step, we leverage large language models (e.g., ChatGPT)  to rewrite the responsebased on the given formatand retrieved evidence(for knowledge-intensive tasks) .
Since certain queries have additional requirements (e.g., specific formatting or specified information) , an adaptive rewriting strategy is employed.
This approach involves initially using LLMs to determine whether the format matches the query requirements.
Subsequently, if it matches, the LLMs rewrite the response accordingly.
We divide the tasks into two categories:Non-knowledge-intensive tasksFor the non-knowledge-intensive tasks, we decide to rewrite a part of the tasks. This decision stems from the observation that certain tasks are not amenable to a standardized format, exemplified by instances such as story generation and poem generation (see Tab.8for details) . We guide LLMs to rewrite the original responses, organizing the query, original response, and the formattogether via the prompt in Tab.11:whereis the reformatted response.Knowledge-intensive tasks.For the knowledge-intensive tasks, we additionally utilize the retrieved evidencecompared to non-knowledge-intensive tasks. Specifically, We guide LLM to rewrite the original response, organizing the query, original response, format, and the retrieved evidencetogether via the prompt in Tab.12:3.3.2Post-processingLength filtering.We find that LLMs sometimes fail to reformat and only output the changed sentences, whose output length plummets.
To filter out the data that fails to be reformatted, we keep the original response instead of using the reformatted response that is less than half the length of the original response.Task-based filtering.To mitigate the problem of error propagation in task classification, we design filtering rules for specific tasks:(i) For code-related tasks (e.g., “code correction”) , the keyword matching rule is employed to ascertain whether both the original and the reformatted versions contain code. If only one of the original responses or the reformatted response incorporates code, it signifies a failure in reformatting, and the original response is retained.(ii) For the “exam problem tutor” task, reformatted responses that do not contain the accurate result will not be accepted.(iii) For the “planning” task, if the query does not contain a planning-related keyword (e.g., plan or planning) , the original answer is retained.Finally, we could acquire the reformatted dataset(denotes asReAligndataset) .
body
Reformatted Alignment
3ReAlign
2402.12219
4Experiments4.1DatasetsFor evaluation of general ability, we select two high-quality manual datasets and one distillation dataset for instruction tuning:
(1) Open-Platypusis an amalgamation of 11 open-source datasets, carefully curated to enhance LLM performance in STEM and logical domains. It consists of 25k questions, with around 90% written by humans and the rest generated by LLM.
(2) No Robotsis a high-quality dataset of 10k instructions and demonstrations created by skilled human annotators.
(3) Alpacais an open-source instruction tuning dataset generated from text-davinci-003by theSelf-Instructmethod, containing 52k samples.
Additionally, we also choose two manual datasets to evaluate the math reasoning after usingReAlign:
(4) GSM8Kis a high-quality grade school math problems dataset created by human problem writers, consisting of 7.5k training problems and 1k test problems.
(5) MATHis a dataset of mathematics competitions problems, including 7.5k for training and 5k for testing.4.2ModelsWe select two well-known open-source base models for fine-tuning:
(1) LLaMA-2-13Bis a open-source pre-trained model using 2T tokens.
(2) Mistral-7Bis the current state-of-the-art base language model at the 7B parameter scale.4.3EvaluationWe evaluateReAlignon general alignment and specific alignment ability including math reasoning, factuality, and readability.4.3.1General AlignmentTo evaluate the general alignment ability, we followto employ the most widely recognized benchmarks, including:AlpacaEval,MT-Bench,Vicuna-Bench. Specifically, we use GPT-3.5 and Auto-J (detailed in §D)  as the evaluators for AlpacaEval due to the cost of GPT-4, which has an extremely strong correlation with human, and GPT-4 for MT-Bench and Vicuna-Bench.4.3.2Specific AlignmentWe evaluate specific perspectives for alignment, including math reasoning, factuality, and readability.ModelDatasetAlpacaEvalMT-BenchVicuna-BenchOverallGPT-3.5 (%) Auto-JFirstSecondAverageLLaMA-2-13BOpen-Platypus55.714.936.695.165.948.286.18+ReAlign58.204.816.894.865.888.456.24No Robots44.254.565.805.155.487.315.44+ReAlign48.134.656.045.205.627.515.65Alpaca46.084.655.554.164.866.555.17+ReAlign49.194.745.834.715.276.845.44Mistral-7BOpen-Platypus59.635.157.295.886.588.966.66+ReAlign61.335.157.436.186.808.866.74No Robots44.224.625.954.945.447.325.45+ReAlign48.264.766.144.795.467.685.68Alpaca51.244.776.065.265.667.145.67+ReAlign52.674.826.505.035.767.335.79Table 3:The results of the general alignment ability on the original datasets and theReAligndatasets.Boldindicates the best result on each dataset. For AlpacaEval, GPT-3.5 denotes the winning rate obtained by using GPT-3.5 as the evaluator. Auto-J denotes the quality of the model’s responses evaluated in a point-wise manner using Auto-J. For MT-Bench, we report the result of the first turn, the second turn, and the average, respectively. For Overall, we calculate the average of AlpacaEval’s winning rate for GPT-3.5 divided by 10, the results for Auto-J, the average MT-Bench results, and the results for Vicuna-Bench.Math Reasoning.To evaluate math reasoning, we finetune LLaMA-2-13B and Mistral-7B onGSM8KandMATHtraining datasets, respectively, and test afterward. The prompt template for training and testing is‘‘Question:\n{input}\n Answer:\nLet’s think step by step.\n’’.
Since both datasets consist of math problems in the same style, we apply forced rewriting instead of adaptive, which does not require the determination of whether the query and format match but rather mandates a rewriting. We determine the accuracy by extracting the last number from the responses and comparing it directly to the ground truth.ModelDatasetGSM8KMATHOverallLLaMA-2-13BGSM8K46.775.0225.90+ReAlign56.635.4631.05MATH14.486.1410.31+ReAlign25.177.1416.16Mistral-7BGSM8K57.627.6832.65+ReAlign62.479.0235.75MATH28.3513.1820.77+ReAlign38.2115.3026.76Table 4:The results of math reasoning onGSM8K,MATHand them +ReAlignbased on LLaMA-2-13B and Mistral-7B. We test models on bothGSM8KandMATHtest sets. We report the accuracy by exact matching.Boldindicates the best result..Factuality.To evaluate the factuality of models, we randomly select 100 cases from the Natural Questions dataset (NQ) , a public Q&A dataset rich in fact-based queries and their verified answers.
We employ GPT-4 to rate these instances on a factuality scale of 1 to 10, considering the question, the response, and the ground truth (referred to as the factuality score) . The evaluation prompt is shown in Tab.17.Readability.To evaluate the readability, we compare a model trained on the original dataset against another model on the dataset enhanced withReAlign, using human and GPT-4 evaluations on the Vicuna-Bench dataset. Since the vicuna bench contains fewer complex questions (e.g., code and math) , the judge can focus on the format rather than the result. We design an evaluation prompt prioritizing readability, refer to Tab.16, and randomize response positions to eliminate bias.4.4ResultsReAlignImproves General Alignment Ability.Following, we conduct experiments on AlpacaEval, MT-Bench, and Vicuna-Bench to evaluate the general alignment ability.
From Tab.3, we can see an increase in almost all three datasets and benchmarks on both the LLaMA-2-13B and Mistral-7B models, showing thatReAligncan significantly improve models’ response quality and conversation ability. Additionally, from the results of MT-Bench, we can see thatReAligncan improve the performance of the second turn of conversations on half the datasets even though it only rewrites the first turn of the instruction data.ReAlignCan Boost Math Reasoning.To evaluate the effect ofReAlignon math reasoning, we applyReAligntoGSM8KandMATHdatasets. As shown in Tab.4,ReAligncan dramatically boost the math reasoning on both datasets, even up to 9.86% onGSM8Kusing LLaMA-2-13B.
Remarkably,ReAlignenhances generalization, demonstrated by cross-domain performance boosts. Specifically, training models using theMATHdataset yields notable improvements in theGSM8Ktest results, and vice versa. For instance, it has been observed that training on theMATHdataset can augmentGSM8Kperformance by 10.69% based on LLaMA-2-13B.
One possible reason is thatReAligncan provide more and clearer intermediate steps and explanations, thereby bolstering the reasoning ability of models.ReAlignCan Enhance Factuality.To evaluate the factuality, we employReAligntoOpen-Platypus,No Robots, andAlpacadatasets with LLaMA-2-13B, subsequently comparing the response to ground truth in NQ samples.
Fig.4showsReAlignelevates the factuality, highlighting its efficacy. This improvement is probably due to the addition of retrieval augmentation.Figure 4:The results of the factuality score.(a) GPT-4 Judgments(b) Human JudgmentsFigure 5:The readability win-rate of the original dataset +ReAlignagainst the original dataset based on LLaMA-2-13B, judged by GPT-4 and human.ReAlignCan Improve Readability.To evaluate the readability of the responses, we use a readability evaluation prompt (refer to Tab.16)  to guide GPT-4 and human to compare the model trained on the original dataset with the model trained with the addition ofReAlign. As shown in Fig.5, we see thatReAligncan improve the readability of three datasets, especially in theOpen-Platypusdataset (i.e., 18.7% improvements in GPT-4 judgments) . It demonstrates that designing different formats for different tasks and reformatting them can effectively improve readability. In addition, human tends to provide more ties for judgments compared to GPT-4. A possible reason is thatReAligncan provide better structure, causing GPT-4 to be limited to surface formats ignoring content and deep structure. In contrast, humans can read more carefully not being limited to surface formats.DatasetResponse Len.ReAlign%Open-Platypus224.92-+ReAlign206.9128.5%No Robots211.99-+ReAlign211.5415.9%Alpaca65.51-+ReAlign72.3829.9%GSM8K130.59-+ReAlign327.65100%MATH243.73-+ReAlign375.35100%Table 5:The datasets analysis includes original datasets and them +ReAlign. Response Len. is the average number of tokens of the responses.ReAlign% denotes the percentage of successful reformatting after the adaptive rewriting.(a) BBH(b) AGIEvalFigure 6:The results of the knowledge abilities, including the Big Bench Hard (BBH)  (3-shot) , and AGIEval (zero-shot) . We evaluate the abilities across theOpen-Platypus,No Robots, andAlpacadatasets, based on LLaMA-2-13B.DatasetGeneral Align.Know. Ab.FSOpen-Platypus6.1839.655.1+ReAlign6.2441.355.5W/o RAG6.1840.65.3W/o Adaption6.1739.85.6Table 6:Ablation study results show that removing retrieval augmentation is indicated by ”W/o RAG” and removing adaptive rewriting by ”W/o Adaption” inReAlign. “General Align.” and “Know. Ab.” denotes general alignment ability and Knowledge Ability, which are the average results. FS denotes Factuality Score.Bolddenotes the best.4.5Analysis4.5.1Datasets AnalysisFirst, we compare the change in the length of responses (i.e., the number of tokens)  between the original datasets and the addition ofReAlign, finding thatOpen-Platypusbecomes shorter andNo Robotsdoes not change much, whileAlpaca,GSM8K, andMATHbecome longer (see Tab.5) .
Second, we calculate the percentage of responses for which the adaptive rewriting method selects rewrite by edit distance (the results are shown in Tab5) . Specifically, we compute the edit distance (including substitution, deletion, and insertion)  on a word basis, then divide the edit distance by the length of the longest of the original and rewritten responses to obtain the edit rate, and finally record those with an edit rate greater than 0.2 as rewritten, and the rest as unrewritten. ForGSM8KandMATHdatasets, all data are ReAligned as adaptive rewriting was not applied to them.4.5.2Alignment TaxWhen the model is fine-tuned on theReAligndataset, a question worth exploring is whether there is a drop in knowledge ability even as alignment ability improves.
To evaluate the knowledge ability, we followto employ the following benchmarks:Big Bench Hard (BBH) andAGIEval, which is multiple choices knowledge-intensive QA task.
As shown in Fig.6, we can see thatReAlignhas little effect on the knowledge-based tasks, indicating that our approach does not impair the knowledge in the original dataset.
It is worth noting that in some casesReAlignwill also provide a significant boost to knowledge, such asOpen-Platypuson AGIEval. Possible reasons are that a well-defined format can facilitate the accuracy of the knowledge-based tasksand that retrieving external information can augment knowledge.Figure 7:The scaling trends inReAligndata percentage, including general alignment ability and knowledge ability. We conduct the experiment in theAlpacadataset based on LLaMA-2-13B.4.5.3Ablation StudiesWe rewrite two variants of theOpen-Platypusdataset and train them based on LLaMA-2-13B for ablation studies:(1) W/o Retrieval Augmentation: We remove the retrieval augmentation fromReAlignand rewrite all tasks without evidences. As shown in Tab.6, the general alignment ability, knowledge ability, and factuality score (FS)  are reduced, indicating the effectiveness of retrieval augmentation. Interestingly, the FS metrics are higher without RAG than in the original dataset, suggesting thatReAlignalso has the potential to improve the factuality.(2) W/o Adaption: We remove the adaptive rewriting fromReAlignand use force rewriting. Tab.6shows the general alignment and knowledge ability decrease. This may be because forced rewriting, while making the responses more structured, ignores the question’s requirements, weakening the instruction-following ability. In addition, FS has increased, probably because forced rewriting leads to a larger amount ofReAligndata, introducing more retrieved knowledge and boosting factuality.4.5.4The Scaling Law ofReAlignWe experiment to explore the impact of the number ofReAlign. Specifically, we randomly sample a(, withFullbeing)  ofReAlignAlpacadata, and fill in the remainder with original responses. The original Alpaca dataset corresponds to.
Interestingly, we find that only 5% of theReAligndata yields a 67% boost in general alignment ability compared to the entireReAligndata (see Fig.7) .
This suggests that only a small amount of data is required to learn style and format, to expose the knowledge and capabilities that were already acquired during pretraining. Additionally, the knowledge capability continues to improve as the amount ofReAligndata improves.4.5.5Case StudyWe show a case from the MT-Bench test set in Tab.7. This example shows that the response given by theReAlignmodel has a better format.
body
Reformatted Alignment
4Experiments
2402.12219
5ConclusionIn this work, we proposeReAlign, a simple and effective method for alignment, which automatically improves the quality of the existing instruction datasets while minimizing labor costs and hallucinations. We create five new high-quality datasets fromOpen-Platypus,No Robots,Alpaca,GSM8K, andMATHand high-quality manual-written natural language formats. Experiments demonstrate thatReAlignsignificantly boosts general alignment ability, math reasoning, factuality, and readability without impairing knowledge ability. Last but not least, we release the code and data to facilitate future research.
body
Reformatted Alignment
5Conclusion
2402.12219
LimitationsFirst, our approach relies on the ability to reformatting models, which is currently less effective in open-source models (e.g., LLaMA2)  but more costly in closed-source models (e.g., GPT-4) .
Second, the task categories we define cannot cover all tasks in reality, as real questions may be more complex and involve multiple tasks. Therefore, it is necessary to define more tasks and formats for a wide range of diverse and regional scenarios.
Last, applyingReAlignonly to single-turn conversations has the potential to hurt the alignment ability of the second-turn conversations, hence extendingReAlignto multi-turn conversation would also be valuable.
body
Reformatted Alignment
Limitations
2402.12219
Ethics StatementWe take ethical considerations very seriously. In this paper, both the datasets and models are publicly available and have been widely adopted by researchers. We ensure that the findings and conclusions of this paper are reported accurately and objectively.
body
Reformatted Alignment
Ethics Statement
2402.12219
AcknowledgementsWe thank the GAIR members for reviewing our paper and giving valuable feedback. We appreciate the authors infor providing the training codebase and the helpfulness.
conclusion
Reformatted Alignment
Acknowledgements