<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.21417] Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models</title><meta property="og:description" content="Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both.
Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow o…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.21417">

<!--Generated on Mon Aug  5 14:53:54 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">Dancing in Chains: Reconciling Instruction Following 
<br class="ltx_break">and Faithfulness in Language Models</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Zhengxuan Wu<sup id="id17.17.id1" class="ltx_sup"><span id="id17.17.id1.1" class="ltx_text ltx_font_italic">1⋄</span></sup>,  Yuhao Zhang<sup id="id18.18.id2" class="ltx_sup"><span id="id18.18.id2.1" class="ltx_text ltx_font_italic">2⋄∗</span></sup>,  Peng Qi<sup id="id19.19.id3" class="ltx_sup"><span id="id19.19.id3.1" class="ltx_text ltx_font_italic">3⋄∗</span></sup>,  Yumo Xu<sup id="id20.20.id4" class="ltx_sup"><span id="id20.20.id4.1" class="ltx_text ltx_font_italic">4∗</span></sup>,  
<br class="ltx_break"><span id="id5.5.1" class="ltx_text ltx_font_bold">Rujun Han<sup id="id5.5.1.1" class="ltx_sup"><span id="id5.5.1.1.1" class="ltx_text ltx_font_medium ltx_font_italic">5</span></sup></span>,  <span id="id6.6.2" class="ltx_text ltx_font_bold">Yian Zhang<sup id="id6.6.2.1" class="ltx_sup"><span id="id6.6.2.1.1" class="ltx_text ltx_font_medium ltx_font_italic">6⋄</span></sup></span>,  <span id="id7.7.3" class="ltx_text ltx_font_bold">Jifan Chen<sup id="id7.7.3.1" class="ltx_sup"><span id="id7.7.3.1.1" class="ltx_text ltx_font_medium ltx_font_italic">4</span></sup></span>,  <span id="id8.8.4" class="ltx_text ltx_font_bold">Bonan Min<sup id="id8.8.4.1" class="ltx_sup"><span id="id8.8.4.1.1" class="ltx_text ltx_font_medium ltx_font_italic">4</span></sup></span>,  <span id="id10.10.6" class="ltx_text ltx_font_bold">Zhiheng Huang<sup id="id10.10.6.1" class="ltx_sup"><span id="id10.10.6.1.1" class="ltx_text ltx_font_medium ltx_font_italic">7⋄</span></sup>
<br class="ltx_break"><sup id="id10.10.6.2" class="ltx_sup"><span id="id10.10.6.2.1" class="ltx_text ltx_font_medium">1</span></sup></span>Stanford University  <sup id="id21.21.id5" class="ltx_sup">2</sup>Samaya AI  <sup id="id22.22.id6" class="ltx_sup">3</sup>Orby AI  <sup id="id23.23.id7" class="ltx_sup">4</sup>AWS AI Labs
<br class="ltx_break"> <sup id="id24.24.id8" class="ltx_sup">5</sup>Google  <sup id="id25.25.id9" class="ltx_sup">6</sup>NVIDIA  <sup id="id26.26.id10" class="ltx_sup">7</sup>Denser.ai
<br class="ltx_break"><span id="id27.27.id11" class="ltx_text ltx_font_typewriter">wuzhengx@stanford.edu</span>  <span id="id28.28.id12" class="ltx_text ltx_font_typewriter">yuhao@samaya.ai</span> <span id="id29.29.id13" class="ltx_text ltx_font_typewriter">peng@orby.ai</span> <span id="id30.30.id14" class="ltx_text ltx_font_typewriter">yumomxu@amazon.com
<br class="ltx_break"></span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_note_type">footnotetext: </span>Equal contribution. <sup id="footnotex1.1" class="ltx_sup"><span id="footnotex1.1.1" class="ltx_text ltx_font_italic">⋄</span></sup>Work done at AWS AI Labs.</span></span></span>
<p id="id31.id1" class="ltx_p">Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both.
Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instruction-tuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies on <span id="id31.id1.1" class="ltx_text ltx_font_bold">Re</span>jection Sampling for Continued <span id="id31.id1.2" class="ltx_text ltx_font_bold">Se</span>lf-instruction <span id="id31.id1.3" class="ltx_text ltx_font_bold">T</span>uning (<span id="id31.id1.4" class="ltx_text ltx_font_bold ltx_font_smallcaps">ReSet</span>), which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as training <span id="id31.id1.5" class="ltx_text ltx_font_bold ltx_font_smallcaps">ReSet</span> with high-quality, yet substantially smaller data (three-fold less) yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs.</p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<div id="p1.16" class="ltx_block ltx_align_bottom">
<p id="p1.16.17" class="ltx_p"><span id="p1.16.17.1" class="ltx_text ltx_font_bold">Dancing in Chains: Reconciling Instruction Following 
<br class="ltx_break">and Faithfulness in Language Models</span></p>
<br class="ltx_break ltx_centering">
<p id="p1.16.16" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p1.16.16.16" class="ltx_text ltx_inline-block" style="width:0.0pt;">

<span id="p1.16.16.16.16" class="ltx_tabular ltx_align_top">
<span id="p1.4.4.4.4.4" class="ltx_tr">
<span id="p1.4.4.4.4.4.4" class="ltx_td ltx_align_center"><span id="p1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Zhengxuan Wu<sup id="p1.1.1.1.1.1.1.1.1" class="ltx_sup"><span id="p1.1.1.1.1.1.1.1.1.1" class="ltx_text ltx_font_medium ltx_font_italic">1⋄</span></sup></span>,<span id="p1.2.2.2.2.2.2.2" class="ltx_text ltx_font_bold">  Yuhao Zhang<sup id="p1.2.2.2.2.2.2.2.1" class="ltx_sup"><span id="p1.2.2.2.2.2.2.2.1.1" class="ltx_text ltx_font_medium ltx_font_italic">2⋄∗</span></sup></span>,<span id="p1.3.3.3.3.3.3.3" class="ltx_text ltx_font_bold">  Peng Qi<sup id="p1.3.3.3.3.3.3.3.1" class="ltx_sup"><span id="p1.3.3.3.3.3.3.3.1.1" class="ltx_text ltx_font_medium ltx_font_italic">3⋄∗</span></sup></span>,<span id="p1.4.4.4.4.4.4.4" class="ltx_text ltx_font_bold">  Yumo Xu<sup id="p1.4.4.4.4.4.4.4.1" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.1.1" class="ltx_text ltx_font_medium ltx_font_italic">4∗</span></sup></span>,<span id="p1.4.4.4.4.4.4.5" class="ltx_text ltx_font_bold"></span></span></span>
<span id="p1.9.9.9.9.9" class="ltx_tr">
<span id="p1.9.9.9.9.9.5" class="ltx_td ltx_align_center"><span id="p1.5.5.5.5.5.1.1" class="ltx_text ltx_font_bold">Rujun Han<sup id="p1.5.5.5.5.5.1.1.1" class="ltx_sup"><span id="p1.5.5.5.5.5.1.1.1.1" class="ltx_text ltx_font_medium ltx_font_italic">5</span></sup></span>,  <span id="p1.6.6.6.6.6.2.2" class="ltx_text ltx_font_bold">Yian Zhang<sup id="p1.6.6.6.6.6.2.2.1" class="ltx_sup"><span id="p1.6.6.6.6.6.2.2.1.1" class="ltx_text ltx_font_medium ltx_font_italic">6⋄</span></sup></span>,  <span id="p1.7.7.7.7.7.3.3" class="ltx_text ltx_font_bold">Jifan Chen<sup id="p1.7.7.7.7.7.3.3.1" class="ltx_sup"><span id="p1.7.7.7.7.7.3.3.1.1" class="ltx_text ltx_font_medium ltx_font_italic">4</span></sup></span>,  <span id="p1.8.8.8.8.8.4.4" class="ltx_text ltx_font_bold">Bonan Min<sup id="p1.8.8.8.8.8.4.4.1" class="ltx_sup"><span id="p1.8.8.8.8.8.4.4.1.1" class="ltx_text ltx_font_medium ltx_font_italic">4</span></sup></span>,  <span id="p1.9.9.9.9.9.5.5" class="ltx_text ltx_font_bold">Zhiheng Huang<sup id="p1.9.9.9.9.9.5.5.1" class="ltx_sup"><span id="p1.9.9.9.9.9.5.5.1.1" class="ltx_text ltx_font_medium ltx_font_italic">7⋄</span></sup></span></span></span>
<span id="p1.13.13.13.13.13" class="ltx_tr">
<span id="p1.13.13.13.13.13.4" class="ltx_td ltx_align_center"><sup id="p1.13.13.13.13.13.4.1" class="ltx_sup">1</sup>Stanford University  <sup id="p1.13.13.13.13.13.4.2" class="ltx_sup">2</sup>Samaya AI  <sup id="p1.13.13.13.13.13.4.3" class="ltx_sup">3</sup>Orby AI  <sup id="p1.13.13.13.13.13.4.4" class="ltx_sup">4</sup>AWS AI Labs</span></span>
<span id="p1.16.16.16.16.16" class="ltx_tr">
<span id="p1.16.16.16.16.16.3" class="ltx_td ltx_align_center"> <sup id="p1.16.16.16.16.16.3.1" class="ltx_sup">5</sup>Google  <sup id="p1.16.16.16.16.16.3.2" class="ltx_sup">6</sup>NVIDIA  <sup id="p1.16.16.16.16.16.3.3" class="ltx_sup">7</sup>Denser.ai</span></span>
<span id="p1.16.16.16.16.17" class="ltx_tr">
<span id="p1.16.16.16.16.17.1" class="ltx_td ltx_align_center"><span id="p1.16.16.16.16.17.1.1" class="ltx_text ltx_font_typewriter">wuzhengx@stanford.edu</span>  <span id="p1.16.16.16.16.17.1.2" class="ltx_text ltx_font_typewriter">yuhao@samaya.ai</span> <span id="p1.16.16.16.16.17.1.3" class="ltx_text ltx_font_typewriter">peng@orby.ai</span> <span id="p1.16.16.16.16.17.1.4" class="ltx_text ltx_font_typewriter">yumomxu@amazon.com</span></span></span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Aligning language models (LMs) with human preferences becomes increasingly important. One main objective is to train LMs to follow human instructions (e.g., answering open-ended questions)
while being faithful (e.g., grounding responses in the given context). However, LMs often suffer from failing to follow human instructions <cite class="ltx_cite ltx_citemacro_cite">Kadavath et al. (<a href="#bib.bib20" title="" class="ltx_ref">2023</a>); Chen et al. (<a href="#bib.bib3" title="" class="ltx_ref">2023</a>); Ji et al. (<a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite> or making up facts that are not grounded in context <cite class="ltx_cite ltx_citemacro_cite">Zhang et al. (<a href="#bib.bib49" title="" class="ltx_ref">2023</a>); Wang et al. (<a href="#bib.bib44" title="" class="ltx_ref">2023a</a>); Huang et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023</a>); Ghosh et al. (<a href="#bib.bib14" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">We trace this problem back to commonly used alignment training datasets, often collected from naturalistic conversations covering a wide range of domains <cite class="ltx_cite ltx_citemacro_cite">Taori et al. (<a href="#bib.bib42" title="" class="ltx_ref">2023</a>); Geng et al. (<a href="#bib.bib13" title="" class="ltx_ref">2023</a>); Chiang et al. (<a href="#bib.bib5" title="" class="ltx_ref">2023</a>); Christiano et al. (<a href="#bib.bib6" title="" class="ltx_ref">2017</a>); Ouyang et al. (<a href="#bib.bib36" title="" class="ltx_ref">2022</a>)</cite>. For instance, Alpaca <cite class="ltx_cite ltx_citemacro_cite">Taori et al. (<a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite> and Dolly-15K <cite class="ltx_cite ltx_citemacro_cite">Conover et al. (<a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite> cover tasks from creative writing to context-dependent QA. Particularly, these tasks may have distinct objectives, and, when mixed, may induce potential conflicts of interest during alignment.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.21417/assets/figures/figure_1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="538" height="560" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Faithfulness scores on context-dependent tasks (QA and summarization) decrease when we fine-tune grounded LLaMA-7B checkpoint with instruction following datasets (orange), and instruction following scores (assessed by GPT-4) decrease when we fine-tune Vicuna-7B with context-dependent tasks (blue).
Our method, <span id="S1.F1.2.1" class="ltx_text ltx_font_smallcaps">ReSet</span> surpasses the vanilla MTL with data mixing, approaching the North Star (upper right corner).
</figcaption>
</figure>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this context, we examine the interaction between instruction following (i.e., how well does the LM follow open-ended instructions) and faithfulness (i.e., is the LM’s response grounded in the context) during alignment training.
Specifically, we study how instruction following and faithfulness scores change when adapting LMs to two types of datasets: 1) <em id="S1.p3.1.1" class="ltx_emph ltx_font_italic">instruction-tuning datasets</em> that are commonly used to train chat-models such as ChatGPT <cite class="ltx_cite ltx_citemacro_cite">OpenAI (<a href="#bib.bib35" title="" class="ltx_ref">2022</a>)</cite> and Llama-2-Chat <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite>; and 2) <em id="S1.p3.1.2" class="ltx_emph ltx_font_italic">context-dependent datasets</em> that require grounding to a provided <em id="S1.p3.1.3" class="ltx_emph ltx_font_italic">context</em>, and are commonly used to train Retrieval-Augmented Generation (RAG) models <cite class="ltx_cite ltx_citemacro_cite">Lewis et al. (<a href="#bib.bib27" title="" class="ltx_ref">2020</a>)</cite> such as Atlas <cite class="ltx_cite ltx_citemacro_cite">Izacard et al. (<a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite> and DSPy <cite class="ltx_cite ltx_citemacro_cite">Khattab et al. (<a href="#bib.bib22" title="" class="ltx_ref">2024</a>)</cite>.
We observe a clear trade-off between the two scores, as shown in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>: a fine-tuned LM with a competitive faithfulness score becomes much less faithful when separately fine-tuned on instruction following datasets. Conversely, an instruction-tuned LM becomes worse at following instructions when fine-tuned on context-dependent datasets. Our findings suggest fine-tuning an LM with either instruction following or context-dependent datasets exclusively may impair its original ability in the other aspect.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">One natural mitigation strategy is to use multi-task learning (MTL) by mixing datasets, which we find as a strong yet sub-optimal baseline.
To achieve a more synergic outcome, we propose a simple yet effective method <span id="S1.p4.1.1" class="ltx_text ltx_font_bold">Re</span>jection Sampling for Continued <span id="S1.p4.1.2" class="ltx_text ltx_font_bold">Se</span>lf-instruction <span id="S1.p4.1.3" class="ltx_text ltx_font_bold">T</span>uning (<span id="S1.p4.1.4" class="ltx_text ltx_font_bold ltx_font_smallcaps">ReSet</span>).
Inspired by recent works in self-instruct <cite class="ltx_cite ltx_citemacro_cite">Zelikman et al. (<a href="#bib.bib48" title="" class="ltx_ref">2022</a>); Wang et al. (<a href="#bib.bib45" title="" class="ltx_ref">2023b</a>)</cite>, <span id="S1.p4.1.5" class="ltx_text ltx_font_smallcaps">ReSet</span> leverages the LM to sample generations for instruction following and task-specific datasets, which is different from vanilla MTL. Generations are then rated by external judges for instruction following and faithfulness scores, where top-rated generations are collected and used to further fine-tune the LM.
Our experiments show that when trained with <span id="S1.p4.1.6" class="ltx_text ltx_font_smallcaps">ReSet</span> using only a single iteration and 8,000 additional fine-tuning examples, LMs see substantial gains in faithfulness scores (up to +18.8%) compared to the MTL baseline, while maintaining their instruction-following scores.
Furthermore, we find that less is more: training with <span id="S1.p4.1.7" class="ltx_text ltx_font_smallcaps">ReSet</span> on higher quality yet three-fold less data yields up to 31.3% improvements on faithfulness among datasets compared to MTL.
Our analyses shed lights on finding and mitigating objective discrepancies in alignment training where datasets encompass different or even conflicting goals.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>We will release our code and evaluation data at<a target="_blank" href="https://github.com/frankaging/dancing-in-chains" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/frankaging/dancing-in-chains</a>.</span></span></span></p>
</div>
<figure id="S1.T1" class="ltx_table">
<div id="S1.T1.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:287.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(67.4pt,-44.7pt) scale(1.45117514530916,1.45117514530916) ;">
<table id="S1.T1.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S1.T1.1.1.1" class="ltx_tr">
<td id="S1.T1.1.1.1.1" class="ltx_td ltx_align_left ltx_border_tt"><span id="S1.T1.1.1.1.1.1" class="ltx_text ltx_font_bold">Dataset</span></td>
<td id="S1.T1.1.1.1.2" class="ltx_td ltx_align_right ltx_border_tt"><span id="S1.T1.1.1.1.2.1" class="ltx_text ltx_font_bold">Train</span></td>
<td id="S1.T1.1.1.1.3" class="ltx_td ltx_align_right ltx_border_tt"><span id="S1.T1.1.1.1.3.1" class="ltx_text ltx_font_bold">Dev</span></td>
<td id="S1.T1.1.1.1.4" class="ltx_td ltx_align_right ltx_border_tt"><span id="S1.T1.1.1.1.4.1" class="ltx_text ltx_font_bold">Test</span></td>
<td id="S1.T1.1.1.1.5" class="ltx_td ltx_align_right ltx_border_tt"><span id="S1.T1.1.1.1.5.1" class="ltx_text ltx_font_bold">Avg. Length</span></td>
</tr>
<tr id="S1.T1.1.1.2" class="ltx_tr">
<td id="S1.T1.1.1.2.1" class="ltx_td ltx_align_left ltx_border_t">Instruction Datasets</td>
<td id="S1.T1.1.1.2.2" class="ltx_td ltx_align_right ltx_border_t">257,307</td>
<td id="S1.T1.1.1.2.3" class="ltx_td ltx_align_right ltx_border_t">2,500</td>
<td id="S1.T1.1.1.2.4" class="ltx_td ltx_align_right ltx_border_t">-</td>
<td id="S1.T1.1.1.2.5" class="ltx_td ltx_align_right ltx_border_t">305</td>
</tr>
<tr id="S1.T1.1.1.3" class="ltx_tr">
<td id="S1.T1.1.1.3.1" class="ltx_td ltx_align_left"><span id="S1.T1.1.1.3.1.1" class="ltx_text ltx_font_italic">Alpaca-15K</span></td>
<td id="S1.T1.1.1.3.2" class="ltx_td ltx_align_right">-</td>
<td id="S1.T1.1.1.3.3" class="ltx_td ltx_align_right">-</td>
<td id="S1.T1.1.1.3.4" class="ltx_td ltx_align_right">31,323</td>
<td id="S1.T1.1.1.3.5" class="ltx_td ltx_align_right">79</td>
</tr>
<tr id="S1.T1.1.1.4" class="ltx_tr">
<td id="S1.T1.1.1.4.1" class="ltx_td ltx_align_left"><span id="S1.T1.1.1.4.1.1" class="ltx_text ltx_font_italic">Vicuna-eval</span></td>
<td id="S1.T1.1.1.4.2" class="ltx_td ltx_align_right">-</td>
<td id="S1.T1.1.1.4.3" class="ltx_td ltx_align_right">-</td>
<td id="S1.T1.1.1.4.4" class="ltx_td ltx_align_right">80</td>
<td id="S1.T1.1.1.4.5" class="ltx_td ltx_align_right">367</td>
</tr>
<tr id="S1.T1.1.1.5" class="ltx_tr">
<td id="S1.T1.1.1.5.1" class="ltx_td ltx_align_left"><span id="S1.T1.1.1.5.1.1" class="ltx_text ltx_font_italic">Koala-eval</span></td>
<td id="S1.T1.1.1.5.2" class="ltx_td ltx_align_right">-</td>
<td id="S1.T1.1.1.5.3" class="ltx_td ltx_align_right">-</td>
<td id="S1.T1.1.1.5.4" class="ltx_td ltx_align_right">180</td>
<td id="S1.T1.1.1.5.5" class="ltx_td ltx_align_right">444</td>
</tr>
<tr id="S1.T1.1.1.6" class="ltx_tr">
<td id="S1.T1.1.1.6.1" class="ltx_td ltx_align_left ltx_border_t">NQ</td>
<td id="S1.T1.1.1.6.2" class="ltx_td ltx_align_right ltx_border_t">69,639</td>
<td id="S1.T1.1.1.6.3" class="ltx_td ltx_align_right ltx_border_t">8,757</td>
<td id="S1.T1.1.1.6.4" class="ltx_td ltx_align_right ltx_border_t">13,368</td>
<td id="S1.T1.1.1.6.5" class="ltx_td ltx_align_right ltx_border_t">6</td>
</tr>
<tr id="S1.T1.1.1.7" class="ltx_tr">
<td id="S1.T1.1.1.7.1" class="ltx_td ltx_align_left">CNN/Daily Mail</td>
<td id="S1.T1.1.1.7.2" class="ltx_td ltx_align_right">287,113</td>
<td id="S1.T1.1.1.7.3" class="ltx_td ltx_align_right">13,368</td>
<td id="S1.T1.1.1.7.4" class="ltx_td ltx_align_right">11,490</td>
<td id="S1.T1.1.1.7.5" class="ltx_td ltx_align_right">76</td>
</tr>
<tr id="S1.T1.1.1.8" class="ltx_tr">
<td id="S1.T1.1.1.8.1" class="ltx_td ltx_align_left">MS MARCO</td>
<td id="S1.T1.1.1.8.2" class="ltx_td ltx_align_right">153,725</td>
<td id="S1.T1.1.1.8.3" class="ltx_td ltx_align_right">2,500</td>
<td id="S1.T1.1.1.8.4" class="ltx_td ltx_align_right">12,466</td>
<td id="S1.T1.1.1.8.5" class="ltx_td ltx_align_right">23</td>
</tr>
<tr id="S1.T1.1.1.9" class="ltx_tr">
<td id="S1.T1.1.1.9.1" class="ltx_td ltx_align_left"><span id="S1.T1.1.1.9.1.1" class="ltx_text ltx_font_italic">BioASQ</span></td>
<td id="S1.T1.1.1.9.2" class="ltx_td ltx_align_right">-</td>
<td id="S1.T1.1.1.9.3" class="ltx_td ltx_align_right">-</td>
<td id="S1.T1.1.1.9.4" class="ltx_td ltx_align_right">1,956</td>
<td id="S1.T1.1.1.9.5" class="ltx_td ltx_align_right">9</td>
</tr>
<tr id="S1.T1.1.1.10" class="ltx_tr">
<td id="S1.T1.1.1.10.1" class="ltx_td ltx_align_left"><span id="S1.T1.1.1.10.1.1" class="ltx_text ltx_font_italic">SearchQA</span></td>
<td id="S1.T1.1.1.10.2" class="ltx_td ltx_align_right">-</td>
<td id="S1.T1.1.1.10.3" class="ltx_td ltx_align_right">-</td>
<td id="S1.T1.1.1.10.4" class="ltx_td ltx_align_right">31,760</td>
<td id="S1.T1.1.1.10.5" class="ltx_td ltx_align_right">4</td>
</tr>
<tr id="S1.T1.1.1.11" class="ltx_tr">
<td id="S1.T1.1.1.11.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S1.T1.1.1.11.1.1" class="ltx_text ltx_font_italic">WikiSum</span></td>
<td id="S1.T1.1.1.11.2" class="ltx_td ltx_align_right ltx_border_bb">-</td>
<td id="S1.T1.1.1.11.3" class="ltx_td ltx_align_right ltx_border_bb">-</td>
<td id="S1.T1.1.1.11.4" class="ltx_td ltx_align_right ltx_border_bb">2,000</td>
<td id="S1.T1.1.1.11.5" class="ltx_td ltx_align_right ltx_border_bb">140</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Data statistics of two-stage fine-tuning experiments.
<span id="S1.T1.3.1" class="ltx_text ltx_font_italic">Italicized</span> datasets are held-out unseen evaluation datasets, while the rest are used for training. “Instruction Datasets” here consist of publicly avaliable datasets such as Dolly-15K <cite class="ltx_cite ltx_citemacro_cite">Conover et al. (<a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite>, ShareGPT, Self-Instruct <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib45" title="" class="ltx_ref">2023b</a>)</cite> and OASST-1 <cite class="ltx_cite ltx_citemacro_cite">Köpf et al. (<a href="#bib.bib23" title="" class="ltx_ref">2023</a>)</cite>. Average length is the averaged response token length, which is calculated with training or testing sets for each dataset with the LLaMA-7B tokenizer.
</figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Background</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">To recap, our goal is to study the interplay of training on instruction following datasets and on datasets that require grounding to a specific context.
To do this, we first construct two separate groups of datasets, each resembling a setting above.
We then adopt a two-stage fine-tuning paradigm that allows us to closely study the effect of each type of training.
We introduce the setup in this section.</p>
</div>
<figure id="S2.T2" class="ltx_table">
<p id="S2.T2.6" class="ltx_p ltx_align_center">
<span id="S2.T2.6.6.6" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:390.3pt;">
<span id="S2.T2.6.6.6.7" class="ltx_p"><span id="S2.T2.6.6.6.7.1" class="ltx_text ltx_font_typewriter">Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. 
<br class="ltx_break"></span></span>
<span id="S2.T2.6.6.6.6" class="ltx_p"><span id="S2.T2.6.6.6.6.6" class="ltx_text ltx_font_typewriter">### Instruction: 
<br class="ltx_break"><math id="S2.T2.1.1.1.1.1.m1.1" class="ltx_Math" alttext="[" display="inline"><semantics id="S2.T2.1.1.1.1.1.m1.1a"><mo mathcolor="#808080" stretchy="false" id="S2.T2.1.1.1.1.1.m1.1.1" xref="S2.T2.1.1.1.1.1.m1.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="S2.T2.1.1.1.1.1.m1.1b"><ci id="S2.T2.1.1.1.1.1.m1.1.1.cmml" xref="S2.T2.1.1.1.1.1.m1.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.1.1.1.1.1.m1.1c">[</annotation></semantics></math><span id="S2.T2.2.2.2.2.2.1" class="ltx_text" style="color:#808080;">Task-specific Instruction Abbreviated<math id="S2.T2.2.2.2.2.2.1.m1.1" class="ltx_Math" alttext="]" display="inline"><semantics id="S2.T2.2.2.2.2.2.1.m1.1a"><mo mathcolor="#808080" stretchy="false" id="S2.T2.2.2.2.2.2.1.m1.1.1" xref="S2.T2.2.2.2.2.2.1.m1.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="S2.T2.2.2.2.2.2.1.m1.1b"><ci id="S2.T2.2.2.2.2.2.1.m1.1.1.cmml" xref="S2.T2.2.2.2.2.2.1.m1.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.2.2.2.2.2.1.m1.1c">]</annotation></semantics></math></span> 
<br class="ltx_break">
<br class="ltx_break">### Input: 
<br class="ltx_break"><math id="S2.T2.3.3.3.3.3.m2.1" class="ltx_Math" alttext="[" display="inline"><semantics id="S2.T2.3.3.3.3.3.m2.1a"><mo mathcolor="#808080" stretchy="false" id="S2.T2.3.3.3.3.3.m2.1.1" xref="S2.T2.3.3.3.3.3.m2.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="S2.T2.3.3.3.3.3.m2.1b"><ci id="S2.T2.3.3.3.3.3.m2.1.1.cmml" xref="S2.T2.3.3.3.3.3.m2.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.3.3.3.3.3.m2.1c">[</annotation></semantics></math><span id="S2.T2.4.4.4.4.4.2" class="ltx_text" style="color:#808080;">Passage: Abbreviated<math id="S2.T2.4.4.4.4.4.2.m1.1" class="ltx_Math" alttext="]" display="inline"><semantics id="S2.T2.4.4.4.4.4.2.m1.1a"><mo mathcolor="#808080" stretchy="false" id="S2.T2.4.4.4.4.4.2.m1.1.1" xref="S2.T2.4.4.4.4.4.2.m1.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="S2.T2.4.4.4.4.4.2.m1.1b"><ci id="S2.T2.4.4.4.4.4.2.m1.1.1.cmml" xref="S2.T2.4.4.4.4.4.2.m1.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.4.4.4.4.4.2.m1.1c">]</annotation></semantics></math></span> 
<br class="ltx_break"><math id="S2.T2.5.5.5.5.5.m3.1" class="ltx_Math" alttext="[" display="inline"><semantics id="S2.T2.5.5.5.5.5.m3.1a"><mo mathcolor="#808080" stretchy="false" id="S2.T2.5.5.5.5.5.m3.1.1" xref="S2.T2.5.5.5.5.5.m3.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="S2.T2.5.5.5.5.5.m3.1b"><ci id="S2.T2.5.5.5.5.5.m3.1.1.cmml" xref="S2.T2.5.5.5.5.5.m3.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.5.5.5.5.5.m3.1c">[</annotation></semantics></math><span id="S2.T2.6.6.6.6.6.3" class="ltx_text" style="color:#808080;">... ...<math id="S2.T2.6.6.6.6.6.3.m1.1" class="ltx_Math" alttext="]" display="inline"><semantics id="S2.T2.6.6.6.6.6.3.m1.1a"><mo mathcolor="#808080" stretchy="false" id="S2.T2.6.6.6.6.6.3.m1.1.1" xref="S2.T2.6.6.6.6.6.3.m1.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="S2.T2.6.6.6.6.6.3.m1.1b"><ci id="S2.T2.6.6.6.6.6.3.m1.1.1.cmml" xref="S2.T2.6.6.6.6.6.3.m1.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.6.6.6.6.6.3.m1.1c">]</annotation></semantics></math></span> 
<br class="ltx_break">
<br class="ltx_break">###Response: 
<br class="ltx_break"><span id="S2.T2.6.6.6.6.6.4" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">Model Generated Answer Goes Here</span></span></span>
</span>
</p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>The prompt template we used for training and evaluation for LLaMA-7B. The instruction field contains task-specific instruction, the input field contains contexts if applicable, and the response field is followed by model’s generation. We use a different template for Vicuna-7B since it uses a different template during instruct-tuning phase. Actual input examples for each dataset and model are included in Appendix <a href="#A1.SS6" title="A.6 Examples of Actual Instructions ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.6</span></a>.</figcaption>
</figure>
<figure id="S2.T3" class="ltx_table">
<div id="S2.T3.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:780.5pt;height:159.5pt;vertical-align:-1.6pt;"><span class="ltx_transformed_inner" style="transform:translate(149.0pt,-30.1pt) scale(1.61739296285203,1.61739296285203) ;">
<table id="S2.T3.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S2.T3.1.1.1" class="ltx_tr">
<td id="S2.T3.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S2.T3.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.1.1.1.1.1.1" class="ltx_p" style="width:85.4pt;"><span id="S2.T3.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Dataset Type</span></span>
</span>
</td>
<td id="S2.T3.1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S2.T3.1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.1.1.1.2.1.1" class="ltx_p" style="width:369.9pt;"><span id="S2.T3.1.1.1.2.1.1.1" class="ltx_text ltx_font_bold">Instruction</span></span>
</span>
</td>
</tr>
<tr id="S2.T3.1.1.2" class="ltx_tr">
<td id="S2.T3.1.1.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S2.T3.1.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.1.1.2.1.1.1" class="ltx_p" style="width:85.4pt;"><span id="S2.T3.1.1.2.1.1.1.1" class="ltx_text ltx_font_bold">Extract QA</span></span>
</span>
</td>
<td id="S2.T3.1.1.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S2.T3.1.1.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.1.1.2.2.1.1" class="ltx_p" style="width:369.9pt;">Answer to the question by <span id="S2.T3.1.1.2.2.1.1.1" class="ltx_text ltx_font_bold">extracting a specific text span</span> from the given passages. Do not include new information beyond the given passages.</span>
</span>
</td>
</tr>
<tr id="S2.T3.1.1.3" class="ltx_tr">
<td id="S2.T3.1.1.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S2.T3.1.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.1.1.3.1.1.1" class="ltx_p" style="width:85.4pt;"><span id="S2.T3.1.1.3.1.1.1.1" class="ltx_text ltx_font_bold">Abstractive QA</span></span>
</span>
</td>
<td id="S2.T3.1.1.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S2.T3.1.1.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.1.1.3.2.1.1" class="ltx_p" style="width:369.9pt;">Answer the question with well-formed sentences. Paraphrase the context in the passages if necessary. <span id="S2.T3.1.1.3.2.1.1.1" class="ltx_text ltx_font_bold">Do not include new information</span> beyond the given passages.</span>
</span>
</td>
</tr>
<tr id="S2.T3.1.1.4" class="ltx_tr">
<td id="S2.T3.1.1.4.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t">
<span id="S2.T3.1.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.1.1.4.1.1.1" class="ltx_p" style="width:85.4pt;"><span id="S2.T3.1.1.4.1.1.1.1" class="ltx_text ltx_font_bold">Summarization</span></span>
</span>
</td>
<td id="S2.T3.1.1.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t">
<span id="S2.T3.1.1.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.1.1.4.2.1.1" class="ltx_p" style="width:369.9pt;">Summarize the text in a few sentences. Using original phrases or paraphrasing them if necessary. <span id="S2.T3.1.1.4.2.1.1.1" class="ltx_text ltx_font_bold">Do not include new information</span> beyond the given passages.</span>
</span>
</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Task specific instructions. We bold texts that indicate that our prompts are designed to be objective-aligned with our instruction following training data (i.e., fine-tuning our model on instruction following datasets should with keeping it to be faithful as well).
</figcaption>
</figure>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Datasets</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">We outline different datasets used for instruction following and context-dependent fine-tuning. Table <a href="#S1.T1" title="Table 1 ‣ 1 Introduction ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows data statistics. For each dataset we carefully design the instructions to well-align the input and output (see Section <a href="#S2.SS3" title="2.3 Instruction Template ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a>).</p>
</div>
<section id="S2.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">2.1.1 </span>Instruction Following Datasets</h4>

<div id="S2.SS1.SSS1.p1" class="ltx_para">
<p id="S2.SS1.SSS1.p1.1" class="ltx_p">We curated an instruction following training dataset by compiling unique examples from publicly available datasets such as Dolly-15K <cite class="ltx_cite ltx_citemacro_cite">Conover et al. (<a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite>,
ShareGPT<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span><a target="_blank" href="https://huggingface.co/datasets/LLMs/Alpaca-ShareGPT" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/LLMs/Alpaca-ShareGPT</a></span></span></span>, Self-Instruct <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib45" title="" class="ltx_ref">2023b</a>)</cite>, and OASST-1 <cite class="ltx_cite ltx_citemacro_cite">Köpf et al. (<a href="#bib.bib23" title="" class="ltx_ref">2023</a>)</cite>. For evaluation, we gather unique examples from Alpaca-15K <cite class="ltx_cite ltx_citemacro_cite">Taori et al. (<a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite>, Vicuna-eval <cite class="ltx_cite ltx_citemacro_cite">Chiang et al. (<a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite>, and Koala-eval <cite class="ltx_cite ltx_citemacro_cite">Geng et al. (<a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<section id="S2.SS1.SSS1.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Data Pre-processing</h5>

<div id="S2.SS1.SSS1.Px1.p1" class="ltx_para">
<p id="S2.SS1.SSS1.Px1.p1.1" class="ltx_p">We exclude examples that originally come with context (e.g., examples labeled as summarization type in Dolly-15K are filtered out) to prevent overlaps with our context-dependent datasets. We retain only unseen examples in our evaluation sets.
For OASST-1, we only include examples with an average human rating higher than 0.5 (i.e., the higher the rating is, the higher the quality is) and that are rated by at least two annotators. To refine ShareGPT, we include only examples with responses that are longer than 10 words, split by whitespace. For other instruction following training datasets, we exclude examples with empty instructions or responses.</p>
</div>
</section>
</section>
<section id="S2.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">2.1.2 </span>Context-Dependent Datasets</h4>

<div id="S2.SS1.SSS2.p1" class="ltx_para">
<p id="S2.SS1.SSS2.p1.1" class="ltx_p">We aim to evaluate the model’s generation for faithfulness to the given input context. We select a range of context-dependent datasets from three task domains: (1) extractive QA including NQ <cite class="ltx_cite ltx_citemacro_cite">Kwiatkowski et al. (<a href="#bib.bib24" title="" class="ltx_ref">2019</a>)</cite>, BioASQ and SearchQA, taken from the RobustQA benchmark <cite class="ltx_cite ltx_citemacro_cite">Han et al. (<a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>; (2) abstractive QA with MS MARCO <cite class="ltx_cite ltx_citemacro_cite">Bajaj et al. (<a href="#bib.bib1" title="" class="ltx_ref">2016</a>)</cite> where the answers are well-formed sentences grounded in context; and (3) abstractive summarization including CNN DailyMail <cite class="ltx_cite ltx_citemacro_cite">Hermann et al. (<a href="#bib.bib16" title="" class="ltx_ref">2015</a>); Nallapati et al. (<a href="#bib.bib34" title="" class="ltx_ref">2016</a>)</cite> and WikiSum <cite class="ltx_cite ltx_citemacro_cite">Liu et al. (<a href="#bib.bib30" title="" class="ltx_ref">2018</a>)</cite>. BioASQ, SearchQA and WikiSum are hold out for evaluation and the rest are for training. One crucial advantage of using these context-dependent datasets is that they provide us with a reliable way of measuring faithfulness, in terms of how well the response is grounded in the given context.</p>
</div>
<section id="S2.SS1.SSS2.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Data Pre-processing</h5>

<div id="S2.SS1.SSS2.Px1.p1" class="ltx_para">
<p id="S2.SS1.SSS2.Px1.p1.1" class="ltx_p">For QA datasets, we include five retrieved passages maximally as the context,<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>Details about the retrieving process can be found in the original papers of RobustQA <cite class="ltx_cite ltx_citemacro_cite">Han et al. (<a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite> and MS MARCO <cite class="ltx_cite ltx_citemacro_cite">Bajaj et al. (<a href="#bib.bib1" title="" class="ltx_ref">2016</a>)</cite>.</span></span></span> where the gold answer is at least mentioned in one of the passages. For MS MARCO, we only include examples where there exist at least one well-formed answers. In cases involving multiple retrieved passages, we concatenate all passages with line breaks inserted between them.</p>
</div>
</section>
</section>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Evaluation Metrics</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">In the context of our datasets, we evaluate our models with three metrics: instruction following score, faithfulness score and task performance score. The standard methods for measuring instruction following and faithfulness of language models are subject to ongoing debate. In this work, we employ widely adopted approaches for these measurements and compile a set of metrics for more stable evaluations.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>Throughout the paper, we sample a subset of the full evaluation data which include 6,000 examples (1,000 examples from each context-dependent evaluation set), and sample 300 examples (100 examples from each instruction following evaluation set) due to limited compute resources.</span></span></span> To present our findings, we report macro-averaged results across all test datasets.</p>
</div>
<section id="S2.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Instruction Following Score</h5>

<div id="S2.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS2.SSS0.Px1.p1.1" class="ltx_p">We adopt the commonly used evaluation paradigm proposed by LLM-as-a-Judge <cite class="ltx_cite ltx_citemacro_cite">Zheng et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>, and zero-shot prompt GPT-4 to provide a rating followed by a short explanation (i.e., named as LLM-as-a-Judge (R) in the paper).<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>We use <span id="footnote5.1" class="ltx_text ltx_font_typewriter">gpt-4-0613</span>.</span></span></span> For the GPT-4 evaluator, we set the temperature to 0 for stability with a maximum generation length of 512. We check instruction following scores only for instruction following evaluation datasets. See Appendix <a href="#A1.SS1" title="A.1 Evaluation Metric: Instruction Following Score ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.1</span></a> for our actual evaluation prompt.</p>
</div>
</section>
<section id="S2.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Faithfulness Score</h5>

<div id="S2.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS2.SSS0.Px2.p1.1" class="ltx_p">For extractive QA datasets, we utilize the span coverage score as our metric (i.e., whether the predicted answer is a span within the context). We apply standard normalization to both the predicted answer and the context (see Appendix <a href="#A1.SS2" title="A.2 Evaluation Metric: Faithfulness Score ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.2</span></a> for details). A score of 1.0 is assigned if the span is covered, and 0.0 otherwise. Additionally, we include unigram and bigram coverage for selected datasets to further refine our faithfulness evaluation in the Appendix (see Figure <a href="#A1.F9" title="Figure 9 ‣ A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> and Figure <a href="#A1.F10" title="Figure 10 ‣ A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>). For abstractive QA and summarization datasets, we employ SummaC-ZeroShot (SummaC-ZS; <cite class="ltx_cite ltx_citemacro_citet">Laban et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite>) to assess whether the provided context (with the question concatenated as a prefix for QA datasets) entails the model-generated answer. Specifically, we segment both the context and the answer into sentences and then use a pretrained NLI model to compute an entailment score between pairs. These NLI scores are aggregated across pairs using max-pooling.</p>
</div>
</section>
<section id="S2.SS2.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Task Performance</h5>

<div id="S2.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS2.SSS0.Px3.p1.1" class="ltx_p">Task performance complements the aforementioned metrics, as task performance is generally expected to decline when models are fine-tuned across domains. We utilize Exact Match (EM)
for extractive QA datasets and the ROUGE-L score <cite class="ltx_cite ltx_citemacro_cite">Lin (<a href="#bib.bib28" title="" class="ltx_ref">2004</a>)</cite>
for abstractive QA and summarization datasets.</p>
</div>
</section>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Instruction Template</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">Our datasets follow the same instruction-tuning template format as used in the Alpaca setting <cite class="ltx_cite ltx_citemacro_cite">Taori et al. (<a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite>. This template includes a header that outlines general guidelines, followed by task-specific instructions, as illustrated in Table <a href="#S2.T2" title="Table 2 ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. For QA tasks, the question is presented after the retrieved passages.</p>
</div>
<section id="S2.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Objective-Aligned Instructions</h5>

<div id="S2.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS3.SSS0.Px1.p1.1" class="ltx_p">We design our task-specific instructions to minimize <em id="S2.SS3.SSS0.Px1.p1.1.1" class="ltx_emph ltx_font_italic">objective-conflict</em> among datasets. For example, in prompting our model for a context-dependent extractive QA task, we explicitly instruct the model to “extract a specific text span from the given passages”. This template helps models to reduce hallucination when fine-tuned with instruction following datasets: as models getting better at understanding human instructions, they also get better at understanding how to <em id="S2.SS3.SSS0.Px1.p1.1.2" class="ltx_emph ltx_font_italic">extracting a span</em> which ensures the answer to be faithful. The instructions for all datasets are depicted in Table <a href="#S2.T3" title="Table 3 ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Our task-specific templates ensure there is no objective conflict when we fine-tune our models with mixed of datasets. We use the same template for training and evaluation for each task.</p>
</div>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2407.21417/assets/x1.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="201" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Two-stage fine-tuning with LLaMA-7B.
</figcaption>
</figure>
<figure id="S2.F3" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_ctx_to_alignment.png" id="S2.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="88" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Macro-averaged faithfulness, instruction following, and task performance scores on corresponding evaluation datasets before and after fine-tuning with instruction following datasets.</figcaption>
</figure>
<figure id="S2.F4" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_seq_length_abstract_stage_2.png" id="S2.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="82" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Average generation token length throughout the instruction following training stage. The first checkpoint is the best checkpoint from the context-dependent training stage. The middle checkpoint is with the lowest evaluation loss during the second stage.</figcaption>
</figure>
<figure id="S2.F5" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_abstract_length_split_stage_2.png" id="S2.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="538" height="294" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Faithfulness scores for abstraction QA and summarization datasets categorized by whether the generation length is strictly shorter or much longer (<math id="S2.F5.2.m1.1" class="ltx_Math" alttext="&gt;" display="inline"><semantics id="S2.F5.2.m1.1b"><mo id="S2.F5.2.m1.1.1" xref="S2.F5.2.m1.1.1.cmml">&gt;</mo><annotation-xml encoding="MathML-Content" id="S2.F5.2.m1.1c"><gt id="S2.F5.2.m1.1.1.cmml" xref="S2.F5.2.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S2.F5.2.m1.1d">&gt;</annotation></semantics></math> 100 tokens) than the golden answer.</figcaption>
</figure>
</section>
</section>
<section id="S2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.4 </span>Two-stage Fine-tuning Paradigm</h3>

<div id="S2.SS4.p1" class="ltx_para">
<p id="S2.SS4.p1.2" class="ltx_p">To understand the trade-off between instruction following and faithfulness when training LMs with both objectives, we formulate a two-stage fine-tuning paradigm to answer our research questions (as illustrated in Figure <a href="#S2.F2" title="Figure 2 ‣ Objective-Aligned Instructions ‣ 2.3 Instruction Template ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>).<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>Our approach is akin to the data inoculation paradigm proposed by <cite class="ltx_cite ltx_citemacro_citet">Liu et al. (<a href="#bib.bib29" title="" class="ltx_ref">2019a</a>)</cite>, albeit with significantly larger models and datasets.</span></span></span> For our first pipeline, we initially fine-tune our LM with context-dependent datasets that require grounding. We then take the best checkpoint from the initial stage to further fine-tune it on instruction following datasets (CD<math id="S2.SS4.p1.1.m1.1" class="ltx_Math" alttext="\to" display="inline"><semantics id="S2.SS4.p1.1.m1.1a"><mo stretchy="false" id="S2.SS4.p1.1.m1.1.1" xref="S2.SS4.p1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.1.m1.1b"><ci id="S2.SS4.p1.1.m1.1.1.cmml" xref="S2.SS4.p1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.1.m1.1c">\to</annotation></semantics></math>IF). Conversely, in our second pipeline, we fine-tune instruct-tuned LM (e.g., Vicuna-7B) with context-dependent datasets (IF<math id="S2.SS4.p1.2.m2.1" class="ltx_Math" alttext="\to" display="inline"><semantics id="S2.SS4.p1.2.m2.1a"><mo stretchy="false" id="S2.SS4.p1.2.m2.1.1" xref="S2.SS4.p1.2.m2.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.2.m2.1b"><ci id="S2.SS4.p1.2.m2.1.1.cmml" xref="S2.SS4.p1.2.m2.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.2.m2.1c">\to</annotation></semantics></math>CD). For both pipelines, we measure instruction following and faithfulness scores before and after training, to gauge the impact of the second-stage training on both capabilities. We follow this paradigm to find evidence of the trade-off in Section <a href="#S3" title="3 Does Fine-tuning with Instruction Following Data Hurt Faithfulness? ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> and Section <a href="#S4" title="4 Does Context-Dependent Fine-tuning Hurt Instruction Following? ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.</p>
</div>
<section id="S2.SS4.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Models</h5>

<div id="S2.SS4.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS4.SSS0.Px1.p1.2" class="ltx_p">We use two models in our two-stage fine-tuning paradigm. We use a base LM LLaMA-7B <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite>, one of the most widely used open-source LM, for the CD<math id="S2.SS4.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\to" display="inline"><semantics id="S2.SS4.SSS0.Px1.p1.1.m1.1a"><mo stretchy="false" id="S2.SS4.SSS0.Px1.p1.1.m1.1.1" xref="S2.SS4.SSS0.Px1.p1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS4.SSS0.Px1.p1.1.m1.1b"><ci id="S2.SS4.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S2.SS4.SSS0.Px1.p1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.SSS0.Px1.p1.1.m1.1c">\to</annotation></semantics></math>IF pipeline. For our IF<math id="S2.SS4.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="\to" display="inline"><semantics id="S2.SS4.SSS0.Px1.p1.2.m2.1a"><mo stretchy="false" id="S2.SS4.SSS0.Px1.p1.2.m2.1.1" xref="S2.SS4.SSS0.Px1.p1.2.m2.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS4.SSS0.Px1.p1.2.m2.1b"><ci id="S2.SS4.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S2.SS4.SSS0.Px1.p1.2.m2.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.SSS0.Px1.p1.2.m2.1c">\to</annotation></semantics></math>CD pipeline, we use Vicuna-7B off-the-shelf as our instruct-tuned LLaMA-7B without retraining one from scratch. Vicuna-7B is one of the most competitive open-source chat-model, and is a fine-tuned LLaMA-7B on conversational data from ShareGPT <cite class="ltx_cite ltx_citemacro_citep">(Zheng et al., <a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>.
Our hypothesis and paradigm are transferable to other base LMs at different scales, although we pick these two models as they are among the first open-sourced LMs during the time frame of this project. Other experimental setup details are included in Appendix <a href="#A1.SS3" title="A.3 Experimental Setup ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.3</span></a>.</p>
</div>
<figure id="S2.F6" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_alignment_to_ctx.png" id="S2.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="88" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Macro-averaged faithfulness, instruction following, and task performance scores on corresponding evaluation datasets before and after fine-tuning with context-dependent datasets.</figcaption>
</figure>
<figure id="S2.F7" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_alignment_length_split_stage_2.png" id="S2.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="147" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>Instruction following scores for 1,000 randomly selected examples from Alpaca-15K (left), and for filtered examples with similar generation lengths (maximally 10 tokens longer) before and after fine-tuning with context-dependent datasets (right).</figcaption>
</figure>
</section>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Does Fine-tuning with Instruction Following Data Hurt Faithfulness?</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">To answer this question, we follow the first pipeline outlined in Figure <a href="#S2.F2" title="Figure 2 ‣ Objective-Aligned Instructions ‣ 2.3 Instruction Template ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, where we take our LLaMA-7B that is fine-tuned on context-dependent datasets and further fine-tune it with instruction following datasets. Our results are shown in Figure <a href="#S2.F3" title="Figure 3 ‣ Objective-Aligned Instructions ‣ 2.3 Instruction Template ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. First of all, instruction following scores increase drastically as expected, from 0.30 to 0.74. Meanwhile, faithfulness scores on abstractive datasets drop 33.0% from 0.82 to 0.55, while the task performance is largely maintained (from 0.34 to 0.32). For extractive datasets, both faithfulness scores and task performance see a relatively small drop. To provide a fine-grained view of these datasets, we provide additional analysis of per-task and per-metric changes of faithfulness score as well as task performance in Appendix <a href="#A1.SS4" title="A.4 Additional Analysis on Instruction Following Fine-tuning ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.4</span></a> and Appendix <a href="#A1.SS5" title="A.5 Quandary of Mixed Training on Abstractive and Extractive QA and Summarization Datasets ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.5</span></a>.</p>
</div>
<div id="S3.p2" class="ltx_para">
<p id="S3.p2.1" class="ltx_p">One potential confounding factor for faithfulness scores dropping on abstractive tasks is the length of the generated response.
The model could simply have generated longer responses (Figure <a href="#S2.F4" title="Figure 4 ‣ Objective-Aligned Instructions ‣ 2.3 Instruction Template ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>) as a result of training on intruction tuning data with longer responses (Table <a href="#S1.T1" title="Table 1 ‣ 1 Introduction ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).
To rule this out, we re-evaluate our models separately with two contrasting groups: evaluated with only those examples with shorter generation length (less than or equal to), and those examples with much longer generation length (100 more tokens) compared with the golden answer. As shown in Figure <a href="#S2.F5" title="Figure 5 ‣ Objective-Aligned Instructions ‣ 2.3 Instruction Template ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, both short and long generations see very substantial drops in faithfulness, while longer generations indeed see larger drops. This nevertheless supports our conclusion that instruction following training hurts faithfulness.</p>
</div>
<figure id="S3.F8" class="ltx_figure"><img src="/html/2407.21417/assets/x2.png" id="S3.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="105" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>The illustration of our proposed method <span id="S3.F8.2.1" class="ltx_text ltx_font_smallcaps">ReSet</span>. It samples generations from the initial vanilla multi-task learning (MTL) checkpoint with seen examples from instruction following and context-dependent datasets. For each example, it generates a set of possible responses with different decoding strategies. Generations are rated by external judges with a weighted scores of task performance, faithfulness and instruction following scores. Then, the top rated generations will be collected to further fine-tune the initial model.</figcaption>
</figure>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Does Context-Dependent Fine-tuning Hurt Instruction Following?</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">To answer this question, we fine-tune Vicuna-7B (LLaMA-7B fine-tuned on ShareGPT) on context-dependent datasets. Figure <a href="#S2.F6" title="Figure 6 ‣ Models ‣ 2.4 Two-stage Fine-tuning Paradigm ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> shows our results. As expected, our model becomes more faithful, with improved task performance. This improvement is partly because Vicuna-7B does not follow our instruction template out-of-the-box. On the other hand, instruction following scores drop by 37.9%, suggesting our model becomes much less human-aligned. Similar to previous findings, one natural concern is that context-dependent training makes our model generate much shorter responses, which leads to lower instruction following scores. To rule out this concern, we randomly sample 200 examples from the Alpaca-15K dataset and only keep those with a minimal sequence length change (a maximum of 10 new tokens) compared to the corresponding Vicuna-7B zero-shot generation. As shown in Figure <a href="#S2.F7" title="Figure 7 ‣ Models ‣ 2.4 Two-stage Fine-tuning Paradigm ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, instruction following scores consistently drop considerably for the filtered setting, regardless of length (36.8% vs. 22.0%). Overall, context-dependent fine-tuning negatively affects the instruction following score, as we expected.</p>
</div>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span><span id="S5.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">ReSet</span>: Reconciling Instruction Following and Faithfulness</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">As shown in the previous two sections, there are clear objective discrepancies between diverse instruction following datasets and context-grounded ones. In this section, we explore two different mitigation strategies, starting from the MTL baseline to our novel method based on <span id="S5.p1.1.1" class="ltx_text ltx_font_bold">Re</span>jection Sampling for Continued <span id="S5.p1.1.2" class="ltx_text ltx_font_bold">Se</span>lf-instruction <span id="S5.p1.1.3" class="ltx_text ltx_font_bold">T</span>uning (<span id="S5.p1.1.4" class="ltx_text ltx_font_bold ltx_font_smallcaps">ReSet</span>), which significantly outperforms MTL.</p>
</div>
<section id="S5.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Our MTL Baseline</h5>

<div id="S5.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px1.p1.1" class="ltx_p">Our MTL simply mixes up the instruction following as well as context-dependent datasets. We up-sample smaller datasets to have an equal number of examples per dataset.</p>
</div>
</section>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Our Method</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p"><span id="S5.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span> starts with a model that is fine-tuned on a mixture of instruction following and context-dependent datasets and reconciles the two objectives with the following steps (Figure <a href="#S3.F8" title="Figure 8 ‣ 3 Does Fine-tuning with Instruction Following Data Hurt Faithfulness? ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>):</p>
</div>
<section id="S5.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Sample Generations</h5>

<div id="S5.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px1.p1.1" class="ltx_p">For a random subset of the training datasets, we sample generations from the checkpoint with different decoding settings. Specifically, we focus on two hyperparameters by randomly changing one of them at a time by enumerating all possible values. For the decoding temperature, it takes on a value from {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7}. For top-k, we set it to take on a value from {5, 10, 20, 50, 70, 90, 100}. When we vary the temperature, we fix top-k to be 0. Similarly, when we vary the top-k, we fix the temperature at 0. For each example, if we run the sampling procedure once, it will sample 7 examples in total.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">External Judges</h5>

<div id="S5.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px2.p1.1" class="ltx_p">We then use a set of external judges to rate our collected generations as in Section <a href="#S2.SS2" title="2.2 Evaluation Metrics ‣ 2 Background ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a>. These judges are offline evaluators. We evaluate generation based on instruction following score, faithfulness score and task performance. One potential limitation is that our model may overfit to existing judges on seen datasets. We thus include unseen datasets to test generalizability.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Top-1 Weighted Score</h5>

<div id="S5.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px3.p1.3" class="ltx_p">For each generation, we collect a set of scores from our external judges. Then, we take a weighted sum of these scores before ranking the generations and picking the top rated one. The score is weighted score of task performance <math id="S5.SS1.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="s_{\text{task}}" display="inline"><semantics id="S5.SS1.SSS0.Px3.p1.1.m1.1a"><msub id="S5.SS1.SSS0.Px3.p1.1.m1.1.1" xref="S5.SS1.SSS0.Px3.p1.1.m1.1.1.cmml"><mi id="S5.SS1.SSS0.Px3.p1.1.m1.1.1.2" xref="S5.SS1.SSS0.Px3.p1.1.m1.1.1.2.cmml">s</mi><mtext id="S5.SS1.SSS0.Px3.p1.1.m1.1.1.3" xref="S5.SS1.SSS0.Px3.p1.1.m1.1.1.3a.cmml">task</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px3.p1.1.m1.1b"><apply id="S5.SS1.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.1.m1.1.1">subscript</csymbol><ci id="S5.SS1.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S5.SS1.SSS0.Px3.p1.1.m1.1.1.2">𝑠</ci><ci id="S5.SS1.SSS0.Px3.p1.1.m1.1.1.3a.cmml" xref="S5.SS1.SSS0.Px3.p1.1.m1.1.1.3"><mtext mathsize="70%" id="S5.SS1.SSS0.Px3.p1.1.m1.1.1.3.cmml" xref="S5.SS1.SSS0.Px3.p1.1.m1.1.1.3">task</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px3.p1.1.m1.1c">s_{\text{task}}</annotation></semantics></math>, instruction following <math id="S5.SS1.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="s_{\text{instr}}" display="inline"><semantics id="S5.SS1.SSS0.Px3.p1.2.m2.1a"><msub id="S5.SS1.SSS0.Px3.p1.2.m2.1.1" xref="S5.SS1.SSS0.Px3.p1.2.m2.1.1.cmml"><mi id="S5.SS1.SSS0.Px3.p1.2.m2.1.1.2" xref="S5.SS1.SSS0.Px3.p1.2.m2.1.1.2.cmml">s</mi><mtext id="S5.SS1.SSS0.Px3.p1.2.m2.1.1.3" xref="S5.SS1.SSS0.Px3.p1.2.m2.1.1.3a.cmml">instr</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px3.p1.2.m2.1b"><apply id="S5.SS1.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.2.m2.1.1">subscript</csymbol><ci id="S5.SS1.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S5.SS1.SSS0.Px3.p1.2.m2.1.1.2">𝑠</ci><ci id="S5.SS1.SSS0.Px3.p1.2.m2.1.1.3a.cmml" xref="S5.SS1.SSS0.Px3.p1.2.m2.1.1.3"><mtext mathsize="70%" id="S5.SS1.SSS0.Px3.p1.2.m2.1.1.3.cmml" xref="S5.SS1.SSS0.Px3.p1.2.m2.1.1.3">instr</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px3.p1.2.m2.1c">s_{\text{instr}}</annotation></semantics></math> and faithfulness scores <math id="S5.SS1.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="s_{\text{faith}}" display="inline"><semantics id="S5.SS1.SSS0.Px3.p1.3.m3.1a"><msub id="S5.SS1.SSS0.Px3.p1.3.m3.1.1" xref="S5.SS1.SSS0.Px3.p1.3.m3.1.1.cmml"><mi id="S5.SS1.SSS0.Px3.p1.3.m3.1.1.2" xref="S5.SS1.SSS0.Px3.p1.3.m3.1.1.2.cmml">s</mi><mtext id="S5.SS1.SSS0.Px3.p1.3.m3.1.1.3" xref="S5.SS1.SSS0.Px3.p1.3.m3.1.1.3a.cmml">faith</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px3.p1.3.m3.1b"><apply id="S5.SS1.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px3.p1.3.m3.1.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.3.m3.1.1">subscript</csymbol><ci id="S5.SS1.SSS0.Px3.p1.3.m3.1.1.2.cmml" xref="S5.SS1.SSS0.Px3.p1.3.m3.1.1.2">𝑠</ci><ci id="S5.SS1.SSS0.Px3.p1.3.m3.1.1.3a.cmml" xref="S5.SS1.SSS0.Px3.p1.3.m3.1.1.3"><mtext mathsize="70%" id="S5.SS1.SSS0.Px3.p1.3.m3.1.1.3.cmml" xref="S5.SS1.SSS0.Px3.p1.3.m3.1.1.3">faith</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px3.p1.3.m3.1c">s_{\text{faith}}</annotation></semantics></math>,</p>
<table id="S5.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S5.Ex1.m1.1" class="ltx_Math" alttext="\mathrm{score}=s_{\text{task}}+2.0*(\mathbb{I}_{\text{instr}}*s_{\text{instr}}+\mathbb{I}_{\text{faith}}*s_{\text{faith}})" display="block"><semantics id="S5.Ex1.m1.1a"><mrow id="S5.Ex1.m1.1.1" xref="S5.Ex1.m1.1.1.cmml"><mi id="S5.Ex1.m1.1.1.3" xref="S5.Ex1.m1.1.1.3.cmml">score</mi><mo id="S5.Ex1.m1.1.1.2" xref="S5.Ex1.m1.1.1.2.cmml">=</mo><mrow id="S5.Ex1.m1.1.1.1" xref="S5.Ex1.m1.1.1.1.cmml"><msub id="S5.Ex1.m1.1.1.1.3" xref="S5.Ex1.m1.1.1.1.3.cmml"><mi id="S5.Ex1.m1.1.1.1.3.2" xref="S5.Ex1.m1.1.1.1.3.2.cmml">s</mi><mtext id="S5.Ex1.m1.1.1.1.3.3" xref="S5.Ex1.m1.1.1.1.3.3a.cmml">task</mtext></msub><mo id="S5.Ex1.m1.1.1.1.2" xref="S5.Ex1.m1.1.1.1.2.cmml">+</mo><mrow id="S5.Ex1.m1.1.1.1.1" xref="S5.Ex1.m1.1.1.1.1.cmml"><mn id="S5.Ex1.m1.1.1.1.1.3" xref="S5.Ex1.m1.1.1.1.1.3.cmml">2.0</mn><mo lspace="0.222em" rspace="0.222em" id="S5.Ex1.m1.1.1.1.1.2" xref="S5.Ex1.m1.1.1.1.1.2.cmml">∗</mo><mrow id="S5.Ex1.m1.1.1.1.1.1.1" xref="S5.Ex1.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S5.Ex1.m1.1.1.1.1.1.1.2" xref="S5.Ex1.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S5.Ex1.m1.1.1.1.1.1.1.1" xref="S5.Ex1.m1.1.1.1.1.1.1.1.cmml"><mrow id="S5.Ex1.m1.1.1.1.1.1.1.1.2" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.cmml"><msub id="S5.Ex1.m1.1.1.1.1.1.1.1.2.2" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.cmml"><mi id="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.2" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.2.cmml">𝕀</mi><mtext id="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.3" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.3a.cmml">instr</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="S5.Ex1.m1.1.1.1.1.1.1.1.2.1" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.1.cmml">∗</mo><msub id="S5.Ex1.m1.1.1.1.1.1.1.1.2.3" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.cmml"><mi id="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.2" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.2.cmml">s</mi><mtext id="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.3" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.3a.cmml">instr</mtext></msub></mrow><mo id="S5.Ex1.m1.1.1.1.1.1.1.1.1" xref="S5.Ex1.m1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="S5.Ex1.m1.1.1.1.1.1.1.1.3" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.cmml"><msub id="S5.Ex1.m1.1.1.1.1.1.1.1.3.2" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.cmml"><mi id="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.2" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.2.cmml">𝕀</mi><mtext id="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.3" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.3a.cmml">faith</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="S5.Ex1.m1.1.1.1.1.1.1.1.3.1" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.1.cmml">∗</mo><msub id="S5.Ex1.m1.1.1.1.1.1.1.1.3.3" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.cmml"><mi id="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.2" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.2.cmml">s</mi><mtext id="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.3" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.3a.cmml">faith</mtext></msub></mrow></mrow><mo stretchy="false" id="S5.Ex1.m1.1.1.1.1.1.1.3" xref="S5.Ex1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.Ex1.m1.1b"><apply id="S5.Ex1.m1.1.1.cmml" xref="S5.Ex1.m1.1.1"><eq id="S5.Ex1.m1.1.1.2.cmml" xref="S5.Ex1.m1.1.1.2"></eq><ci id="S5.Ex1.m1.1.1.3.cmml" xref="S5.Ex1.m1.1.1.3">score</ci><apply id="S5.Ex1.m1.1.1.1.cmml" xref="S5.Ex1.m1.1.1.1"><plus id="S5.Ex1.m1.1.1.1.2.cmml" xref="S5.Ex1.m1.1.1.1.2"></plus><apply id="S5.Ex1.m1.1.1.1.3.cmml" xref="S5.Ex1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S5.Ex1.m1.1.1.1.3.1.cmml" xref="S5.Ex1.m1.1.1.1.3">subscript</csymbol><ci id="S5.Ex1.m1.1.1.1.3.2.cmml" xref="S5.Ex1.m1.1.1.1.3.2">𝑠</ci><ci id="S5.Ex1.m1.1.1.1.3.3a.cmml" xref="S5.Ex1.m1.1.1.1.3.3"><mtext mathsize="70%" id="S5.Ex1.m1.1.1.1.3.3.cmml" xref="S5.Ex1.m1.1.1.1.3.3">task</mtext></ci></apply><apply id="S5.Ex1.m1.1.1.1.1.cmml" xref="S5.Ex1.m1.1.1.1.1"><times id="S5.Ex1.m1.1.1.1.1.2.cmml" xref="S5.Ex1.m1.1.1.1.1.2"></times><cn type="float" id="S5.Ex1.m1.1.1.1.1.3.cmml" xref="S5.Ex1.m1.1.1.1.1.3">2.0</cn><apply id="S5.Ex1.m1.1.1.1.1.1.1.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1"><plus id="S5.Ex1.m1.1.1.1.1.1.1.1.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.1"></plus><apply id="S5.Ex1.m1.1.1.1.1.1.1.1.2.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2"><times id="S5.Ex1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.1"></times><apply id="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.2">𝕀</ci><ci id="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.3a.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.3"><mtext mathsize="70%" id="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.2.3">instr</mtext></ci></apply><apply id="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.2.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.2">𝑠</ci><ci id="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.3a.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.3"><mtext mathsize="70%" id="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.3.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.2.3.3">instr</mtext></ci></apply></apply><apply id="S5.Ex1.m1.1.1.1.1.1.1.1.3.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3"><times id="S5.Ex1.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.1"></times><apply id="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.2">𝕀</ci><ci id="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.3a.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.3"><mtext mathsize="70%" id="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.2.3">faith</mtext></ci></apply><apply id="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.2.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.2">𝑠</ci><ci id="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.3a.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.3"><mtext mathsize="70%" id="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.3.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1.3.3.3">faith</mtext></ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.Ex1.m1.1c">\mathrm{score}=s_{\text{task}}+2.0*(\mathbb{I}_{\text{instr}}*s_{\text{instr}}+\mathbb{I}_{\text{faith}}*s_{\text{faith}})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S5.SS1.SSS0.Px3.p1.5" class="ltx_p">where we use <math id="S5.SS1.SSS0.Px3.p1.4.m1.1" class="ltx_Math" alttext="\mathbb{I}_{\text{instr}}" display="inline"><semantics id="S5.SS1.SSS0.Px3.p1.4.m1.1a"><msub id="S5.SS1.SSS0.Px3.p1.4.m1.1.1" xref="S5.SS1.SSS0.Px3.p1.4.m1.1.1.cmml"><mi id="S5.SS1.SSS0.Px3.p1.4.m1.1.1.2" xref="S5.SS1.SSS0.Px3.p1.4.m1.1.1.2.cmml">𝕀</mi><mtext id="S5.SS1.SSS0.Px3.p1.4.m1.1.1.3" xref="S5.SS1.SSS0.Px3.p1.4.m1.1.1.3a.cmml">instr</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px3.p1.4.m1.1b"><apply id="S5.SS1.SSS0.Px3.p1.4.m1.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.4.m1.1.1"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px3.p1.4.m1.1.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.4.m1.1.1">subscript</csymbol><ci id="S5.SS1.SSS0.Px3.p1.4.m1.1.1.2.cmml" xref="S5.SS1.SSS0.Px3.p1.4.m1.1.1.2">𝕀</ci><ci id="S5.SS1.SSS0.Px3.p1.4.m1.1.1.3a.cmml" xref="S5.SS1.SSS0.Px3.p1.4.m1.1.1.3"><mtext mathsize="70%" id="S5.SS1.SSS0.Px3.p1.4.m1.1.1.3.cmml" xref="S5.SS1.SSS0.Px3.p1.4.m1.1.1.3">instr</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px3.p1.4.m1.1c">\mathbb{I}_{\text{instr}}</annotation></semantics></math> and <math id="S5.SS1.SSS0.Px3.p1.5.m2.1" class="ltx_Math" alttext="\mathbb{I}_{\text{faith}}" display="inline"><semantics id="S5.SS1.SSS0.Px3.p1.5.m2.1a"><msub id="S5.SS1.SSS0.Px3.p1.5.m2.1.1" xref="S5.SS1.SSS0.Px3.p1.5.m2.1.1.cmml"><mi id="S5.SS1.SSS0.Px3.p1.5.m2.1.1.2" xref="S5.SS1.SSS0.Px3.p1.5.m2.1.1.2.cmml">𝕀</mi><mtext id="S5.SS1.SSS0.Px3.p1.5.m2.1.1.3" xref="S5.SS1.SSS0.Px3.p1.5.m2.1.1.3a.cmml">faith</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px3.p1.5.m2.1b"><apply id="S5.SS1.SSS0.Px3.p1.5.m2.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.5.m2.1.1"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px3.p1.5.m2.1.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.5.m2.1.1">subscript</csymbol><ci id="S5.SS1.SSS0.Px3.p1.5.m2.1.1.2.cmml" xref="S5.SS1.SSS0.Px3.p1.5.m2.1.1.2">𝕀</ci><ci id="S5.SS1.SSS0.Px3.p1.5.m2.1.1.3a.cmml" xref="S5.SS1.SSS0.Px3.p1.5.m2.1.1.3"><mtext mathsize="70%" id="S5.SS1.SSS0.Px3.p1.5.m2.1.1.3.cmml" xref="S5.SS1.SSS0.Px3.p1.5.m2.1.1.3">faith</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px3.p1.5.m2.1c">\mathbb{I}_{\text{faith}}</annotation></semantics></math> to indicate whether the example is from our instruction following datasets or context-dependent datasets.
We pick the top rated sample per example and combine them into a continued fine-tuning dataset as described next.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Continued Fine-tuning</h5>

<div id="S5.SS1.SSS0.Px4.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px4.p1.1" class="ltx_p">For <span id="S5.SS1.SSS0.Px4.p1.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span>, we randomly sample training data from each of our four dataset in Table <a href="#S1.T1" title="Table 1 ‣ 1 Introduction ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and collect 2,000 additional fine-tuning examples per dataset. In total, our new fine-tuning dataset has 8,000 examples. With this small collected dataset, we further fine-tune our starting checkpoint model for a single epoch with a smaller learning rate to avoid overfitting. This continued fine-tuning step is very lightweight as the training data is usually less than 1% of the MTL training step. Overall, <span id="S5.SS1.SSS0.Px4.p1.1.2" class="ltx_text ltx_font_smallcaps">ReSet</span> resembles a reject sampling based preference learning paradigm, which has been proven effective <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite> while drastically saving training costs as well as increasing stability. Other experimental setup details are included in Appendix <a href="#A1.SS3" title="A.3 Experimental Setup ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.3</span></a>.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px5" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Supercharged <span id="S5.SS1.SSS0.Px5.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span> (<span id="S5.SS1.SSS0.Px5.2.2" class="ltx_text ltx_font_smallcaps">ReSet</span>-S)</h5>

<div id="S5.SS1.SSS0.Px5.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px5.p1.1" class="ltx_p">In addition, we evaluate <span id="S5.SS1.SSS0.Px5.p1.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span> with a different setting to test the impact of the quality of our continued fine-tuning dataset. Specifically, we <em id="S5.SS1.SSS0.Px5.p1.1.2" class="ltx_emph ltx_font_italic">supercharge</em> the quality of our additional fine-tuning dataset by sampling 1<math id="S5.SS1.SSS0.Px5.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.SS1.SSS0.Px5.p1.1.m1.1a"><mo id="S5.SS1.SSS0.Px5.p1.1.m1.1.1" xref="S5.SS1.SSS0.Px5.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px5.p1.1.m1.1b"><times id="S5.SS1.SSS0.Px5.p1.1.m1.1.1.cmml" xref="S5.SS1.SSS0.Px5.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px5.p1.1.m1.1c">\times</annotation></semantics></math> more generations. We swap our instruction following judge from GPT-4 to the weaker ChatGPT<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span>We use <span id="footnote7.1" class="ltx_text ltx_font_typewriter">gpt-4-0613</span>.</span></span></span>. One potential benefits is that ChatGPT is a weaker judge with a lower recall for good generations. As a result, examples rated high by ChatGPT may have higher quality.
While sampling for generation, we decrease the curated fine-tuning dataset by 3-fold (2,000 examples in total). We fine-tune our checkpoint model with the same setup.</p>
</div>
<figure id="S5.T4" class="ltx_table">
<div id="S5.T4.1" class="ltx_inline-block ltx_transformed_outer" style="width:433.6pt;height:107.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-108.7pt,27.1pt) scale(0.665968179821523,0.665968179821523) ;">
<table id="S5.T4.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T4.1.1.1" class="ltx_tr">
<td id="S5.T4.1.1.1.1" class="ltx_td ltx_border_tt"></td>
<td id="S5.T4.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="9"><span id="S5.T4.1.1.1.2.1" class="ltx_text ltx_font_bold">Faithfulness</span></td>
<td id="S5.T4.1.1.1.3" class="ltx_td ltx_border_tt"></td>
<td id="S5.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S5.T4.1.1.1.4.1" class="ltx_text ltx_font_bold">Instruction Following</span></td>
</tr>
<tr id="S5.T4.1.1.2" class="ltx_tr">
<td id="S5.T4.1.1.2.1" class="ltx_td"></td>
<td id="S5.T4.1.1.2.2" class="ltx_td ltx_align_center" colspan="4"><span id="S5.T4.1.1.2.2.1" class="ltx_text ltx_font_bold">Abstractive Datasets</span></td>
<td id="S5.T4.1.1.2.3" class="ltx_td"></td>
<td id="S5.T4.1.1.2.4" class="ltx_td ltx_align_center" colspan="4"><span id="S5.T4.1.1.2.4.1" class="ltx_text ltx_font_bold">Extractive Datasets</span></td>
<td id="S5.T4.1.1.2.5" class="ltx_td"></td>
<td id="S5.T4.1.1.2.6" class="ltx_td ltx_align_center"><span id="S5.T4.1.1.2.6.1" class="ltx_text ltx_font_bold">Datasets</span></td>
</tr>
<tr id="S5.T4.1.1.3" class="ltx_tr">
<td id="S5.T4.1.1.3.1" class="ltx_td ltx_border_t"></td>
<td id="S5.T4.1.1.3.2" class="ltx_td ltx_align_center ltx_border_t">CNN DailyMail</td>
<td id="S5.T4.1.1.3.3" class="ltx_td ltx_align_center ltx_border_t">MS MARCO</td>
<td id="S5.T4.1.1.3.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T4.1.1.3.4.1" class="ltx_text ltx_font_italic">WikiSum</span></td>
<td id="S5.T4.1.1.3.5" class="ltx_td ltx_align_center ltx_border_t">Overall</td>
<td id="S5.T4.1.1.3.6" class="ltx_td ltx_border_t"></td>
<td id="S5.T4.1.1.3.7" class="ltx_td ltx_align_center ltx_border_t">NQ</td>
<td id="S5.T4.1.1.3.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T4.1.1.3.8.1" class="ltx_text ltx_font_italic">SearchQA</span></td>
<td id="S5.T4.1.1.3.9" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T4.1.1.3.9.1" class="ltx_text ltx_font_italic">BioASQ</span></td>
<td id="S5.T4.1.1.3.10" class="ltx_td ltx_align_center ltx_border_t">Overall</td>
<td id="S5.T4.1.1.3.11" class="ltx_td ltx_border_t"></td>
<td id="S5.T4.1.1.3.12" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T4.1.1.3.12.1" class="ltx_text ltx_font_italic">Alignment Eval</span></td>
</tr>
<tr id="S5.T4.1.1.4" class="ltx_tr">
<td id="S5.T4.1.1.4.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S5.T4.1.1.4.1.1" class="ltx_text ltx_font_bold">Vicuna-7B</span></td>
<td id="S5.T4.1.1.4.2" class="ltx_td ltx_align_center ltx_border_t">0.36</td>
<td id="S5.T4.1.1.4.3" class="ltx_td ltx_align_center ltx_border_t">0.59</td>
<td id="S5.T4.1.1.4.4" class="ltx_td ltx_align_center ltx_border_t">0.34</td>
<td id="S5.T4.1.1.4.5" class="ltx_td ltx_align_center ltx_border_t">0.43</td>
<td id="S5.T4.1.1.4.6" class="ltx_td ltx_border_t"></td>
<td id="S5.T4.1.1.4.7" class="ltx_td ltx_align_center ltx_border_t">0.06</td>
<td id="S5.T4.1.1.4.8" class="ltx_td ltx_align_center ltx_border_t">0.21</td>
<td id="S5.T4.1.1.4.9" class="ltx_td ltx_align_center ltx_border_t">0.02</td>
<td id="S5.T4.1.1.4.10" class="ltx_td ltx_align_center ltx_border_t">0.10</td>
<td id="S5.T4.1.1.4.11" class="ltx_td ltx_border_t"></td>
<td id="S5.T4.1.1.4.12" class="ltx_td ltx_align_center ltx_border_t">0.79</td>
</tr>
<tr id="S5.T4.1.1.5" class="ltx_tr">
<td id="S5.T4.1.1.5.1" class="ltx_td ltx_align_left"><span id="S5.T4.1.1.5.1.1" class="ltx_text ltx_font_bold">w/o MTL</span></td>
<td id="S5.T4.1.1.5.2" class="ltx_td ltx_align_center">0.67</td>
<td id="S5.T4.1.1.5.3" class="ltx_td ltx_align_center"><span id="S5.T4.1.1.5.3.1" class="ltx_text ltx_font_bold">0.93</span></td>
<td id="S5.T4.1.1.5.4" class="ltx_td ltx_align_center">0.85</td>
<td id="S5.T4.1.1.5.5" class="ltx_td ltx_align_center">0.82</td>
<td id="S5.T4.1.1.5.6" class="ltx_td"></td>
<td id="S5.T4.1.1.5.7" class="ltx_td ltx_align_center">0.89</td>
<td id="S5.T4.1.1.5.8" class="ltx_td ltx_align_center">0.94</td>
<td id="S5.T4.1.1.5.9" class="ltx_td ltx_align_center">0.93</td>
<td id="S5.T4.1.1.5.10" class="ltx_td ltx_align_center">0.92</td>
<td id="S5.T4.1.1.5.11" class="ltx_td"></td>
<td id="S5.T4.1.1.5.12" class="ltx_td ltx_align_center">0.49</td>
</tr>
<tr id="S5.T4.1.1.6" class="ltx_tr">
<td id="S5.T4.1.1.6.1" class="ltx_td ltx_align_left"><span id="S5.T4.1.1.6.1.1" class="ltx_text ltx_font_bold">w/ MTL</span></td>
<td id="S5.T4.1.1.6.2" class="ltx_td ltx_align_center">0.67</td>
<td id="S5.T4.1.1.6.3" class="ltx_td ltx_align_center">0.92</td>
<td id="S5.T4.1.1.6.4" class="ltx_td ltx_align_center">0.80</td>
<td id="S5.T4.1.1.6.5" class="ltx_td ltx_align_center">0.80</td>
<td id="S5.T4.1.1.6.6" class="ltx_td"></td>
<td id="S5.T4.1.1.6.7" class="ltx_td ltx_align_center">0.90</td>
<td id="S5.T4.1.1.6.8" class="ltx_td ltx_align_center">0.94</td>
<td id="S5.T4.1.1.6.9" class="ltx_td ltx_align_center">0.93</td>
<td id="S5.T4.1.1.6.10" class="ltx_td ltx_align_center">0.92</td>
<td id="S5.T4.1.1.6.11" class="ltx_td"></td>
<td id="S5.T4.1.1.6.12" class="ltx_td ltx_align_center">0.75</td>
</tr>
<tr id="S5.T4.1.1.7" class="ltx_tr">
<td id="S5.T4.1.1.7.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S5.T4.1.1.7.1.1" class="ltx_text ltx_font_bold">w/ <span id="S5.T4.1.1.7.1.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span></span></td>
<td id="S5.T4.1.1.7.2" class="ltx_td ltx_align_center ltx_border_t">0.80</td>
<td id="S5.T4.1.1.7.3" class="ltx_td ltx_align_center ltx_border_t">0.90</td>
<td id="S5.T4.1.1.7.4" class="ltx_td ltx_align_center ltx_border_t">0.88</td>
<td id="S5.T4.1.1.7.5" class="ltx_td ltx_align_center ltx_border_t">0.86</td>
<td id="S5.T4.1.1.7.6" class="ltx_td ltx_border_t"></td>
<td id="S5.T4.1.1.7.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T4.1.1.7.7.1" class="ltx_text ltx_font_bold">0.96</span></td>
<td id="S5.T4.1.1.7.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T4.1.1.7.8.1" class="ltx_text ltx_font_bold">0.99</span></td>
<td id="S5.T4.1.1.7.9" class="ltx_td ltx_align_center ltx_border_t">0.92</td>
<td id="S5.T4.1.1.7.10" class="ltx_td ltx_align_center ltx_border_t">0.96</td>
<td id="S5.T4.1.1.7.11" class="ltx_td ltx_border_t"></td>
<td id="S5.T4.1.1.7.12" class="ltx_td ltx_align_center ltx_border_t">0.73</td>
</tr>
<tr id="S5.T4.1.1.8" class="ltx_tr">
<td id="S5.T4.1.1.8.1" class="ltx_td ltx_align_left"><span id="S5.T4.1.1.8.1.1" class="ltx_text ltx_font_bold">w/ MTL+<span id="S5.T4.1.1.8.1.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span></span></td>
<td id="S5.T4.1.1.8.2" class="ltx_td ltx_align_center">0.77</td>
<td id="S5.T4.1.1.8.3" class="ltx_td ltx_align_center"><span id="S5.T4.1.1.8.3.1" class="ltx_text ltx_font_bold">0.93</span></td>
<td id="S5.T4.1.1.8.4" class="ltx_td ltx_align_center"><span id="S5.T4.1.1.8.4.1" class="ltx_text ltx_font_bold">0.95</span></td>
<td id="S5.T4.1.1.8.5" class="ltx_td ltx_align_center">0.85</td>
<td id="S5.T4.1.1.8.6" class="ltx_td"></td>
<td id="S5.T4.1.1.8.7" class="ltx_td ltx_align_center"><span id="S5.T4.1.1.8.7.1" class="ltx_text ltx_font_bold">0.96</span></td>
<td id="S5.T4.1.1.8.8" class="ltx_td ltx_align_center">0.97</td>
<td id="S5.T4.1.1.8.9" class="ltx_td ltx_align_center"><span id="S5.T4.1.1.8.9.1" class="ltx_text ltx_font_bold">0.98</span></td>
<td id="S5.T4.1.1.8.10" class="ltx_td ltx_align_center"><span id="S5.T4.1.1.8.10.1" class="ltx_text ltx_font_bold">0.97</span></td>
<td id="S5.T4.1.1.8.11" class="ltx_td"></td>
<td id="S5.T4.1.1.8.12" class="ltx_td ltx_align_center">0.76</td>
</tr>
<tr id="S5.T4.1.1.9" class="ltx_tr">
<td id="S5.T4.1.1.9.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S5.T4.1.1.9.1.1" class="ltx_text ltx_font_bold">w/ MTL+<span id="S5.T4.1.1.9.1.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span>-S</span></td>
<td id="S5.T4.1.1.9.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.9.2.1" class="ltx_text ltx_font_bold">0.88</span></td>
<td id="S5.T4.1.1.9.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.9.3.1" class="ltx_text ltx_font_bold">0.93</span></td>
<td id="S5.T4.1.1.9.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.9.4.1" class="ltx_text ltx_font_bold">0.95</span></td>
<td id="S5.T4.1.1.9.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.9.5.1" class="ltx_text ltx_font_bold">0.92</span></td>
<td id="S5.T4.1.1.9.6" class="ltx_td ltx_border_bb"></td>
<td id="S5.T4.1.1.9.7" class="ltx_td ltx_align_center ltx_border_bb">0.94</td>
<td id="S5.T4.1.1.9.8" class="ltx_td ltx_align_center ltx_border_bb">0.97</td>
<td id="S5.T4.1.1.9.9" class="ltx_td ltx_align_center ltx_border_bb">0.97</td>
<td id="S5.T4.1.1.9.10" class="ltx_td ltx_align_center ltx_border_bb">0.96</td>
<td id="S5.T4.1.1.9.11" class="ltx_td ltx_border_bb"></td>
<td id="S5.T4.1.1.9.12" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.9.12.1" class="ltx_text ltx_font_bold">0.77</span></td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 4: </span>Faithfulness and alignment scores on testing datasets, and unseen datasets are <span id="S5.T4.3.1" class="ltx_text ltx_font_italic">italicized</span>.
Scores are averaged across three distinct runs. Higher scores are better. Overall scores are macro-average across datasets.</figcaption>
</figure>
</section>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Results</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">Table <a href="#S5.T4" title="Table 4 ‣ Supercharged ReSet (ReSet-S) ‣ 5.1 Our Method ‣ 5 ReSet: Reconciling Instruction Following and Faithfulness ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows results on models fine-tuned with other baselines:
(1) <span id="S5.SS2.p1.1.1" class="ltx_text ltx_font_bold">Vicuna-7B</span>: Evaluated with Vicuna-7B in a zero-shot manner.
(2) <span id="S5.SS2.p1.1.2" class="ltx_text ltx_font_bold">w/o MTL</span>: We fine-tune Vicuna-7B with context-dependent datasets without any MTL objective. The purpose is to establish a potential upper bound of faithfulness score without any further training for instruction following.
(3) <span id="S5.SS2.p1.1.3" class="ltx_text ltx_font_bold">w/ MTL</span>: We fine-tune Vicuna-7B with baseline MTL by mixing instruction following and context-dependent datasets together.
(4) <span id="S5.SS2.p1.1.4" class="ltx_text ltx_font_bold">w/ <span id="S5.SS2.p1.1.4.1" class="ltx_text ltx_font_smallcaps">ReSet</span></span>: We fine-tune Vicuna-7B directly with our collected continued fine-tuning dataset (8,000 examples) from our <span id="S5.SS2.p1.1.5" class="ltx_text ltx_font_smallcaps">ReSet</span> pipeline without MTL training first. Note that the curated dataset comprises of sampled generations from our MTL checkpoint. The purpose is to evaluate whether MTL is necessary for model improvements.
(5) <span id="S5.SS2.p1.1.6" class="ltx_text ltx_font_bold">w/ MTL+<span id="S5.SS2.p1.1.6.1" class="ltx_text ltx_font_smallcaps">ReSet</span></span>: We follow Figure <a href="#S3.F8" title="Figure 8 ‣ 3 Does Fine-tuning with Instruction Following Data Hurt Faithfulness? ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> to fine-tune our MTL checkpoint with curated dataset.
(6) <span id="S5.SS2.p1.1.7" class="ltx_text ltx_font_bold">w/ MTL+<span id="S5.SS2.p1.1.7.1" class="ltx_text ltx_font_smallcaps">ReSet</span>-S</span>: We follow Figure <a href="#S3.F8" title="Figure 8 ‣ 3 Does Fine-tuning with Instruction Following Data Hurt Faithfulness? ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> to fine-tune our MTL checkpoint with the high-quality version of our curated dataset (2,000 examples).</p>
</div>
<div id="S5.SS2.p2" class="ltx_para">
<p id="S5.SS2.p2.1" class="ltx_p">We evaluate our models on both seen and unseen testing datasets. As shown in Table <a href="#S5.T4" title="Table 4 ‣ Supercharged ReSet (ReSet-S) ‣ 5.1 Our Method ‣ 5 ReSet: Reconciling Instruction Following and Faithfulness ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, fine-tuning Vicuna-7B with a mixture of datasets close the gap on instruction following score substantially (from 0.49 to 0.75) while leaving headroom across the board. Next, models fine-tuned with <span id="S5.SS2.p2.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span> significantly outperform the MTL baseline on both seen and unseen testing datasets. Our results also suggest directly fine-tuning Vicuna-7B performs worse compared to fine-tuning a model checkpoint after MTL training. Last but not least, our results with <span id="S5.SS2.p2.1.2" class="ltx_text ltx_font_smallcaps">ReSet</span>-S provide strong evidence that data quality is more important than data quantity and that using multiple iterations of sampling helps, as <span id="S5.SS2.p2.1.3" class="ltx_text ltx_font_smallcaps">ReSet</span>-S achieves similar or better performance with 3-fold less training data. Due to space limit, we present qualitative model generations in Appendix <a href="#A1.SS7" title="A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.7</span></a>.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Related Work</h2>

<section id="S6.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Instruction Following of LMs</h5>

<div id="S6.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S6.SS0.SSS0.Px1.p1.1" class="ltx_p">There are a variety of instruction following training datasets <cite class="ltx_cite ltx_citemacro_cite">Taori et al. (<a href="#bib.bib42" title="" class="ltx_ref">2023</a>); Geng et al. (<a href="#bib.bib13" title="" class="ltx_ref">2023</a>); Chiang et al. (<a href="#bib.bib5" title="" class="ltx_ref">2023</a>); Wang et al. (<a href="#bib.bib45" title="" class="ltx_ref">2023b</a>)</cite> covering tasks from wildly different domains such as poetry creation to context-dependent summarization. In this paper, we focus on a particular trade-off between instruction following and model faithfulness, which are fundamental objectives for modern LMs. In addition to training datasets, LM generations are often assessed by human experts <cite class="ltx_cite ltx_citemacro_cite">Ouyang et al. (<a href="#bib.bib36" title="" class="ltx_ref">2022</a>)</cite> or model scorers <cite class="ltx_cite ltx_citemacro_cite">Zheng et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>); Ethayarajh et al. (<a href="#bib.bib11" title="" class="ltx_ref">2022</a>)</cite>. We use GPT-4 to access instruction following scores as in <cite class="ltx_cite ltx_citemacro_citet">Chiang et al. (<a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
</section>
<section id="S6.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Faithfulness and Groundedness of LMs</h5>

<div id="S6.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S6.SS0.SSS0.Px2.p1.1" class="ltx_p">Being faithful or grounded is crucial for tasks like context-dependent QA or summarizations, reducing hallucinations for LMs <cite class="ltx_cite ltx_citemacro_cite">Zhang et al. (<a href="#bib.bib49" title="" class="ltx_ref">2023</a>); Wang et al. (<a href="#bib.bib44" title="" class="ltx_ref">2023a</a>); Huang et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023</a>); Kadavath et al. (<a href="#bib.bib20" title="" class="ltx_ref">2023</a>); Chen et al. (<a href="#bib.bib3" title="" class="ltx_ref">2023</a>); Ji et al. (<a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite>
Unlike task performance, faithfulness measures whether generated answers are based on the given context <cite class="ltx_cite ltx_citemacro_cite">Rashkin et al. (<a href="#bib.bib40" title="" class="ltx_ref">2021</a>); Dziri et al. (<a href="#bib.bib10" title="" class="ltx_ref">2022</a>); Paranjape et al. (<a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite>. For context-dependent QA and summarization benchmarks, common metrics include subsequence-based lexical matching <cite class="ltx_cite ltx_citemacro_cite">Papineni et al. (<a href="#bib.bib37" title="" class="ltx_ref">2002</a>); Lin (<a href="#bib.bib28" title="" class="ltx_ref">2004</a>); Banerjee and Lavie (<a href="#bib.bib2" title="" class="ltx_ref">2005</a>)</cite>, natural language inference <cite class="ltx_cite ltx_citemacro_citep">(NLI;  Laban et al., <a href="#bib.bib25" title="" class="ltx_ref">2022</a>; Fabbri et al., <a href="#bib.bib12" title="" class="ltx_ref">2022</a>)</cite>, and more recently, LLMs as factuality scorers <cite class="ltx_cite ltx_citemacro_cite">Chiang and Lee (<a href="#bib.bib4" title="" class="ltx_ref">2023</a>); Liu et al. (<a href="#bib.bib32" title="" class="ltx_ref">2023</a>); Kamalloo et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite>. We use lexical matching and NLI to measure faithfulness by checking whether the answer is grounded.</p>
</div>
</section>
<section id="S6.SS0.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Instruction Following Training with LMs</h5>

<div id="S6.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S6.SS0.SSS0.Px3.p1.1" class="ltx_p">Base LMs are often tuned to follow human instructions <cite class="ltx_cite ltx_citemacro_cite">Wei et al. (<a href="#bib.bib47" title="" class="ltx_ref">2022</a>); Mishra et al. (<a href="#bib.bib33" title="" class="ltx_ref">2022</a>); Wang et al. (<a href="#bib.bib46" title="" class="ltx_ref">2022</a>); Chung et al. (<a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite>. Various post-hoc fine-tuning techniques have been proposed to align base LMs with human preferences, such as supervised fine-tuning on instruction-following datasets <cite class="ltx_cite ltx_citemacro_cite">Taori et al. (<a href="#bib.bib42" title="" class="ltx_ref">2023</a>); Chiang et al. (<a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite> or variances of RLHF and RLAIF <cite class="ltx_cite ltx_citemacro_cite">Schulman et al. (<a href="#bib.bib41" title="" class="ltx_ref">2017</a>); Ouyang et al. (<a href="#bib.bib36" title="" class="ltx_ref">2022</a>); Lee et al. (<a href="#bib.bib26" title="" class="ltx_ref">2023</a>); Rafailov et al. (<a href="#bib.bib39" title="" class="ltx_ref">2023</a>); Touvron et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite>. These techniques can require training policy and reward models which is not cost-efficient. Inspired by alignment training with reject-sampling <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite>, we propose a simple yet effective method that uses reject-sampling by self-instruct. On the other hand, there exist various advance MTL techniques <cite class="ltx_cite ltx_citemacro_cite">Liu et al. (<a href="#bib.bib31" title="" class="ltx_ref">2019b</a>); Crawshaw (<a href="#bib.bib9" title="" class="ltx_ref">2020</a>)</cite> prior to the development of LMs. However, <span id="S6.SS0.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span> is the only variant that leverages LM’s self-instruct generations to further fine-tune the LM. More importantly, <span id="S6.SS0.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_smallcaps">ReSet</span> is complimentary to different instruct-tuning techniques by leveraging the LM and external evaluators to generate high-quality continued fine-tuning datasets.
Compared with recent works showing how instruct-tuning may cause hallucination <cite class="ltx_cite ltx_citemacro_cite">Ghosh et al. (<a href="#bib.bib14" title="" class="ltx_ref">2024</a>)</cite>, we provide a more rigorous analysis with our two-stage fine-tuning paradigm and focus on the interaction between instruction following and faithfulness.</p>
</div>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Conclusion</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">In this paper, we demonstrate a clear trade-off between instruction following and faithfulness when fine-tuning LMs on datasets with differing objectives.
To alleviate this trade-off, we proposed <span id="S7.p1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">ReSet</span>, a simple yet effective iterative method that significantly outperforms the MTL baseline in both instruction following and faithfulness scores. The lightweight and iterative nature of <span id="S7.p1.1.2" class="ltx_text ltx_font_smallcaps">ReSet</span> makes it extensible for future refinement at minimum cost and integrable with recent instruct-tuning techniques. We contribute to the broader goal of creating more reliable, accurate, and user-aligned language technologies.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Limitations</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">The limitations of our work are as follows:</p>
<ul id="Sx1.I1" class="ltx_itemize">
<li id="Sx1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="Sx1.I1.i1.p1" class="ltx_para">
<p id="Sx1.I1.i1.p1.1" class="ltx_p">Our study primarily focuses on the LLaMA-7B and Vicuna-7B models, that are among the best open-source models at the time of this work. While we posit that our findings and the proposed <span id="Sx1.I1.i1.p1.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span> method could generalize across other language models, our findings could remain speculative without evaluating on more current model types (e.g., LLaMA-2, Mistral or Mixtral at various scales).</p>
</div>
</li>
<li id="Sx1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="Sx1.I1.i2.p1" class="ltx_para">
<p id="Sx1.I1.i2.p1.1" class="ltx_p">The datasets chosen for fine-tuning and evaluation, though comprehensive, are not exhaustive. There are other interesting datasets that are not covered in this study. For instance, long-form QA where the answers are much longer than 1-2 phrases or sentences. Our instruction following datasets can also be further categorized into creativity-driven, world-knowledge driven and others to help us to disentangle discrepancies in objectives better across datasets.</p>
</div>
</li>
<li id="Sx1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="Sx1.I1.i3.p1" class="ltx_para">
<p id="Sx1.I1.i3.p1.1" class="ltx_p">Our evaluation relies heavily on automated metrics and external judges like external LMs for assessing instruction following and faithfulness. While these methods are standard, they cannot fully encapsulate the nuanced understanding and preferences of human evaluators. For future research, evaluating responses with human annotators would provide additional validations.</p>
</div>
</li>
<li id="Sx1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="Sx1.I1.i4.p1" class="ltx_para">
<p id="Sx1.I1.i4.p1.1" class="ltx_p">Although the purpose of our study is to study the objective discrepancies in the datasets and come up with mitigation strategies without another novel training paradigm, it would strengthen our results if we can compare our method with more recent alignment training methods.</p>
</div>
</li>
<li id="Sx1.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="Sx1.I1.i5.p1" class="ltx_para">
<p id="Sx1.I1.i5.p1.1" class="ltx_p">Although we evaluate <span id="Sx1.I1.i5.p1.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span> on unseen datasets, our method still has the potential to overfit to certain evaluators. Future work may use a different set of evaluators for a more robust evaluation. Human evaluation is challenging for our unseen testing datasets, such as WikiSum, because the input and response are extremely long, which could also make human ratings unstable.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="Sx2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Ethics Statement</h2>

<div id="Sx2.p1" class="ltx_para">
<p id="Sx2.p1.1" class="ltx_p">In this paper, we delve into the nuances of instruct-tuning LMs, a process that involves balancing a trade-off between instruction following and faithfulness. We feel that the immediate ethical and societal consequences of our research may be limited, and none which we feel must be specifically highlighted here.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bajaj et al. (2016)</span>
<span class="ltx_bibblock">
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1611.09268" title="" class="ltx_ref ltx_href">Ms marco: A human generated machine reading comprehension dataset</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Banerjee and Lavie (2005)</span>
<span class="ltx_bibblock">
Satanjeev Banerjee and Alon Lavie. 2005.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/W05-0909/" title="" class="ltx_ref ltx_href">METEOR: An automatic metric for MT evaluation with improved correlation with human judgments</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization</em>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2023)</span>
<span class="ltx_bibblock">
Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, and Junxian He. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2310.00741" title="" class="ltx_ref ltx_href">Felm: Benchmarking factuality evaluation of large language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track</em>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chiang and Lee (2023)</span>
<span class="ltx_bibblock">
Cheng-Han Chiang and Hung-yi Lee. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2305.01937" title="" class="ltx_ref ltx_href">Can large language models be an alternative to human evaluations?</a>

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chiang et al. (2023)</span>
<span class="ltx_bibblock">
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://lmsys.org/blog/2023-03-30-vicuna/" title="" class="ltx_ref ltx_href">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Christiano et al. (2017)</span>
<span class="ltx_bibblock">
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1706.03741" title="" class="ltx_ref ltx_href">Deep reinforcement learning from human preferences</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chung et al. (2022)</span>
<span class="ltx_bibblock">
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2210.11416" title="" class="ltx_ref ltx_href">Scaling instruction-finetuned language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.11416</em>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Conover et al. (2023)</span>
<span class="ltx_bibblock">
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm" title="" class="ltx_ref ltx_href">Free dolly: Introducing the world’s first truly open instruction-tuned llm</a>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Crawshaw (2020)</span>
<span class="ltx_bibblock">
Michael Crawshaw. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2009.09796" title="" class="ltx_ref ltx_href">Multi-task learning with deep neural networks: A survey</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2009.09796</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dziri et al. (2022)</span>
<span class="ltx_bibblock">
Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2204.07931" title="" class="ltx_ref ltx_href">On the origin of hallucinations in conversational models: Is it the datasets or the models?</a>

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics (NAACL)</em>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ethayarajh et al. (2022)</span>
<span class="ltx_bibblock">
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2110.08420" title="" class="ltx_ref ltx_href">Understanding dataset difficulty with <math id="bib.bib11.1.1.m1.1" class="ltx_Math" alttext="\mathcal{V}" display="inline"><semantics id="bib.bib11.1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="bib.bib11.1.1.m1.1.1" xref="bib.bib11.1.1.m1.1.1.cmml">𝒱</mi><annotation-xml encoding="MathML-Content" id="bib.bib11.1.1.m1.1b"><ci id="bib.bib11.1.1.m1.1.1.cmml" xref="bib.bib11.1.1.m1.1.1">𝒱</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib11.1.1.m1.1c">\mathcal{V}</annotation></semantics></math>-usable information</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.2.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning (ICML)</em>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fabbri et al. (2022)</span>
<span class="ltx_bibblock">
Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2112.08542" title="" class="ltx_ref ltx_href">QAFactEval: Improved QA-based factual consistency evaluation for summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)</em>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geng et al. (2023)</span>
<span class="ltx_bibblock">
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://bair.berkeley.edu/blog/2023/04/03/koala/" title="" class="ltx_ref ltx_href">Koala: A dialogue model for academic research</a>.

</span>
<span class="ltx_bibblock">Blog post.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ghosh et al. (2024)</span>
<span class="ltx_bibblock">
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Ramaneswaran S, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2402.05119" title="" class="ltx_ref ltx_href">A closer look at the limitations of instruction tuning</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.05119</em>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Han et al. (2023)</span>
<span class="ltx_bibblock">
Rujun Han, Peng Qi, Yuhao Zhang, Lan Liu, Juliette Burger, William Yang Wang, Zhiheng Huang, Bing Xiang, and Dan Roth. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/2023.findings-acl.263" title="" class="ltx_ref ltx_href">RobustQA: Benchmarking the robustness of domain adaptation for open-domain question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Findings of Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hermann et al. (2015)</span>
<span class="ltx_bibblock">
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1506.03340" title="" class="ltx_ref ltx_href">Teaching machines to read and comprehend</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2023)</span>
<span class="ltx_bibblock">
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2311.05232" title="" class="ltx_ref ltx_href">A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.05232</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Izacard et al. (2023)</span>
<span class="ltx_bibblock">
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2208.03299" title="" class="ltx_ref ltx_href">Atlas: Few-shot learning with retrieval augmented language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">The Journal of Machine Learning Research (JMLR)</em>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ji et al. (2023)</span>
<span class="ltx_bibblock">
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2202.03629" title="" class="ltx_ref ltx_href">Survey of hallucination in natural language generation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">ACM Computing Surveys</em>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kadavath et al. (2023)</span>
<span class="ltx_bibblock">
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2207.05221" title="" class="ltx_ref ltx_href">Language models (mostly) know what they know</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Findings of Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kamalloo et al. (2023)</span>
<span class="ltx_bibblock">
Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2305.06984" title="" class="ltx_ref ltx_href">Evaluating open-domain question answering in the era of large language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khattab et al. (2024)</span>
<span class="ltx_bibblock">
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Mateia Zaharia, and Christopher Potts. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2310.03714" title="" class="ltx_ref ltx_href">DSPy: Compiling declarative language model calls into self-improving pipelines</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations (ICLR)</em>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Köpf et al. (2023)</span>
<span class="ltx_bibblock">
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2304.07327" title="" class="ltx_ref ltx_href">Openassistant conversations–democratizing large language model alignment</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kwiatkowski et al. (2019)</span>
<span class="ltx_bibblock">
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/Q19-1026" title="" class="ltx_ref ltx_href">Natural questions: A benchmark for question answering research</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association of Computational Linguistics (TACL)</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Laban et al. (2022)</span>
<span class="ltx_bibblock">
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2111.09525" title="" class="ltx_ref ltx_href">Summac: Re-visiting nli-based models for inconsistency detection in summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association of Computational Linguistics (TACL)</em>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2023)</span>
<span class="ltx_bibblock">
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2309.00267" title="" class="ltx_ref ltx_href">Rlaif: Scaling reinforcement learning from human feedback with ai feedback</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.00267</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lewis et al. (2020)</span>
<span class="ltx_bibblock">
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2005.11401" title="" class="ltx_ref ltx_href">Retrieval-augmented generation for knowledge-intensive nlp tasks</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin (2004)</span>
<span class="ltx_bibblock">
Chin-Yew Lin. 2004.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/W04-1013/" title="" class="ltx_ref ltx_href">ROUGE: A package for automatic evaluation of summaries</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Text Summarization Branches Out Workshop at Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2019a)</span>
<span class="ltx_bibblock">
Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1904.02668" title="" class="ltx_ref ltx_href">Inoculation by fine-tuning: A method for analyzing challenge datasets</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics (NAACL)</em>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2018)</span>
<span class="ltx_bibblock">
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1801.10198" title="" class="ltx_ref ltx_href">Generating wikipedia by summarizing long sequences</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations (ICLR)</em>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2019b)</span>
<span class="ltx_bibblock">
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1901.11504" title="" class="ltx_ref ltx_href">Multi-task deep neural networks for natural language understanding</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023)</span>
<span class="ltx_bibblock">
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2303.16634" title="" class="ltx_ref ltx_href">G-eval: NLG evaluation using gpt-4 with better human alignment</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mishra et al. (2022)</span>
<span class="ltx_bibblock">
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2104.08773" title="" class="ltx_ref ltx_href">Cross-task generalization via natural language crowdsourcing instructions</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nallapati et al. (2016)</span>
<span class="ltx_bibblock">
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/K16-1028" title="" class="ltx_ref ltx_href">Abstractive text summarization using sequence-to-sequence RNNs and beyond</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning</em>, pages 280–290, Berlin, Germany. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2022)</span>
<span class="ltx_bibblock">
OpenAI. 2022.

</span>
<span class="ltx_bibblock">Introducing chatgpt.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">URL https://openai.com/blog/chatgpt</em>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ouyang et al. (2022)</span>
<span class="ltx_bibblock">
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2203.02155" title="" class="ltx_ref ltx_href">Training language models to follow instructions with human feedback</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Papineni et al. (2002)</span>
<span class="ltx_bibblock">
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/P02-1040" title="" class="ltx_ref ltx_href">Bleu: a method for automatic evaluation of machine translation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Paranjape et al. (2022)</span>
<span class="ltx_bibblock">
Ashwin Paranjape, Omar Khattab, Christopher Potts, Matei Zaharia, and Christopher D Manning. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2110.07752" title="" class="ltx_ref ltx_href">Hindsight: Posterior-guided training of retrievers for improved open-ended generation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations (ICLR)</em>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rafailov et al. (2023)</span>
<span class="ltx_bibblock">
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2305.18290" title="" class="ltx_ref ltx_href">Direct preference optimization: Your language model is secretly a reward model</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rashkin et al. (2021)</span>
<span class="ltx_bibblock">
Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2107.06963" title="" class="ltx_ref ltx_href">Increasing faithfulness in knowledge-grounded dialogue with controllable features</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP)</em>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schulman et al. (2017)</span>
<span class="ltx_bibblock">
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1707.06347" title="" class="ltx_ref ltx_href">Proximal policy optimization algorithms</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1707.06347</em>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Taori et al. (2023)</span>
<span class="ltx_bibblock">
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.

</span>
<span class="ltx_bibblock">Stanford alpaca: An instruction-following llama model.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/tatsu-lab/stanford_alpaca" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tatsu-lab/stanford_alpaca</a>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas
Scialom. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2307.09288" title="" class="ltx_ref ltx_href">Llama 2: Open foundation and fine-tuned chat models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.09288</em>.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023a)</span>
<span class="ltx_bibblock">
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2310.07521" title="" class="ltx_ref ltx_href">Survey on factuality in large language models: Knowledge, retrieval and domain-specificity</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.07521</em>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023b)</span>
<span class="ltx_bibblock">
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2212.10560" title="" class="ltx_ref ltx_href">Self-instruct: Aligning language model with self generated instructions</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022)</span>
<span class="ltx_bibblock">
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2204.07705" title="" class="ltx_ref ltx_href">Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022)</span>
<span class="ltx_bibblock">
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2109.01652" title="" class="ltx_ref ltx_href">Finetuned language models are zero-shot learners</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations (ICLR)</em>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zelikman et al. (2022)</span>
<span class="ltx_bibblock">
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2203.14465" title="" class="ltx_ref ltx_href">Star: Bootstrapping reasoning with reasoning</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2023)</span>
<span class="ltx_bibblock">
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2309.01219" title="" class="ltx_ref ltx_href">Siren’s song in the ai ocean: A survey on hallucination in large language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.01219</em>.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. (2023)</span>
<span class="ltx_bibblock">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2306.05685" title="" class="ltx_ref ltx_href">Judging llm-as-a-judge with mt-bench and chatbot arena</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track</em>.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Appendix</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Evaluation Metric: Instruction Following Score</h3>

<div id="A1.SS1.p1" class="ltx_para">
<p id="A1.SS1.p1.1" class="ltx_p">Table <a href="#A1.T5" title="Table 5 ‣ A.1 Evaluation Metric: Instruction Following Score ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> shows our actual prompt template for evaluation. We follow the prompt template provided in the LLM-as-a-Judge <cite class="ltx_cite ltx_citemacro_cite">Zheng et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite> paper. We normalize the return rating to [0, 1].</p>
</div>
<figure id="A1.T5" class="ltx_table">
<p id="A1.T5.14" class="ltx_p ltx_align_center">
<span id="A1.T5.14.14.14" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:867.2pt;">
<span id="A1.T5.10.10.10.10" class="ltx_p"><math id="A1.T5.1.1.1.1.m1.1" class="ltx_Math" alttext="[" display="inline"><semantics id="A1.T5.1.1.1.1.m1.1a"><mo stretchy="false" id="A1.T5.1.1.1.1.m1.1.1" xref="A1.T5.1.1.1.1.m1.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="A1.T5.1.1.1.1.m1.1b"><ci id="A1.T5.1.1.1.1.m1.1.1.cmml" xref="A1.T5.1.1.1.1.m1.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.1.1.1.1.m1.1c">[</annotation></semantics></math><span id="A1.T5.10.10.10.10.9" class="ltx_text ltx_font_typewriter">System<math id="A1.T5.2.2.2.2.1.m1.1" class="ltx_Math" alttext="]" display="inline"><semantics id="A1.T5.2.2.2.2.1.m1.1a"><mo stretchy="false" id="A1.T5.2.2.2.2.1.m1.1.1" xref="A1.T5.2.2.2.2.1.m1.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="A1.T5.2.2.2.2.1.m1.1b"><ci id="A1.T5.2.2.2.2.1.m1.1.1.cmml" xref="A1.T5.2.2.2.2.1.m1.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.2.2.2.2.1.m1.1c">]</annotation></semantics></math>
<br class="ltx_break">Please act as an impartial judge and evaluate
the quality of the response provided by an AI assistant to the user question displayed
below. Your evaluation should consider factors such as the helpfulness, relevance,
accuracy, depth, creativity, and level of detail of the response. Begin your evaluation
by providing a short explanation (strictly 1-2 short sentences). Be as objective as possible. After providing your
explanation, please rate the response on a scale of 1 to 10 by strictly following
this format: ‘‘<math id="A1.T5.3.3.3.3.2.m2.1" class="ltx_math_unparsed" alttext="[[" display="inline"><semantics id="A1.T5.3.3.3.3.2.m2.1a"><mrow id="A1.T5.3.3.3.3.2.m2.1b"><mo stretchy="false" id="A1.T5.3.3.3.3.2.m2.1.1">[</mo><mo stretchy="false" id="A1.T5.3.3.3.3.2.m2.1.2">[</mo></mrow><annotation encoding="application/x-tex" id="A1.T5.3.3.3.3.2.m2.1c">[[</annotation></semantics></math>rating<math id="A1.T5.4.4.4.4.3.m3.1" class="ltx_math_unparsed" alttext="]]" display="inline"><semantics id="A1.T5.4.4.4.4.3.m3.1a"><mrow id="A1.T5.4.4.4.4.3.m3.1b"><mo stretchy="false" id="A1.T5.4.4.4.4.3.m3.1.1">]</mo><mo stretchy="false" id="A1.T5.4.4.4.4.3.m3.1.2">]</mo></mrow><annotation encoding="application/x-tex" id="A1.T5.4.4.4.4.3.m3.1c">]]</annotation></semantics></math>’’, for example: ‘‘Rating: <math id="A1.T5.5.5.5.5.4.m4.1" class="ltx_math_unparsed" alttext="[[" display="inline"><semantics id="A1.T5.5.5.5.5.4.m4.1a"><mrow id="A1.T5.5.5.5.5.4.m4.1b"><mo stretchy="false" id="A1.T5.5.5.5.5.4.m4.1.1">[</mo><mo stretchy="false" id="A1.T5.5.5.5.5.4.m4.1.2">[</mo></mrow><annotation encoding="application/x-tex" id="A1.T5.5.5.5.5.4.m4.1c">[[</annotation></semantics></math>5<math id="A1.T5.6.6.6.6.5.m5.1" class="ltx_math_unparsed" alttext="]]" display="inline"><semantics id="A1.T5.6.6.6.6.5.m5.1a"><mrow id="A1.T5.6.6.6.6.5.m5.1b"><mo stretchy="false" id="A1.T5.6.6.6.6.5.m5.1.1">]</mo><mo stretchy="false" id="A1.T5.6.6.6.6.5.m5.1.2">]</mo></mrow><annotation encoding="application/x-tex" id="A1.T5.6.6.6.6.5.m5.1c">]]</annotation></semantics></math>’’. 
<br class="ltx_break">
<br class="ltx_break"><math id="A1.T5.7.7.7.7.6.m6.1" class="ltx_Math" alttext="[" display="inline"><semantics id="A1.T5.7.7.7.7.6.m6.1a"><mo stretchy="false" id="A1.T5.7.7.7.7.6.m6.1.1" xref="A1.T5.7.7.7.7.6.m6.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="A1.T5.7.7.7.7.6.m6.1b"><ci id="A1.T5.7.7.7.7.6.m6.1.1.cmml" xref="A1.T5.7.7.7.7.6.m6.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.7.7.7.7.6.m6.1c">[</annotation></semantics></math>Question<math id="A1.T5.8.8.8.8.7.m7.1" class="ltx_Math" alttext="]" display="inline"><semantics id="A1.T5.8.8.8.8.7.m7.1a"><mo stretchy="false" id="A1.T5.8.8.8.8.7.m7.1.1" xref="A1.T5.8.8.8.8.7.m7.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="A1.T5.8.8.8.8.7.m7.1b"><ci id="A1.T5.8.8.8.8.7.m7.1.1.cmml" xref="A1.T5.8.8.8.8.7.m7.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.8.8.8.8.7.m7.1c">]</annotation></semantics></math> 
<br class="ltx_break"><math id="A1.T5.9.9.9.9.8.m8.1" class="ltx_Math" alttext="[" display="inline"><semantics id="A1.T5.9.9.9.9.8.m8.1a"><mo mathcolor="#808080" stretchy="false" id="A1.T5.9.9.9.9.8.m8.1.1" xref="A1.T5.9.9.9.9.8.m8.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="A1.T5.9.9.9.9.8.m8.1b"><ci id="A1.T5.9.9.9.9.8.m8.1.1.cmml" xref="A1.T5.9.9.9.9.8.m8.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.9.9.9.9.8.m8.1c">[</annotation></semantics></math><span id="A1.T5.10.10.10.10.9.1" class="ltx_text" style="color:#808080;">The Question Abbreviated<math id="A1.T5.10.10.10.10.9.1.m1.1" class="ltx_Math" alttext="]" display="inline"><semantics id="A1.T5.10.10.10.10.9.1.m1.1a"><mo mathcolor="#808080" stretchy="false" id="A1.T5.10.10.10.10.9.1.m1.1.1" xref="A1.T5.10.10.10.10.9.1.m1.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="A1.T5.10.10.10.10.9.1.m1.1b"><ci id="A1.T5.10.10.10.10.9.1.m1.1.1.cmml" xref="A1.T5.10.10.10.10.9.1.m1.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.10.10.10.10.9.1.m1.1c">]</annotation></semantics></math></span> 
<br class="ltx_break"></span></span>
<span id="A1.T5.14.14.14.14" class="ltx_p"><math id="A1.T5.11.11.11.11.m1.1" class="ltx_Math" alttext="[" display="inline"><semantics id="A1.T5.11.11.11.11.m1.1a"><mo stretchy="false" id="A1.T5.11.11.11.11.m1.1.1" xref="A1.T5.11.11.11.11.m1.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="A1.T5.11.11.11.11.m1.1b"><ci id="A1.T5.11.11.11.11.m1.1.1.cmml" xref="A1.T5.11.11.11.11.m1.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.11.11.11.11.m1.1c">[</annotation></semantics></math><span id="A1.T5.14.14.14.14.3" class="ltx_text ltx_font_typewriter">The Start of Assistant’s Answer<math id="A1.T5.12.12.12.12.1.m1.1" class="ltx_Math" alttext="]" display="inline"><semantics id="A1.T5.12.12.12.12.1.m1.1a"><mo stretchy="false" id="A1.T5.12.12.12.12.1.m1.1.1" xref="A1.T5.12.12.12.12.1.m1.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="A1.T5.12.12.12.12.1.m1.1b"><ci id="A1.T5.12.12.12.12.1.m1.1.1.cmml" xref="A1.T5.12.12.12.12.1.m1.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.12.12.12.12.1.m1.1c">]</annotation></semantics></math> 
<br class="ltx_break">Answer: <span id="A1.T5.14.14.14.14.3.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">Model Generated Answer Goes Here</span>
<br class="ltx_break"><math id="A1.T5.13.13.13.13.2.m2.1" class="ltx_Math" alttext="[" display="inline"><semantics id="A1.T5.13.13.13.13.2.m2.1a"><mo stretchy="false" id="A1.T5.13.13.13.13.2.m2.1.1" xref="A1.T5.13.13.13.13.2.m2.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="A1.T5.13.13.13.13.2.m2.1b"><ci id="A1.T5.13.13.13.13.2.m2.1.1.cmml" xref="A1.T5.13.13.13.13.2.m2.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.13.13.13.13.2.m2.1c">[</annotation></semantics></math>The End of Assistant’s Answer<math id="A1.T5.14.14.14.14.3.m3.1" class="ltx_Math" alttext="]" display="inline"><semantics id="A1.T5.14.14.14.14.3.m3.1a"><mo stretchy="false" id="A1.T5.14.14.14.14.3.m3.1.1" xref="A1.T5.14.14.14.14.3.m3.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="A1.T5.14.14.14.14.3.m3.1b"><ci id="A1.T5.14.14.14.14.3.m3.1.1.cmml" xref="A1.T5.14.14.14.14.3.m3.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.14.14.14.14.3.m3.1c">]</annotation></semantics></math></span></span>
</span>
</p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>The template of the prompt we used for evaluating instruction following scores using GPT-4. This template is adopted from LLM-as-a-Judge <cite class="ltx_cite ltx_citemacro_cite">Zheng et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite> paper.</figcaption>
</figure>
<figure id="A1.T6" class="ltx_table">
<p id="A1.T6.6" class="ltx_p ltx_align_center">
<span id="A1.T6.6.6.6" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:433.6pt;">
<span id="A1.T6.6.6.6.7" class="ltx_p"><span id="A1.T6.6.6.6.7.1" class="ltx_text ltx_font_typewriter">A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user’s questions. 
<br class="ltx_break"></span></span>
<span id="A1.T6.6.6.6.6" class="ltx_p"><span id="A1.T6.6.6.6.6.6" class="ltx_text ltx_font_typewriter">### USER: <math id="A1.T6.1.1.1.1.1.m1.1" class="ltx_Math" alttext="[" display="inline"><semantics id="A1.T6.1.1.1.1.1.m1.1a"><mo mathcolor="#808080" stretchy="false" id="A1.T6.1.1.1.1.1.m1.1.1" xref="A1.T6.1.1.1.1.1.m1.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="A1.T6.1.1.1.1.1.m1.1b"><ci id="A1.T6.1.1.1.1.1.m1.1.1.cmml" xref="A1.T6.1.1.1.1.1.m1.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.1.1.1.1.1.m1.1c">[</annotation></semantics></math><span id="A1.T6.2.2.2.2.2.1" class="ltx_text" style="color:#808080;">Task-specific Instruction Abbreviated<math id="A1.T6.2.2.2.2.2.1.m1.1" class="ltx_Math" alttext="]" display="inline"><semantics id="A1.T6.2.2.2.2.2.1.m1.1a"><mo mathcolor="#808080" stretchy="false" id="A1.T6.2.2.2.2.2.1.m1.1.1" xref="A1.T6.2.2.2.2.2.1.m1.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="A1.T6.2.2.2.2.2.1.m1.1b"><ci id="A1.T6.2.2.2.2.2.1.m1.1.1.cmml" xref="A1.T6.2.2.2.2.2.1.m1.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.2.2.2.2.2.1.m1.1c">]</annotation></semantics></math></span> 
<br class="ltx_break"><math id="A1.T6.3.3.3.3.3.m2.1" class="ltx_Math" alttext="[" display="inline"><semantics id="A1.T6.3.3.3.3.3.m2.1a"><mo mathcolor="#808080" stretchy="false" id="A1.T6.3.3.3.3.3.m2.1.1" xref="A1.T6.3.3.3.3.3.m2.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="A1.T6.3.3.3.3.3.m2.1b"><ci id="A1.T6.3.3.3.3.3.m2.1.1.cmml" xref="A1.T6.3.3.3.3.3.m2.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.3.3.3.3.3.m2.1c">[</annotation></semantics></math><span id="A1.T6.4.4.4.4.4.2" class="ltx_text" style="color:#808080;">Passage: Abbreviated<math id="A1.T6.4.4.4.4.4.2.m1.1" class="ltx_Math" alttext="]" display="inline"><semantics id="A1.T6.4.4.4.4.4.2.m1.1a"><mo mathcolor="#808080" stretchy="false" id="A1.T6.4.4.4.4.4.2.m1.1.1" xref="A1.T6.4.4.4.4.4.2.m1.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="A1.T6.4.4.4.4.4.2.m1.1b"><ci id="A1.T6.4.4.4.4.4.2.m1.1.1.cmml" xref="A1.T6.4.4.4.4.4.2.m1.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.4.4.4.4.4.2.m1.1c">]</annotation></semantics></math></span> 
<br class="ltx_break"><math id="A1.T6.5.5.5.5.5.m3.1" class="ltx_Math" alttext="[" display="inline"><semantics id="A1.T6.5.5.5.5.5.m3.1a"><mo mathcolor="#808080" stretchy="false" id="A1.T6.5.5.5.5.5.m3.1.1" xref="A1.T6.5.5.5.5.5.m3.1.1.cmml">[</mo><annotation-xml encoding="MathML-Content" id="A1.T6.5.5.5.5.5.m3.1b"><ci id="A1.T6.5.5.5.5.5.m3.1.1.cmml" xref="A1.T6.5.5.5.5.5.m3.1.1">[</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.5.5.5.5.5.m3.1c">[</annotation></semantics></math><span id="A1.T6.6.6.6.6.6.3" class="ltx_text" style="color:#808080;">... ...<math id="A1.T6.6.6.6.6.6.3.m1.1" class="ltx_Math" alttext="]" display="inline"><semantics id="A1.T6.6.6.6.6.6.3.m1.1a"><mo mathcolor="#808080" stretchy="false" id="A1.T6.6.6.6.6.6.3.m1.1.1" xref="A1.T6.6.6.6.6.6.3.m1.1.1.cmml">]</mo><annotation-xml encoding="MathML-Content" id="A1.T6.6.6.6.6.6.3.m1.1b"><ci id="A1.T6.6.6.6.6.6.3.m1.1.1.cmml" xref="A1.T6.6.6.6.6.6.3.m1.1.1">]</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.6.6.6.6.6.3.m1.1c">]</annotation></semantics></math></span> 
<br class="ltx_break">###ASSISTANT: <span id="A1.T6.6.6.6.6.6.4" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">Model Generated Answer Goes Here</span></span></span>
</span>
</p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>The prompt template we used for training and evaluation for Vicuna-7B. The instruction field contains task-specific instruction, the input field contains contexts if applicable, and the response field is followed by the model generation.</figcaption>
</figure>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Evaluation Metric: Faithfulness Score</h3>

<section id="A1.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Text Normalization</h5>

<div id="A1.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px1.p1.1" class="ltx_p">We use regular expressions to replace spaces around hyphens, slashes, and before “’s”. We then remove all the articles (e.g., “a” and “the”) and punctuations. We lowercase all letters for simplicity.</p>
</div>
</section>
<section id="A1.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Common Failure Mode</h5>

<div id="A1.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px2.p1.1" class="ltx_p">The authors also human label model’s generation to check whether model’s generations are paraphrased version of the golden answers. The most common failure mode is the model extracting a non-existent span or a wrong span. This supports our findings of models being unfaithful.</p>
</div>
</section>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Experimental Setup</h3>

<div id="A1.SS3.p1" class="ltx_para">
<p id="A1.SS3.p1.4" class="ltx_p">We train our model for a maximum of a single epoch<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span>Experiments with up to three epochs showed minimal changes in results.</span></span></span> across all training jobs. We up-sample the smaller datasets to match the number of examples in the larger ones when combing datasets for training. The learning rate is set at <math id="A1.SS3.p1.1.m1.1" class="ltx_Math" alttext="1\times 10^{-5}" display="inline"><semantics id="A1.SS3.p1.1.m1.1a"><mrow id="A1.SS3.p1.1.m1.1.1" xref="A1.SS3.p1.1.m1.1.1.cmml"><mn id="A1.SS3.p1.1.m1.1.1.2" xref="A1.SS3.p1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A1.SS3.p1.1.m1.1.1.1" xref="A1.SS3.p1.1.m1.1.1.1.cmml">×</mo><msup id="A1.SS3.p1.1.m1.1.1.3" xref="A1.SS3.p1.1.m1.1.1.3.cmml"><mn id="A1.SS3.p1.1.m1.1.1.3.2" xref="A1.SS3.p1.1.m1.1.1.3.2.cmml">10</mn><mrow id="A1.SS3.p1.1.m1.1.1.3.3" xref="A1.SS3.p1.1.m1.1.1.3.3.cmml"><mo id="A1.SS3.p1.1.m1.1.1.3.3a" xref="A1.SS3.p1.1.m1.1.1.3.3.cmml">−</mo><mn id="A1.SS3.p1.1.m1.1.1.3.3.2" xref="A1.SS3.p1.1.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.1.m1.1b"><apply id="A1.SS3.p1.1.m1.1.1.cmml" xref="A1.SS3.p1.1.m1.1.1"><times id="A1.SS3.p1.1.m1.1.1.1.cmml" xref="A1.SS3.p1.1.m1.1.1.1"></times><cn type="integer" id="A1.SS3.p1.1.m1.1.1.2.cmml" xref="A1.SS3.p1.1.m1.1.1.2">1</cn><apply id="A1.SS3.p1.1.m1.1.1.3.cmml" xref="A1.SS3.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A1.SS3.p1.1.m1.1.1.3.1.cmml" xref="A1.SS3.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A1.SS3.p1.1.m1.1.1.3.2.cmml" xref="A1.SS3.p1.1.m1.1.1.3.2">10</cn><apply id="A1.SS3.p1.1.m1.1.1.3.3.cmml" xref="A1.SS3.p1.1.m1.1.1.3.3"><minus id="A1.SS3.p1.1.m1.1.1.3.3.1.cmml" xref="A1.SS3.p1.1.m1.1.1.3.3"></minus><cn type="integer" id="A1.SS3.p1.1.m1.1.1.3.3.2.cmml" xref="A1.SS3.p1.1.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.1.m1.1c">1\times 10^{-5}</annotation></semantics></math> with a batch size of 16 for our faithfulness-driven training. For training jobs involving instruction following datasets, we increase the batch size to 32. Training is conducted using <span id="A1.SS3.p1.4.1" class="ltx_text ltx_font_typewriter">bfloat16</span> precision with a maximum sequence length of 2048. The weight decay is set to <math id="A1.SS3.p1.2.m2.1" class="ltx_Math" alttext="0.05" display="inline"><semantics id="A1.SS3.p1.2.m2.1a"><mn id="A1.SS3.p1.2.m2.1.1" xref="A1.SS3.p1.2.m2.1.1.cmml">0.05</mn><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.2.m2.1b"><cn type="float" id="A1.SS3.p1.2.m2.1.1.cmml" xref="A1.SS3.p1.2.m2.1.1">0.05</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.2.m2.1c">0.05</annotation></semantics></math>, with a <span id="A1.SS3.p1.4.2" class="ltx_text ltx_font_typewriter">cosine</span> learning rate scheduler and a warm-up ratio of <math id="A1.SS3.p1.3.m3.1" class="ltx_Math" alttext="0.03" display="inline"><semantics id="A1.SS3.p1.3.m3.1a"><mn id="A1.SS3.p1.3.m3.1.1" xref="A1.SS3.p1.3.m3.1.1.cmml">0.03</mn><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.3.m3.1b"><cn type="float" id="A1.SS3.p1.3.m3.1.1.cmml" xref="A1.SS3.p1.3.m3.1.1">0.03</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.3.m3.1c">0.03</annotation></semantics></math>. We save checkpoints every 100 training steps and evaluate them based on perplexity scores on the evaluation set. The best-performing checkpoint is then selected for the next stage of training. Our models are trained using the stage-3 <span id="A1.SS3.p1.4.3" class="ltx_text ltx_font_typewriter">deepspeed</span> library. We train each model with three random seeds and average the results for consistency. For each training job, our models are trained on 8<math id="A1.SS3.p1.4.m4.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A1.SS3.p1.4.m4.1a"><mo id="A1.SS3.p1.4.m4.1.1" xref="A1.SS3.p1.4.m4.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.4.m4.1b"><times id="A1.SS3.p1.4.m4.1.1.cmml" xref="A1.SS3.p1.4.m4.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.4.m4.1c">\times</annotation></semantics></math>A100 GPUs within a single-node setup, with the total training time not exceeding 24 hours. For model generation, we employ greedy decoding with a maximum generation length of 480, which aligns with the maximum response length across the training datasets.</p>
</div>
<div id="A1.SS3.p2" class="ltx_para">
<p id="A1.SS3.p2.1" class="ltx_p">For the contined fine-tuning step in <span id="A1.SS3.p2.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span>, we use a smaller learning rate of <math id="A1.SS3.p2.1.m1.1" class="ltx_Math" alttext="8\times 10^{-6}" display="inline"><semantics id="A1.SS3.p2.1.m1.1a"><mrow id="A1.SS3.p2.1.m1.1.1" xref="A1.SS3.p2.1.m1.1.1.cmml"><mn id="A1.SS3.p2.1.m1.1.1.2" xref="A1.SS3.p2.1.m1.1.1.2.cmml">8</mn><mo lspace="0.222em" rspace="0.222em" id="A1.SS3.p2.1.m1.1.1.1" xref="A1.SS3.p2.1.m1.1.1.1.cmml">×</mo><msup id="A1.SS3.p2.1.m1.1.1.3" xref="A1.SS3.p2.1.m1.1.1.3.cmml"><mn id="A1.SS3.p2.1.m1.1.1.3.2" xref="A1.SS3.p2.1.m1.1.1.3.2.cmml">10</mn><mrow id="A1.SS3.p2.1.m1.1.1.3.3" xref="A1.SS3.p2.1.m1.1.1.3.3.cmml"><mo id="A1.SS3.p2.1.m1.1.1.3.3a" xref="A1.SS3.p2.1.m1.1.1.3.3.cmml">−</mo><mn id="A1.SS3.p2.1.m1.1.1.3.3.2" xref="A1.SS3.p2.1.m1.1.1.3.3.2.cmml">6</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p2.1.m1.1b"><apply id="A1.SS3.p2.1.m1.1.1.cmml" xref="A1.SS3.p2.1.m1.1.1"><times id="A1.SS3.p2.1.m1.1.1.1.cmml" xref="A1.SS3.p2.1.m1.1.1.1"></times><cn type="integer" id="A1.SS3.p2.1.m1.1.1.2.cmml" xref="A1.SS3.p2.1.m1.1.1.2">8</cn><apply id="A1.SS3.p2.1.m1.1.1.3.cmml" xref="A1.SS3.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="A1.SS3.p2.1.m1.1.1.3.1.cmml" xref="A1.SS3.p2.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A1.SS3.p2.1.m1.1.1.3.2.cmml" xref="A1.SS3.p2.1.m1.1.1.3.2">10</cn><apply id="A1.SS3.p2.1.m1.1.1.3.3.cmml" xref="A1.SS3.p2.1.m1.1.1.3.3"><minus id="A1.SS3.p2.1.m1.1.1.3.3.1.cmml" xref="A1.SS3.p2.1.m1.1.1.3.3"></minus><cn type="integer" id="A1.SS3.p2.1.m1.1.1.3.3.2.cmml" xref="A1.SS3.p2.1.m1.1.1.3.3.2">6</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p2.1.m1.1c">8\times 10^{-6}</annotation></semantics></math> and keep other settings the same.</p>
</div>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.4 </span>Additional Analysis on Instruction Following Fine-tuning</h3>

<div id="A1.SS4.p1" class="ltx_para">
<p id="A1.SS4.p1.1" class="ltx_p">Figure <a href="#A1.F11" title="Figure 11 ‣ A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> shows how instruction following scores vary during fine-tuning LLaMA-7B with context-dependent tasks. Figure <a href="#A1.F12" title="Figure 12 ‣ A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a> shows how instruction following scores vary during the second-stage fine-tuning with instruction following datasets.</p>
</div>
</section>
<section id="A1.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.5 </span>Quandary of Mixed Training on Abstractive and Extractive QA and Summarization Datasets</h3>

<div id="A1.SS5.p1" class="ltx_para">
<p id="A1.SS5.p1.1" class="ltx_p">Figure <a href="#A1.F9" title="Figure 9 ‣ A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> shows faithfulness score and task performance across related datasets when we fine-tune LLaMA-7B on context-dependent datasets. One suprising finding is that throughout the fine-tuning process, the faithfulness scores on extractive QA datasets gradually decrease, while task performance scores gradually increase. On the other hand, this trend is not salient for abstractive tasks as shown in Figure <a href="#A1.F13" title="Figure 13 ‣ A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>.</p>
</div>
</section>
<section id="A1.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.6 </span>Examples of Actual Instructions</h3>

<div id="A1.SS6.p1" class="ltx_para">
<p id="A1.SS6.p1.1" class="ltx_p">From Table <a href="#A1.T9" title="Table 9 ‣ A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> to Table <a href="#A1.T15" title="Table 15 ‣ A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>, we show actual model inputs from each testing datasets.</p>
</div>
</section>
<section id="A1.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.7 </span>Qualitative Examples</h3>

<div id="A1.SS7.p1" class="ltx_para">
<p id="A1.SS7.p1.1" class="ltx_p">Table <a href="#A1.T7" title="Table 7 ‣ A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> and Table <a href="#A1.T8" title="Table 8 ‣ A.7 Qualitative Examples ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> show two qualitative examples of actual model generations from our experiments.</p>
</div>
<figure id="A1.F9" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_quandary_stage_1.png" id="A1.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="250" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Individual faithfulness score and task performance with extractive QA datasets evaluated with three distinct model checkpoints of LLaMA-7B fine-tuned on context-dependent datasets. The middle checkpoint is the one with lowest in-training evaluation loss.</figcaption>
</figure>
<figure id="A1.F10" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_qa_summ_stage_2.png" id="A1.F10.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="250" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>Individual faithfulness score and task performance with extractive QA datasets across evaluated with three distinct model checkpoints through the instruction following fine-tuning. The middle checkpoint is the one with lowest in-training evaluation loss.</figcaption>
</figure>
<figure id="A1.F11" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_alignment_stage_1.png" id="A1.F11.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="200" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>Individual instruction following score on alignment datasets evaluated with three distinct model checkpoints of LLaMA-7B fine-tuned on context-dependent datasets. The middle checkpoint is the one with lowest in-training evaluation loss.</figcaption>
</figure>
<figure id="A1.F12" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_alignment_stage_2.png" id="A1.F12.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="200" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 12: </span>Individual instruction following score on alignment datasets evaluated with three distinct model checkpoints through the instruction following fine-tuning. The middle checkpoint is the one with lowest in-training evaluation loss.</figcaption>
</figure>
<figure id="A1.F13" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_abstract_stage_1.png" id="A1.F13.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="200" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 13: </span>Individual faithfulness score and task performance with abstractive QA and summarization datasets across evaluated with three distinct model checkpoints of LLaMA-7B fine-tuned on context-dependent datasets. The middle checkpoint is the one with lowest in-training evaluation loss.</figcaption>
</figure>
<figure id="A1.F14" class="ltx_figure"><img src="/html/2407.21417/assets/figures/paper_abstract_stage_2.png" id="A1.F14.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="200" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 14: </span>Individual faithfulness score and task performance with abstractive QA and summarization datasets across evaluated with three distinct model checkpoints through the instruction following fine-tuning. The middle checkpoint is the one with lowest in-training evaluation loss.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A1.T7" class="ltx_table">
<table id="A1.T7.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A1.T7.1.1" class="ltx_tr">
<td id="A1.T7.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="A1.T7.1.1.1.1" class="ltx_text ltx_font_bold">Testing Example from WikiSum Dataset</span></td>
</tr>
<tr id="A1.T7.1.2" class="ltx_tr">
<td id="A1.T7.1.2.1" class="ltx_td ltx_align_center ltx_border_t">Passage</td>
<td id="A1.T7.1.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A1.T7.1.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T7.1.2.2.1.1" class="ltx_p" style="width:398.3pt;">Take your pencil and on the top of the page, about two inches from the right start drawing a line to 2 inches (5.1cm) below the right top corner. The line should loop down and have points. (see the picture) Draw straight lines from the points in your first line to the corner. Make lines parallel to your fist line going all the way up. You should have 5 or 6 lines. Get a paper and make a cross on it, try to make both lines a similar length (using a ruler will help) Draw diagonal lines through the centre, dividing the paper up from 4 to 8 sections. Make sure they are smaller than the cross you made before. Start connecting the lines with inverted arcs, this is an arc ), from the inside out. Once you have reached the end of the web, elongate the diagonal lines, (this will make it look like it has supports). Draw a spider by making a fuzzy ball, then drawing legs (eight of them) on your web. Or see the spider drawing tip. Finished. Draw a circle and draw a cross section which also extends outside the circle. Draw two diagonal lines at the midpoint of the cross-sections which form an X-mark. Draw squares which descend in size as it nears the centre point. Draw the corners or vertices of the square along the diagonal lines. Draw diamond shapes descending in size as it nears the centre point. Draw the vertices along the lines of the cross-section. Draw curves to connect the lines 2̆013 from the squares to the diamonds, much like forming bridges. Trace with a pen and erase unnecessary lines. You may add drawings for spiders. Color to your liking!</span>
</span>
</td>
</tr>
<tr id="A1.T7.1.3" class="ltx_tr">
<td id="A1.T7.1.3.1" class="ltx_td ltx_align_center ltx_border_t" colspan="2"><span id="A1.T7.1.3.1.1" class="ltx_text ltx_font_bold">Golden Answer</span></td>
</tr>
<tr id="A1.T7.1.4" class="ltx_tr">
<td id="A1.T7.1.4.1" class="ltx_td ltx_align_center ltx_border_t">Response</td>
<td id="A1.T7.1.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A1.T7.1.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T7.1.4.2.1.1" class="ltx_p" style="width:398.3pt;">To draw a spider web, start by drawing a cross with 2 lines of equal length. Next, draw diagonal lines through the center to divide your web into 4 to 8 equal sections. Then, connect the lines with inverted arcs, or curved lines, from the inside out. When you reach the end of the web, elongate the diagonal lines to make the web look like it’s hanging from something, like a tree or shrub. Finally, draw a spider in your web by making a fuzzy ball with 8 legs.</span>
</span>
</td>
</tr>
<tr id="A1.T7.1.5" class="ltx_tr">
<td id="A1.T7.1.5.1" class="ltx_td ltx_align_center ltx_border_t" colspan="2"><span id="A1.T7.1.5.1.1" class="ltx_text ltx_font_bold">Response from LLaMA-7B Fine-tuned on Context-Dependent Datasets</span></td>
</tr>
<tr id="A1.T7.1.6" class="ltx_tr">
<td id="A1.T7.1.6.1" class="ltx_td ltx_align_center ltx_border_t">Response</td>
<td id="A1.T7.1.6.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A1.T7.1.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T7.1.6.2.1.1" class="ltx_p" style="width:398.3pt;">Draw a cross on a piece of paper. Draw a spider by making a fuzzy ball, then drawing legs (eight of them) on your web.</span>
</span>
</td>
</tr>
<tr id="A1.T7.1.7" class="ltx_tr">
<td id="A1.T7.1.7.1" class="ltx_td ltx_align_center ltx_border_t" colspan="2"><span id="A1.T7.1.7.1.1" class="ltx_text ltx_font_bold">Response from LLaMA-7B Further Fine-tuned on Instruction Following Datasets</span></td>
</tr>
<tr id="A1.T7.1.8" class="ltx_tr">
<td id="A1.T7.1.8.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">Response</td>
<td id="A1.T7.1.8.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t">
<span id="A1.T7.1.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T7.1.8.2.1.1" class="ltx_p" style="width:398.3pt;">This is a simple spider web drawing tutorial. It is a simple and easy to follow tutorial. It is a great way to practice your drawing skills.</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>WikiSum testing example from the <span id="A1.T7.3.1" class="ltx_text ltx_font_bold">first</span> pipeline of our two-stage fine-tuning paradigm where we take LLaMA-7B and fine-tune it on context-dependent datasets first, and then further fine-tune it with human alignment datasets.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A1.T8" class="ltx_table">
<table id="A1.T8.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A1.T8.1.1" class="ltx_tr">
<td id="A1.T8.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="A1.T8.1.1.1.1" class="ltx_text ltx_font_bold">Testing Example from the Alpaca-15K Dataset</span></td>
</tr>
<tr id="A1.T8.1.2" class="ltx_tr">
<td id="A1.T8.1.2.1" class="ltx_td ltx_align_center ltx_border_t">Instruction</td>
<td id="A1.T8.1.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A1.T8.1.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T8.1.2.2.1.1" class="ltx_p" style="width:398.3pt;">Construct a sentence that adequately describes a baby duck.</span>
</span>
</td>
</tr>
<tr id="A1.T8.1.3" class="ltx_tr">
<td id="A1.T8.1.3.1" class="ltx_td ltx_align_center ltx_border_t" colspan="2"><span id="A1.T8.1.3.1.1" class="ltx_text ltx_font_bold">Vicuna-7B Zero-Shot</span></td>
</tr>
<tr id="A1.T8.1.4" class="ltx_tr">
<td id="A1.T8.1.4.1" class="ltx_td ltx_align_center ltx_border_t">Response</td>
<td id="A1.T8.1.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A1.T8.1.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T8.1.4.2.1.1" class="ltx_p" style="width:398.3pt;">A baby duck is a fluffy, downy creature with a round body and webbed feet, often seen quacking and waddling around with its mother.</span>
</span>
</td>
</tr>
<tr id="A1.T8.1.5" class="ltx_tr">
<td id="A1.T8.1.5.1" class="ltx_td ltx_align_center ltx_border_t" colspan="2"><span id="A1.T8.1.5.1.1" class="ltx_text ltx_font_bold">Examples from Vicuna-7B w/ Context Dependent Fine-tuning</span></td>
</tr>
<tr id="A1.T8.1.6" class="ltx_tr">
<td id="A1.T8.1.6.1" class="ltx_td ltx_align_center ltx_border_t">Response</td>
<td id="A1.T8.1.6.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A1.T8.1.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T8.1.6.2.1.1" class="ltx_p" style="width:398.3pt;">A baby duck is a young duck that has not yet developed its full adult plumage.</span>
</span>
</td>
</tr>
<tr id="A1.T8.1.7" class="ltx_tr">
<td id="A1.T8.1.7.1" class="ltx_td ltx_align_center ltx_border_t" colspan="2"><span id="A1.T8.1.7.1.1" class="ltx_text ltx_font_bold">Examples from Vicuna-7B w/ MTL Fine-tuning</span></td>
</tr>
<tr id="A1.T8.1.8" class="ltx_tr">
<td id="A1.T8.1.8.1" class="ltx_td ltx_align_center ltx_border_t">Response</td>
<td id="A1.T8.1.8.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A1.T8.1.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T8.1.8.2.1.1" class="ltx_p" style="width:398.3pt;">A baby duck is a small, fluffy, and adorable bird that is often seen waddling around with its parents.</span>
</span>
</td>
</tr>
<tr id="A1.T8.1.9" class="ltx_tr">
<td id="A1.T8.1.9.1" class="ltx_td ltx_align_center ltx_border_t" colspan="2"><span id="A1.T8.1.9.1.1" class="ltx_text ltx_font_bold">Examples from Vicuna-7B w/ <span id="A1.T8.1.9.1.1.1" class="ltx_text ltx_font_smallcaps">ReSet</span>-S</span></td>
</tr>
<tr id="A1.T8.1.10" class="ltx_tr">
<td id="A1.T8.1.10.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">Response</td>
<td id="A1.T8.1.10.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t">
<span id="A1.T8.1.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T8.1.10.2.1.1" class="ltx_p" style="width:398.3pt;">A baby duck is a small, fluffy, and adorable bird that is often seen swimming in ponds or lakes.</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 8: </span>Qualitative examples from the <span id="A1.T8.3.1" class="ltx_text ltx_font_bold">second</span> pipeline of our two-stage fine-tuning paradigm where we take Vicuna-7B and fine-tune it on context-dependent datasets with various fine-tuning methods.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A1.T9" class="ltx_table">
<p id="A1.T9.1" class="ltx_p ltx_align_center">
<span id="A1.T9.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:867.2pt;">
<span id="A1.T9.1.1.1" class="ltx_p"><span id="A1.T9.1.1.1.1" class="ltx_text ltx_font_typewriter">Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

<br class="ltx_break"></span></span>
<span id="A1.T9.1.1.2" class="ltx_p"><span id="A1.T9.1.1.2.1" class="ltx_text ltx_font_typewriter">### Instruction: 
<br class="ltx_break">Answer to the question by extracting a specific text span from the given passages. Do not include new information beyond the given passages.

<br class="ltx_break"></span></span>
<span id="A1.T9.1.1.3" class="ltx_p"><span id="A1.T9.1.1.3.1" class="ltx_text ltx_font_typewriter">### Input:
<br class="ltx_break">Question: a bond that the issuer has the right to pay off before its maturity date

<br class="ltx_break"></span></span>
<span id="A1.T9.1.1.4" class="ltx_p"><span id="A1.T9.1.1.4.1" class="ltx_text ltx_font_typewriter">Passage: does well. Companies also reserve the right to call their bonds, which mean they can call it sooner than the maturity date. Often there is a clause in the contract that allows this; for example, if a bond issuer wishes to rebuy a 30-year bond at the 25th year, they must pay a premium. If a bond is called, it means that less interest is paid out. Failure to pay a bond effectively means bankruptcy. Bondholders who have not received their interest can throw an offending company into bankruptcy, or seize its assets if that is stipulated in the contract.
<br class="ltx_break">Passage: Callable bond A callable bond (also called redeemable bond) is a type of bond (debt security) that allows the issuer of the bond to retain the privilege of redeeming the bond at some point before the bond reaches its date of maturity. In other words, on the call date(s), the issuer has the right, but not the obligation, to buy back the bonds from the bond holders at a defined call price. Technically speaking, the bonds are not really bought and held by the issuer but are instead cancelled immediately. The call price will usually exceed the par or issue
<br class="ltx_break">Passage: options embedded. Callable bond A callable bond (also called redeemable bond) is a type of bond (debt security) that allows the issuer of the bond to retain the privilege of redeeming the bond at some point before the bond reaches its date of maturity. In other words, on the call date(s), the issuer has the right, but not the obligation, to buy back the bonds from the bond holders at a defined call price. Technically speaking, the bonds are not really bought and held by the issuer but are instead cancelled immediately. The call price will usually exceed the par
<br class="ltx_break">Passage: the amount on which the issuer pays interest, and which, most commonly, has to be repaid at the end of the term. Some structured bonds can have a redemption amount which is different from the face amount and can be linked to the performance of particular assets. The issuer has to repay the nominal amount on the maturity date. As long as all due payments have been made, the issuer has no further obligations to the bond holders after the maturity date. The length of time until the maturity date is often referred to as the term or tenure or
<br class="ltx_break">Passage: Bond (finance) In finance, a bond is an instrument of indebtedness of the bond issuer to the holders. The most common types of bonds include municipal bonds and corporate bonds. The bond is a debt security, under which the issuer owes the holders a debt and (depending on the terms of the bond) is obliged to pay them interest (the coupon) or to repay the principal at a later date, termed the maturity date. Interest is usually payable at fixed intervals (semiannual, annual, sometimes monthly). Very often the bond is negotiable, that is, the ownership of the instrument can be

<br class="ltx_break"></span></span>
<span id="A1.T9.1.1.5" class="ltx_p"><span id="A1.T9.1.1.5.1" class="ltx_text ltx_font_typewriter">### Response:
<br class="ltx_break"><span id="A1.T9.1.1.5.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">Model Generated Answer Goes Here</span></span></span>
</span>
</p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 9: </span>An example of NQ dataset with our instruction template for LLaMA-7B. Vicuna-7B’s template is slightly different as shown in Table <a href="#A1.T6" title="Table 6 ‣ A.1 Evaluation Metric: Instruction Following Score ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A1.T10" class="ltx_table">
<p id="A1.T10.1" class="ltx_p ltx_align_center">
<span id="A1.T10.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:867.2pt;">
<span id="A1.T10.1.1.1" class="ltx_p"><span id="A1.T10.1.1.1.1" class="ltx_text ltx_font_typewriter">Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

<br class="ltx_break"></span></span>
<span id="A1.T10.1.1.2" class="ltx_p"><span id="A1.T10.1.1.2.1" class="ltx_text ltx_font_typewriter">### Instruction:
<br class="ltx_break">Summarize the text in a few sentences. Using original phrases or paraphrasing them if necessary. Do not include new information beyond the given passages.
<br class="ltx_break"></span></span>
<span id="A1.T10.1.1.3" class="ltx_p"><span id="A1.T10.1.1.3.1" class="ltx_text ltx_font_typewriter">### Input:
<br class="ltx_break">(Billboard)Considering the Academy of Country Music Awards celebrated its 50th anniversary on Sunday night at the Dallas Cowboys stadium, it was bound to be bigger than any previous year’s ACMs. Plus, as hosts Blake Shelton and Luke Bryan were quick to point out, everything is bigger in Texas. Billboard: 2015 ACM Awards: See All the Photos . But bigger isn’t always better. Here’s our breakdown of the 10 best and 5 worst moments at the 2015 ACMs. The Best . Eric Church &amp; Keith Urban provide an opening wallop . With a full stadium and millions of home viewers watching, two of country’s leading men successfully lit the fuse for the 50th annual ACM Awards with a slick joint kickoff performance. Eric Church’s P̈ledge Allegiance To The Hagẅas a fine throwback, but Keith Urban’s powerhouse R̈aise ’Em Upl̈ived up to its title -- and then some. Setting a Guinness World Record . Not only was this the ACMs biggest audience ever, but the 2015 ACM Awards brought in the biggest audience for a live TV awards show ever. Now that’s how you celebrate half a century. Reba McEntire demonstrates how it’s done . During a night that found a couple performers sounding a little weak in the vocal department, Reba McEntire showed the entire Cowboys stadium -- and many younger singers who don’t have half her energy -- how it’s done. Reba is eternal. Taylor Swift singing in the audience . Whether it was during Eric Church or Martina McBride, T-Swift was heating up the audience by singing and swaying along to country staples. Taylor might have gone pop for 1̈989,b̈ut she was deep in the heart of Texas for this annual country extravaganza. Billboard: 2015 ACM Awards: And the Winners Are... Garth Brooks’ All-American salute . During Brooks’ performance of Äll-American Kid,ẗhe country giant welcomed a slew of U.S. military representatives into AT&amp;T Stadium, in an emotional moment that put the red, white and blue front and center. Brooks ended the performance by saluting the U.S. armed forces ẅho are here and are all around the world for protecting our freedom,änd also shouted out his home state of Oklahoma, who are grieving 20 years after the Oklahoma City bombing on Sunday. A truly classy moment from one of the best ever. Taylor Swift’s mom makes an appearance . Sure, Andrea Swift’s presentation of a special award for her daughter came with a fun origin tale about the creation of L̈ove Storyänd some lovely words about the 25-year-old superstar. But the sight of the elder Swift -- just days after Taylor confirmed that her mother had been diagnosed with cancer -- walking to the podium was more than enough to yield one of the night’s most poignant moments. Christina Aguilera joins Rascal Flatts . Aguilera is far from country, but with a voice as malleable as hers, she can pass for a song or two. After singing a bit of the tune she did while guesting on Nashville, Xtina joined longtime hitmakers Rascal Flatts for R̈iotf̈rom their recent album R̈ewind.Äguilera and Gary LeVox trading vocals was the rare unexpected artist pairing that actually worked. Miranda Lambert domination . In addition to kicking ass during her M̈ama’s Broken Heart/̈L̈ittle Red Wagonm̈edley, Lambert justly owned the night when it came to awards. If there’s one thing the country community loves more than Miranda Lambert, it’s giving Miranda Lambert awards. Billboard: Watch Little Big Town Bring Provocative ’Girl Crush’ to ACMs . Little Big Town don’t back down . Despite some mild controversy over their song G̈irl Crush,L̈ittle Big Town brought the poignant ballad to the awards show, giving the ACMs one of its more melancholy moments. Jason Aldean comes on strong . For a night featuring some shaky vocals, Aldean brought his silky yet powerful country croon to the ACMs during a massive medley. It’s hard to see an audience get weak in the knees when you’re watching at home, but it’s fair to assume that’s what happened during his performance. The Worst . Tony Romo . The Dallas Cowboys QB was understandably a little stiff on the mic (athletes usually aren’t the most charismatic public speakers) but the whole gag with Shelton asking Romo to toss Bryan a pass went on waaaaay too long. On the plus side, Bryan caught the pass. On the other hand, there was a tired play on words about balls. The length . Three hours for the 50th ACMs? Sure, why not. Three and a half hours? That’s pushing it. They could have shaved off the last half hour by cutting a couple of the performers who only sang half a song, and shortened a few of the massive commercial breaks. All of the Milestone Awards . Taylor Swift was given an extended honor at this year’s ACM Awards, but some of the Milestone Awards -- especially those given to Reba McEntire, Kenny Chesney and George Strait -- seemed rushed for the country giants they were saluting. It’s understandable since 2015 is the 50th anniversary of the ACMs, but sometimes, less (recipients) is more. Steven Tyler’s facial hair . See link. Twitter calling out Taylor Swift . Plenty of country fans went after T-Swizzle on Twitter, berating her for attending the ACMs after äbandoningc̈ountry music for pop. The truth is, Swift has just as many country classics under her belt as any other artist in her age range. She might have moved to pop, but don’t underplay her importance to the genre that birthed her. Billboard. All Rights Reserved.
<br class="ltx_break"></span></span>
<span id="A1.T10.1.1.4" class="ltx_p"><span id="A1.T10.1.1.4.1" class="ltx_text ltx_font_typewriter">### Response:
<br class="ltx_break"><span id="A1.T10.1.1.4.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">Model Generated Answer Goes Here</span></span></span>
</span>
</p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 10: </span>An example of CNN DailyMail dataset with our instruction template for LLaMA-7B. Vicuna-7B’s template is slightly different as shown in Table <a href="#A1.T6" title="Table 6 ‣ A.1 Evaluation Metric: Instruction Following Score ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A1.T11" class="ltx_table">
<p id="A1.T11.1" class="ltx_p ltx_align_center">
<span id="A1.T11.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:867.2pt;">
<span id="A1.T11.1.1.1" class="ltx_p"><span id="A1.T11.1.1.1.1" class="ltx_text ltx_font_typewriter">Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
<br class="ltx_break"></span></span>
<span id="A1.T11.1.1.2" class="ltx_p"><span id="A1.T11.1.1.2.1" class="ltx_text ltx_font_typewriter">### Instruction:
<br class="ltx_break">Answer the question with well-formed sentences. Paraphrase the context in the passages if necessary. Do not include new information beyond the given passages.
<br class="ltx_break"></span></span>
<span id="A1.T11.1.1.3" class="ltx_p"><span id="A1.T11.1.1.3.1" class="ltx_text ltx_font_typewriter">### Input:
<br class="ltx_break">Question: 2015 college half term holiday dates
<br class="ltx_break"></span></span>
<span id="A1.T11.1.1.4" class="ltx_p"><span id="A1.T11.1.1.4.1" class="ltx_text ltx_font_typewriter">Passage: UK School Half Term Dates 2015, 2016 and 2017. Research the latest school holidays and term dates for England, Scotland, Wales and Ireland. Welcome to Half Term Dates. The place to view School Holidays for the UK, Ireland, France and Australia. We list the latest published half term times on one handy website.
<br class="ltx_break">Passage: UK School Half Term Dates 2015, 2016 and 2017. Research the latest school holidays and term dates for England, Scotland, Wales and Ireland. Welcome to Half Term Dates. The place to view School Holidays for the UK, Ireland, France and Australia.
<br class="ltx_break">Passage: UK School Half Term Dates 2015, 2016 and 2017. Research the latest school holidays and term dates for England, Scotland, Wales and Ireland.
<br class="ltx_break">Passage: Find your child2̆019s school term, half term and holiday dates on your local council 2019s website.
<br class="ltx_break">Passage: School term dates-guide. These are the school term and holiday dates for schools in Lambeth. The dates shown do not take account of the five professional development days when schools are closed to pupils, or any other changes.
<br class="ltx_break">Passage: Find your child 2019s school term, half term and holiday dates on your local council 2019s website. School term and holiday dates vary across the UK.
<br class="ltx_break">Passage: Irish School Holidays, Mid Term Dates. Research the official half term breaks and school holidays in Ireland. The Irish authorities do not enforce the same holidays as the UK. Noticeable differences being the lack of early June half term and a 9 week long summer holiday. Commencing end of June for most Irish schools.
<br class="ltx_break">Passage: School term dates-guide. These are the school term and holiday dates for schools in Lambeth. The dates shown do not take account of the five professional development days when schools are closed to pupils, or any other changes. Please check with your child’s school for more detailed information.
<br class="ltx_break">Passage: School term dates and holidays 2014/15. Here are the school term dates and holidays for Sandwell’s primary, secondary and special schools in 2014/15. Some academies and voluntary aided church schools may not follow this schedule. You are advised to check with these schools directly.
<br class="ltx_break">Passage: Holiday Dates for college students*: 1 Autumn Half Term: 26 October 2015 to 30 October 2015 (teaching re-starts from 3/4 November 2015 due to staff P&amp;D days 2013 check with your tutor). 2 Winter Break: 18 December 2015 to 5 January 2016 (teaching re-starts from 5/6 January 2016 due to staff P&amp;D days 2013 check with your tutor).
<br class="ltx_break"></span></span>
<span id="A1.T11.1.1.5" class="ltx_p"><span id="A1.T11.1.1.5.1" class="ltx_text ltx_font_typewriter">### Response:
<br class="ltx_break"><span id="A1.T11.1.1.5.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">Model Generated Answer Goes Here</span></span></span>
</span>
</p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 11: </span>An example of MS MARCO dataset with our instruction template for LLaMA-7B. Vicuna-7B’s template is slightly different as shown in Table <a href="#A1.T6" title="Table 6 ‣ A.1 Evaluation Metric: Instruction Following Score ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A1.T12" class="ltx_table">
<p id="A1.T12.1" class="ltx_p ltx_align_center">
<span id="A1.T12.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:867.2pt;">
<span id="A1.T12.1.1.1" class="ltx_p"><span id="A1.T12.1.1.1.1" class="ltx_text ltx_font_typewriter">Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
<br class="ltx_break"></span></span>
<span id="A1.T12.1.1.2" class="ltx_p"><span id="A1.T12.1.1.2.1" class="ltx_text ltx_font_typewriter">### Instruction:
<br class="ltx_break">Answer to the question by extracting a specific text span from the given passages. Do not include new information beyond the given passages.
<br class="ltx_break"></span></span>
<span id="A1.T12.1.1.3" class="ltx_p"><span id="A1.T12.1.1.3.1" class="ltx_text ltx_font_typewriter">### Input:
<br class="ltx_break">Question: abnormalities in which chromosomes were linked to the moyamoya disease?
<br class="ltx_break"></span></span>
<span id="A1.T12.1.1.4" class="ltx_p"><span id="A1.T12.1.1.4.1" class="ltx_text ltx_font_typewriter">Passage: moyamoya disease. We postulate that a protein encoded on chromosome 21 may be related to the pathogenesis of moyamoya disease. Although the neuronal substrate is abnormal in Down syndrome patients, recovery from hemiplegic stroke in patients with MM-DS is comparable to recovery in patients with primary moyamoya.
<br class="ltx_break">Passage: Moyamoya disease (MIM 252350) is characterized by stenosis or occlusion of the terminal portions of the bilateral internal carotid arteries and by abnormal vascular networks at the base of the brain. There is a high incidence of moyamoya disease in Asia, especially in Japan. Multifactorial inheritance is estimated with lambda(s)&gt;40. Previous linkage studies have indicated that susceptibility loci for the disease are located on chromosomes 3p, 6q, and 17q. In the present study, we searched for loci linked to the disease in 12 Japanese families using 428 microsatellite markers and found significant evidence for linkage to 8q23 [maximum LOD score
<br class="ltx_break">Passage: We reported an autopsy case of Down’s syndrome with moyamoya syndrome. A 30-year-old male with Down’s syndrome suffered from a cerebral infarction and died of brain herniation. Cerebral angiography showed vascular abnormalities that were the same as moyamoya disease. Pathological findings revealed multiple stenosis of main trunk of the cerebral arteries. Pathologically, the stenosed vessels showed eccentric intimal thickness with cholesterin deposit, unlike moyamoya disease. There are only two previous reports of autopsied cases of Down’s syndrome with moyamoya syndrome. We postulate that a protein encoded on chromosome 21 may be related to the pathogenesis of Down’s syndrome with moyamoya
<br class="ltx_break">Passage: other). The karyotype was normal. No mutation in the RFN213 gene was found, and none of the HLA types linked to moyamoya disease or described in similar familial cases were identified. By describing these multisystemic associations, polycystic kidney disease for the second time, and intestinal malformation for the first time in the literature, our report expands the phenotypic variability of moyamoya syndrome. The coexistence of disparate malformations among close relatives suggests an underlying common genetic background predisposing to structural or physiological abnormalities in different tissues and organs.
<br class="ltx_break">Passage: OBJECTIVE: We report a detailed description of a family affected by a hereditary multisystem disorder associated with moyamoya syndrome.METHODS: In this family case report, we evaluated 9 members of the same family originating from Algeria. Investigations included neuroimaging, cardiologic and ophthalmologic evaluation, hormonal testing, hemoglobin electrophoresis, chromosomal karyotyping, muscle biopsy for morphology, immunohistochemistry and enzyme assays, mtDNA mutation screening, and haplotype analysis of 2 loci previously linked to moyamoya, on chromosomes 10 (ACTA2) and 17.RESULTS: Five males related through a maternal lineage were affected, suggesting an X-linked inheritance. Four of them had symptomatic moyamoya syndrome with an onset of acute
<br class="ltx_break"></span></span>
<span id="A1.T12.1.1.5" class="ltx_p"><span id="A1.T12.1.1.5.1" class="ltx_text ltx_font_typewriter">### Response:
<br class="ltx_break"><span id="A1.T12.1.1.5.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">Model Generated Answer Goes Here</span></span></span>
</span>
</p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 12: </span>An example of BioASQ dataset with our instruction template for LLaMA-7B. Vicuna-7B’s template is slightly different as shown in Table <a href="#A1.T6" title="Table 6 ‣ A.1 Evaluation Metric: Instruction Following Score ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A1.T13" class="ltx_table">
<p id="A1.T13.1" class="ltx_p ltx_align_center">
<span id="A1.T13.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:867.2pt;">
<span id="A1.T13.1.1.1" class="ltx_p"><span id="A1.T13.1.1.1.1" class="ltx_text ltx_font_typewriter">Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
<br class="ltx_break"></span></span>
<span id="A1.T13.1.1.2" class="ltx_p"><span id="A1.T13.1.1.2.1" class="ltx_text ltx_font_typewriter">### Instruction:
<br class="ltx_break">Answer to the question by extracting a specific text span from the given passages. Do not include new information beyond the given passages.
<br class="ltx_break"></span></span>
<span id="A1.T13.1.1.3" class="ltx_p"><span id="A1.T13.1.1.3.1" class="ltx_text ltx_font_typewriter">### Input:
<br class="ltx_break">Question: 3̈0 days in the holeïf you can’t name this old peter frampton band whose name refers to a forced apology
<br class="ltx_break">Passage: CLASSIC YUMMY ROCKERS | 3̈0 Days in the Holeïf you can’t name this old 
<br class="ltx_break">Peter Frampton band whose name refers to a forced apology | Humble Pie. right:.
<br class="ltx_break">Passage: 30 Days in the Hole by Humble Pie song meaning, lyric interpretation, video and 
<br class="ltx_break">... Steve Marriot had said that Peter Frampton had heard the early stages of this ... 
<br class="ltx_break">of mine with the ’Dead End Kids’ from the movie of the same name, D̈ead End,̈ 
<br class="ltx_break">... Ä Dirty room:̈ one full of evidence that can get you busted; especially if you...
<br class="ltx_break">Passage: 3̈0 Days in the Holeïs the seventh single by English rock group Humble Pie, 
<br class="ltx_break">from the band’s ... The song refers to Borstal - s̈ome seeds and dust, and you got 
<br class="ltx_break">Borstal-̈ referring to Borstal Prison and its borstal ilk ... Humble Pie’s Greatest 
<br class="ltx_break">Hits {Featuring Peter Frampton &amp; Steve Marriott]; Best of Humble Pie; Classics 
<br class="ltx_break">Volume...
<br class="ltx_break">Passage: Apr 1, 2012 ... Perhaps it was my new loose and mellow attitude, perhaps I had ... As I wrote 
<br class="ltx_break">earlier this month, it is now illegal to sell an old piano with ... Needless to say this 
<br class="ltx_break">would devastate the antique industry and force a ...... 30 Days In The Hole ... This 
<br class="ltx_break">song of Humble Pie was recorded after Frampton left the band...
<br class="ltx_break">Passage: The Irish town of Kerry lends its name to this colorful breed of pooch | A Kerry 
<br class="ltx_break">blue .... Peter Frampton band whose name refers to a forced apology | Humble 
<br class="ltx_break">Pie.
<br class="ltx_break"></span></span>
<span id="A1.T13.1.1.4" class="ltx_p"><span id="A1.T13.1.1.4.1" class="ltx_text ltx_font_typewriter">### Response:
<br class="ltx_break"><span id="A1.T13.1.1.4.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">Model Generated Answer Goes Here</span></span></span>
</span>
</p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 13: </span>An example of SearchQA dataset with our instruction template for LLaMA-7B. Vicuna-7B’s template is slightly different as shown in Table <a href="#A1.T6" title="Table 6 ‣ A.1 Evaluation Metric: Instruction Following Score ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A1.T14" class="ltx_table">
<p id="A1.T14.1" class="ltx_p ltx_align_center">
<span id="A1.T14.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:867.2pt;">
<span id="A1.T14.1.1.1" class="ltx_p"><span id="A1.T14.1.1.1.1" class="ltx_text ltx_font_typewriter">Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
<br class="ltx_break"></span></span>
<span id="A1.T14.1.1.2" class="ltx_p"><span id="A1.T14.1.1.2.1" class="ltx_text ltx_font_typewriter">### Instruction:
<br class="ltx_break">Summarize the text in a few sentences. Using original phrases or paraphrasing them if necessary. Do not include new information beyond the given passages.
<br class="ltx_break"></span></span>
<span id="A1.T14.1.1.3" class="ltx_p"><span id="A1.T14.1.1.3.1" class="ltx_text ltx_font_typewriter">### Input:
<br class="ltx_break">. Condition score the horse Condition scoring is a process in which you assess the amount of fat the horse has on it. Condition scoring requires that you look at and feel the horse’s body and assess the amount of fat it is carrying in specific areas. With condition scoring, you can evaluate whether the horse is in ideal condition or not. Once you look at and feel an area, you will write down your assessment in a chart made specifically for condition scoring. Condition scoring can take some instruction and practice, so you may want to consult with your veterinarian for some guidance on the procedure. A horse needs some fat to get it through the winter but it shouldn’t have so much fat on it that it becomes obese. Use a weight tape. A weight tape is a tool used to approximate a horse’s weight. It is a measuring tape that is wrapped around a horse’s back and chest and the measurement markings are in pounds or kilograms. Using a weight tape will not give you a completely accurate measurement. It is only an estimate. It is best used for assessing change over time. Have the horse weighed. If you are bringing your horse to a veterinary clinic or a center that has a horse scale, then you can have the horse actually weighed. However, this is not usually available to horse owners on a regular basis. Using a scale is the most accurate way to weigh a horse. Measure the horse’s weight regularly. In order to get an accurate understanding of changes to the horse’s weight over time, you will need to measure it on a regular basis. If you are very concerned about a horse’s weight, this can be every week. If you think the horse is maintaining weight fine, then every couple of weeks should be fine. Use the same type of weight measurement every time you measure the horse. This is the only way to really assess changes in weight over time. Take the horse’s winter coat into consideration. When a horse is in the cold during the winter, it can grow a thick winter coat. This can interfere with weight tape measurements and condition scoring, which is why it is important to put your hands on your horse as you take these measurements. This will enable you to feel for fat pockets and ribs beneath the horse’s wooly coat. With this coat change in mind, try to be consistent with the pressure and placement of weight tape and how you feel the horse’s body when condition scoring. If you focus on consistency, any changes you document will usually signal a real trend in weight change. Be sure to remove a horse’s blanket daily to assess weight gain or loss. Record your horse’s weight over time. To track the horse’s weight, you will need to know what its weight was before the winter. Then, you will need to keep a record of its weight throughout the winter. Make sure to write down each weight along with the date in a journal or notebook. This will be helpful to share with your horse’s veterinarian later on. Be sure that you measure the horse’s weight the same way every time. For instance, if you use a horse tape to measure the weight in the Fall, then use it to make subsequent measurements. Get professional help. If you are unsure if your horse’s weight loss is of concern or you are unsure how to help your horse gain weight, you should get some professional advice. Talk to your veterinarian about what and how much you should be feeding your horse to fatten it up. Your veterinarian will also be able to recommend further bloodwork and diagnostics if they suspect that something else, such as parasites or disease, may be causing your horse’s weight loss. Weight loss, if at all, should be gradual. Contact your veterinarian immediately if you notice extreme changes in your horse’s weight over a short period of time. Check your horse’s manure. Your horse’s manure can provide some helpful clues about their eating and drinking habits. If you notice anything different, then call your veterinarian. Some instances where you would want to call your horse’s vet include stool that is: Runny or wetter than usual, such as diarrhea. Drier than usual and/or less frequent, which may indicate constipation. This could be caused by not being able to access water because it is frozen. Check your horse’s water often in cold weather to make sure it is not frozen. A different texture or color than is normal for your horse. Identify the signs of an emergency. If your horse has lost too much weight it can be an emergency that requires immediate veterinary care. Assess your horse’s vital signs. Make sure that it is breathing regularly and that it has a strong pulse. A horse with irregular breathing or a weak pulse, or that is weak and unable to move, is in desperate need of emergency veterinary care. If any of its vital signs are not strong, you should call a veterinarian immediately. Help your horse regain weight. Horses should eat 2 - 2.5% of their body weight in good quality bulky foods, such as grass or hay, during winter. For the average 1,000 pound horse, this would mean eating 25 pounds. Make sure that your horse is getting enough. Getting your horse to gain weight requires that you feed it more calories every day than you have been. Discuss making one of the following changes to your horse’s diet with your veterinarian: Give the horse unrestricted access to hay around the clock. Change to a higher grade of hay. Add in a complete feed. Add 4 to 8 ounces of corn or cocosoya oil to a small amount of grain. Add beet pulp, sweet feed, or concentrates to your horse’s diet. 
<br class="ltx_break"></span></span>
<span id="A1.T14.1.1.4" class="ltx_p"><span id="A1.T14.1.1.4.1" class="ltx_text ltx_font_typewriter">### Response:
<br class="ltx_break"><span id="A1.T14.1.1.4.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">Model Generated Answer Goes Here</span></span></span>
</span>
</p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 14: </span>An example of WikiSum dataset with our instruction template for LLaMA-7B. Vicuna-7B’s template is slightly different as shown in Table <a href="#A1.T6" title="Table 6 ‣ A.1 Evaluation Metric: Instruction Following Score ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A1.T15" class="ltx_table">
<p id="A1.T15.1" class="ltx_p ltx_align_center">
<span id="A1.T15.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:867.2pt;">
<span id="A1.T15.1.1.1" class="ltx_p"><span id="A1.T15.1.1.1.1" class="ltx_text ltx_font_typewriter">Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
<br class="ltx_break"></span></span>
<span id="A1.T15.1.1.2" class="ltx_p"><span id="A1.T15.1.1.2.1" class="ltx_text ltx_font_typewriter">### Instruction:
<br class="ltx_break">Compile a list of ethical considerations for the development of AI systems.
<br class="ltx_break"></span></span>
<span id="A1.T15.1.1.3" class="ltx_p"><span id="A1.T15.1.1.3.1" class="ltx_text ltx_font_typewriter">### Input:
<br class="ltx_break">
<br class="ltx_break"></span></span>
<span id="A1.T15.1.1.4" class="ltx_p"><span id="A1.T15.1.1.4.1" class="ltx_text ltx_font_typewriter">### Response:
<br class="ltx_break"><span id="A1.T15.1.1.4.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">Model Generated Answer Goes Here</span></span></span>
</span>
</p>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 15: </span>An example of Alpaca-15K with our instruction template for LLaMA-7B. Vicuna-7B’s template is slightly different as shown in Table <a href="#A1.T6" title="Table 6 ‣ A.1 Evaluation Metric: Instruction Following Score ‣ Appendix A Appendix ‣ Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.21416" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.21417" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.21417">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.21417" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.21418" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 14:53:54 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.21417
Modern language models (LMs)  need to follow human instructions while being faithful; yet, they often fail to achieve both.
Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions)  and faithfulness (i.e., ground responses in given context)  when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instruction-tuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL)  with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies onRejection Sampling for ContinuedSelf-instructionTuning (ReSet) , which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as trainingReSetwith high-quality, yet substantially smaller data (three-fold less)  yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs.
abstract
Dancing in Chains: Reconciling Instruction Followingand Faithfulness in Language Models
Abstract
2407.21417
1IntroductionAligning language models (LMs)  with human preferences becomes increasingly important. One main objective is to train LMs to follow human instructions (e.g., answering open-ended questions) 
while being faithful (e.g., grounding responses in the given context) . However, LMs often suffer from failing to follow human instructionsor making up facts that are not grounded in context.We trace this problem back to commonly used alignment training datasets, often collected from naturalistic conversations covering a wide range of domains. For instance, Alpacaand Dolly-15Kcover tasks from creative writing to context-dependent QA. Particularly, these tasks may have distinct objectives, and, when mixed, may induce potential conflicts of interest during alignment.Figure 1:Faithfulness scores on context-dependent tasks (QA and summarization)  decrease when we fine-tune grounded LLaMA-7B checkpoint with instruction following datasets (orange) , and instruction following scores (assessed by GPT-4)  decrease when we fine-tune Vicuna-7B with context-dependent tasks (blue) .
Our method,ReSetsurpasses the vanilla MTL with data mixing, approaching the North Star (upper right corner) .In this context, we examine the interaction between instruction following (i.e., how well does the LM follow open-ended instructions)  and faithfulness (i.e., is the LM’s response grounded in the context)  during alignment training.
Specifically, we study how instruction following and faithfulness scores change when adapting LMs to two types of datasets: 1) instruction-tuning datasetsthat are commonly used to train chat-models such as ChatGPTand Llama-2-Chat; and 2) context-dependent datasetsthat require grounding to a providedcontext, and are commonly used to train Retrieval-Augmented Generation (RAG)  modelssuch as Atlasand DSPy.
We observe a clear trade-off between the two scores, as shown in Figure1: a fine-tuned LM with a competitive faithfulness score becomes much less faithful when separately fine-tuned on instruction following datasets. Conversely, an instruction-tuned LM becomes worse at following instructions when fine-tuned on context-dependent datasets. Our findings suggest fine-tuning an LM with either instruction following or context-dependent datasets exclusively may impair its original ability in the other aspect.One natural mitigation strategy is to use multi-task learning (MTL)  by mixing datasets, which we find as a strong yet sub-optimal baseline.
To achieve a more synergic outcome, we propose a simple yet effective methodRejection Sampling for ContinuedSelf-instructionTuning (ReSet) .
Inspired by recent works in self-instruct,ReSetleverages the LM to sample generations for instruction following and task-specific datasets, which is different from vanilla MTL. Generations are then rated by external judges for instruction following and faithfulness scores, where top-rated generations are collected and used to further fine-tune the LM.
Our experiments show that when trained withReSetusing only a single iteration and 8,000 additional fine-tuning examples, LMs see substantial gains in faithfulness scores (up to +18.8%)  compared to the MTL baseline, while maintaining their instruction-following scores.
Furthermore, we find that less is more: training withReSeton higher quality yet three-fold less data yields up to 31.3% improvements on faithfulness among datasets compared to MTL.
Our analyses shed lights on finding and mitigating objective discrepancies in alignment training where datasets encompass different or even conflicting goals.111We will release our code and evaluation data athttps://github.com/frankaging/dancing-in-chains.DatasetTrainDevTestAvg. LengthInstruction Datasets257,3072,500-305Alpaca-15K--31,32379Vicuna-eval--80367Koala-eval--180444NQ69,6398,75713,3686CNN/Daily Mail287,11313,36811,49076MS MARCO153,7252,50012,46623BioASQ--1,9569SearchQA--31,7604WikiSum--2,000140Table 1:Data statistics of two-stage fine-tuning experiments.Italicizeddatasets are held-out unseen evaluation datasets, while the rest are used for training. “Instruction Datasets” here consist of publicly avaliable datasets such as Dolly-15K, ShareGPT, Self-Instructand OASST-1. Average length is the averaged response token length, which is calculated with training or testing sets for each dataset with the LLaMA-7B tokenizer.
introduction
Dancing in Chains: Reconciling Instruction Followingand Faithfulness in Language Models
1Introduction
2407.21417
2BackgroundTo recap, our goal is to study the interplay of training on instruction following datasets and on datasets that require grounding to a specific context.
To do this, we first construct two separate groups of datasets, each resembling a setting above.
We then adopt a two-stage fine-tuning paradigm that allows us to closely study the effect of each type of training.
We introduce the setup in this section.Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.### Instruction:Task-specific Instruction Abbreviated### Input:Passage: Abbreviated... ...###Response:Model Generated Answer Goes HereTable 2:The prompt template we used for training and evaluation for LLaMA-7B. The instruction field contains task-specific instruction, the input field contains contexts if applicable, and the response field is followed by model’s generation. We use a different template for Vicuna-7B since it uses a different template during instruct-tuning phase. Actual input examples for each dataset and model are included in AppendixA.6.Dataset TypeInstructionExtract QAAnswer to the question byextracting a specific text spanfrom the given passages. Do not include new information beyond the given passages.Abstractive QAAnswer the question with well-formed sentences. Paraphrase the context in the passages if necessary.Do not include new informationbeyond the given passages.SummarizationSummarize the text in a few sentences. Using original phrases or paraphrasing them if necessary.Do not include new informationbeyond the given passages.Table 3:Task specific instructions. We bold texts that indicate that our prompts are designed to be objective-aligned with our instruction following training data (i.e., fine-tuning our model on instruction following datasets should with keeping it to be faithful as well) .2.1DatasetsWe outline different datasets used for instruction following and context-dependent fine-tuning. Table1shows data statistics. For each dataset we carefully design the instructions to well-align the input and output (see Section2.3) .2.1.1Instruction Following DatasetsWe curated an instruction following training dataset by compiling unique examples from publicly available datasets such as Dolly-15K,
ShareGPT222https://huggingface.co/datasets/LLMs/Alpaca-ShareGPT, Self-Instruct, and OASST-1. For evaluation, we gather unique examples from Alpaca-15K, Vicuna-eval, and Koala-eval.Data Pre-processingWe exclude examples that originally come with context (e.g., examples labeled as summarization type in Dolly-15K are filtered out)  to prevent overlaps with our context-dependent datasets. We retain only unseen examples in our evaluation sets.
For OASST-1, we only include examples with an average human rating higher than 0.5 (i.e., the higher the rating is, the higher the quality is)  and that are rated by at least two annotators. To refine ShareGPT, we include only examples with responses that are longer than 10 words, split by whitespace. For other instruction following training datasets, we exclude examples with empty instructions or responses.2.1.2Context-Dependent DatasetsWe aim to evaluate the model’s generation for faithfulness to the given input context. We select a range of context-dependent datasets from three task domains: (1)  extractive QA including NQ, BioASQ and SearchQA, taken from the RobustQA benchmark; (2)  abstractive QA with MS MARCOwhere the answers are well-formed sentences grounded in context; and (3)  abstractive summarization including CNN DailyMailand WikiSum. BioASQ, SearchQA and WikiSum are hold out for evaluation and the rest are for training. One crucial advantage of using these context-dependent datasets is that they provide us with a reliable way of measuring faithfulness, in terms of how well the response is grounded in the given context.Data Pre-processingFor QA datasets, we include five retrieved passages maximally as the context,333Details about the retrieving process can be found in the original papers of RobustQAand MS MARCO.where the gold answer is at least mentioned in one of the passages. For MS MARCO, we only include examples where there exist at least one well-formed answers. In cases involving multiple retrieved passages, we concatenate all passages with line breaks inserted between them.2.2Evaluation MetricsIn the context of our datasets, we evaluate our models with three metrics: instruction following score, faithfulness score and task performance score. The standard methods for measuring instruction following and faithfulness of language models are subject to ongoing debate. In this work, we employ widely adopted approaches for these measurements and compile a set of metrics for more stable evaluations.444Throughout the paper, we sample a subset of the full evaluation data which include 6,000 examples (1,000 examples from each context-dependent evaluation set) , and sample 300 examples (100 examples from each instruction following evaluation set)  due to limited compute resources.To present our findings, we report macro-averaged results across all test datasets.Instruction Following ScoreWe adopt the commonly used evaluation paradigm proposed by LLM-as-a-Judge, and zero-shot prompt GPT-4 to provide a rating followed by a short explanation (i.e., named as LLM-as-a-Judge (R)  in the paper) .555We usegpt-4-0613.For the GPT-4 evaluator, we set the temperature to 0 for stability with a maximum generation length of 512. We check instruction following scores only for instruction following evaluation datasets. See AppendixA.1for our actual evaluation prompt.Faithfulness ScoreFor extractive QA datasets, we utilize the span coverage score as our metric (i.e., whether the predicted answer is a span within the context) . We apply standard normalization to both the predicted answer and the context (see AppendixA.2for details) . A score of 1.0 is assigned if the span is covered, and 0.0 otherwise. Additionally, we include unigram and bigram coverage for selected datasets to further refine our faithfulness evaluation in the Appendix (see Figure9and Figure10) . For abstractive QA and summarization datasets, we employ SummaC-ZeroShot (SummaC-ZS;)  to assess whether the provided context (with the question concatenated as a prefix for QA datasets)  entails the model-generated answer. Specifically, we segment both the context and the answer into sentences and then use a pretrained NLI model to compute an entailment score between pairs. These NLI scores are aggregated across pairs using max-pooling.Task PerformanceTask performance complements the aforementioned metrics, as task performance is generally expected to decline when models are fine-tuned across domains. We utilize Exact Match (EM) 
for extractive QA datasets and the ROUGE-L scorefor abstractive QA and summarization datasets.2.3Instruction TemplateOur datasets follow the same instruction-tuning template format as used in the Alpaca setting. This template includes a header that outlines general guidelines, followed by task-specific instructions, as illustrated in Table2. For QA tasks, the question is presented after the retrieved passages.Objective-Aligned InstructionsWe design our task-specific instructions to minimizeobjective-conflictamong datasets. For example, in prompting our model for a context-dependent extractive QA task, we explicitly instruct the model to “extract a specific text span from the given passages”. This template helps models to reduce hallucination when fine-tuned with instruction following datasets: as models getting better at understanding human instructions, they also get better at understanding how toextracting a spanwhich ensures the answer to be faithful. The instructions for all datasets are depicted in Table3. Our task-specific templates ensure there is no objective conflict when we fine-tune our models with mixed of datasets. We use the same template for training and evaluation for each task.Figure 2:Two-stage fine-tuning with LLaMA-7B.Figure 3:Macro-averaged faithfulness, instruction following, and task performance scores on corresponding evaluation datasets before and after fine-tuning with instruction following datasets.Figure 4:Average generation token length throughout the instruction following training stage. The first checkpoint is the best checkpoint from the context-dependent training stage. The middle checkpoint is with the lowest evaluation loss during the second stage.Figure 5:Faithfulness scores for abstraction QA and summarization datasets categorized by whether the generation length is strictly shorter or much longer (100 tokens)  than the golden answer.2.4Two-stage Fine-tuning ParadigmTo understand the trade-off between instruction following and faithfulness when training LMs with both objectives, we formulate a two-stage fine-tuning paradigm to answer our research questions (as illustrated in Figure2) .666Our approach is akin to the data inoculation paradigm proposed by, albeit with significantly larger models and datasets.For our first pipeline, we initially fine-tune our LM with context-dependent datasets that require grounding. We then take the best checkpoint from the initial stage to further fine-tune it on instruction following datasets (CDIF) . Conversely, in our second pipeline, we fine-tune instruct-tuned LM (e.g., Vicuna-7B)  with context-dependent datasets (IFCD) . For both pipelines, we measure instruction following and faithfulness scores before and after training, to gauge the impact of the second-stage training on both capabilities. We follow this paradigm to find evidence of the trade-off in Section3and Section4.ModelsWe use two models in our two-stage fine-tuning paradigm. We use a base LM LLaMA-7B, one of the most widely used open-source LM, for the CDIF pipeline. For our IFCD pipeline, we use Vicuna-7B off-the-shelf as our instruct-tuned LLaMA-7B without retraining one from scratch. Vicuna-7B is one of the most competitive open-source chat-model, and is a fine-tuned LLaMA-7B on conversational data from ShareGPT.
Our hypothesis and paradigm are transferable to other base LMs at different scales, although we pick these two models as they are among the first open-sourced LMs during the time frame of this project. Other experimental setup details are included in AppendixA.3.Figure 6:Macro-averaged faithfulness, instruction following, and task performance scores on corresponding evaluation datasets before and after fine-tuning with context-dependent datasets.Figure 7:Instruction following scores for 1,000 randomly selected examples from Alpaca-15K (left) , and for filtered examples with similar generation lengths (maximally 10 tokens longer)  before and after fine-tuning with context-dependent datasets (right) .
body
Dancing in Chains: Reconciling Instruction Followingand Faithfulness in Language Models
2Background
2407.21417
3Does Fine-tuning with Instruction Following Data Hurt Faithfulness?To answer this question, we follow the first pipeline outlined in Figure2, where we take our LLaMA-7B that is fine-tuned on context-dependent datasets and further fine-tune it with instruction following datasets. Our results are shown in Figure3. First of all, instruction following scores increase drastically as expected, from 0.30 to 0.74. Meanwhile, faithfulness scores on abstractive datasets drop 33.0% from 0.82 to 0.55, while the task performance is largely maintained (from 0.34 to 0.32) . For extractive datasets, both faithfulness scores and task performance see a relatively small drop. To provide a fine-grained view of these datasets, we provide additional analysis of per-task and per-metric changes of faithfulness score as well as task performance in AppendixA.4and AppendixA.5.One potential confounding factor for faithfulness scores dropping on abstractive tasks is the length of the generated response.
The model could simply have generated longer responses (Figure4)  as a result of training on intruction tuning data with longer responses (Table1) .
To rule this out, we re-evaluate our models separately with two contrasting groups: evaluated with only those examples with shorter generation length (less than or equal to) , and those examples with much longer generation length (100 more tokens)  compared with the golden answer. As shown in Figure5, both short and long generations see very substantial drops in faithfulness, while longer generations indeed see larger drops. This nevertheless supports our conclusion that instruction following training hurts faithfulness.Figure 8:The illustration of our proposed methodReSet. It samples generations from the initial vanilla multi-task learning (MTL)  checkpoint with seen examples from instruction following and context-dependent datasets. For each example, it generates a set of possible responses with different decoding strategies. Generations are rated by external judges with a weighted scores of task performance, faithfulness and instruction following scores. Then, the top rated generations will be collected to further fine-tune the initial model.
body
Dancing in Chains: Reconciling Instruction Followingand Faithfulness in Language Models
3Does Fine-tuning with Instruction Following Data Hurt Faithfulness?
2407.21417
4Does Context-Dependent Fine-tuning Hurt Instruction Following?To answer this question, we fine-tune Vicuna-7B (LLaMA-7B fine-tuned on ShareGPT)  on context-dependent datasets. Figure6shows our results. As expected, our model becomes more faithful, with improved task performance. This improvement is partly because Vicuna-7B does not follow our instruction template out-of-the-box. On the other hand, instruction following scores drop by 37.9%, suggesting our model becomes much less human-aligned. Similar to previous findings, one natural concern is that context-dependent training makes our model generate much shorter responses, which leads to lower instruction following scores. To rule out this concern, we randomly sample 200 examples from the Alpaca-15K dataset and only keep those with a minimal sequence length change (a maximum of 10 new tokens)  compared to the corresponding Vicuna-7B zero-shot generation. As shown in Figure7, instruction following scores consistently drop considerably for the filtered setting, regardless of length (36.8% vs. 22.0%) . Overall, context-dependent fine-tuning negatively affects the instruction following score, as we expected.
body
Dancing in Chains: Reconciling Instruction Followingand Faithfulness in Language Models
4Does Context-Dependent Fine-tuning Hurt Instruction Following?
2407.21417
5ReSet: Reconciling Instruction Following and FaithfulnessAs shown in the previous two sections, there are clear objective discrepancies between diverse instruction following datasets and context-grounded ones. In this section, we explore two different mitigation strategies, starting from the MTL baseline to our novel method based onRejection Sampling for ContinuedSelf-instructionTuning (ReSet) , which significantly outperforms MTL.Our MTL BaselineOur MTL simply mixes up the instruction following as well as context-dependent datasets. We up-sample smaller datasets to have an equal number of examples per dataset.5.1Our MethodReSetstarts with a model that is fine-tuned on a mixture of instruction following and context-dependent datasets and reconciles the two objectives with the following steps (Figure8) :Sample GenerationsFor a random subset of the training datasets, we sample generations from the checkpoint with different decoding settings. Specifically, we focus on two hyperparameters by randomly changing one of them at a time by enumerating all possible values. For the decoding temperature, it takes on a value from {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7}. For top-k, we set it to take on a value from {5, 10, 20, 50, 70, 90, 100}. When we vary the temperature, we fix top-k to be 0. Similarly, when we vary the top-k, we fix the temperature at 0. For each example, if we run the sampling procedure once, it will sample 7 examples in total.External JudgesWe then use a set of external judges to rate our collected generations as in Section2.2. These judges are offline evaluators. We evaluate generation based on instruction following score, faithfulness score and task performance. One potential limitation is that our model may overfit to existing judges on seen datasets. We thus include unseen datasets to test generalizability.Top-1 Weighted ScoreFor each generation, we collect a set of scores from our external judges. Then, we take a weighted sum of these scores before ranking the generations and picking the top rated one. The score is weighted score of task performance, instruction followingand faithfulness scores,where we useandto indicate whether the example is from our instruction following datasets or context-dependent datasets.
We pick the top rated sample per example and combine them into a continued fine-tuning dataset as described next.Continued Fine-tuningForReSet, we randomly sample training data from each of our four dataset in Table1and collect 2,000 additional fine-tuning examples per dataset. In total, our new fine-tuning dataset has 8,000 examples. With this small collected dataset, we further fine-tune our starting checkpoint model for a single epoch with a smaller learning rate to avoid overfitting. This continued fine-tuning step is very lightweight as the training data is usually less than 1% of the MTL training step. Overall,ReSetresembles a reject sampling based preference learning paradigm, which has been proven effectivewhile drastically saving training costs as well as increasing stability. Other experimental setup details are included in AppendixA.3.SuperchargedReSet(ReSet-S) In addition, we evaluateReSetwith a different setting to test the impact of the quality of our continued fine-tuning dataset. Specifically, wesuperchargethe quality of our additional fine-tuning dataset by sampling 1more generations. We swap our instruction following judge from GPT-4 to the weaker ChatGPT777We usegpt-4-0613.. One potential benefits is that ChatGPT is a weaker judge with a lower recall for good generations. As a result, examples rated high by ChatGPT may have higher quality.
While sampling for generation, we decrease the curated fine-tuning dataset by 3-fold (2,000 examples in total) . We fine-tune our checkpoint model with the same setup.FaithfulnessInstruction FollowingAbstractive DatasetsExtractive DatasetsDatasetsCNN DailyMailMS MARCOWikiSumOverallNQSearchQABioASQOverallAlignment EvalVicuna-7B0.360.590.340.430.060.210.020.100.79w/o MTL0.670.930.850.820.890.940.930.920.49w/ MTL0.670.920.800.800.900.940.930.920.75w/ReSet0.800.900.880.860.960.990.920.960.73w/ MTL+ReSet0.770.930.950.850.960.970.980.970.76w/ MTL+ReSet-S0.880.930.950.920.940.970.970.960.77Table 4:Faithfulness and alignment scores on testing datasets, and unseen datasets areitalicized.
Scores are averaged across three distinct runs. Higher scores are better. Overall scores are macro-average across datasets.5.2ResultsTable4shows results on models fine-tuned with other baselines:
(1) Vicuna-7B: Evaluated with Vicuna-7B in a zero-shot manner.
(2) w/o MTL: We fine-tune Vicuna-7B with context-dependent datasets without any MTL objective. The purpose is to establish a potential upper bound of faithfulness score without any further training for instruction following.
(3) w/ MTL: We fine-tune Vicuna-7B with baseline MTL by mixing instruction following and context-dependent datasets together.
(4) w/ReSet: We fine-tune Vicuna-7B directly with our collected continued fine-tuning dataset (8,000 examples)  from ourReSetpipeline without MTL training first. Note that the curated dataset comprises of sampled generations from our MTL checkpoint. The purpose is to evaluate whether MTL is necessary for model improvements.
(5) w/ MTL+ReSet: We follow Figure8to fine-tune our MTL checkpoint with curated dataset.
(6) w/ MTL+ReSet-S: We follow Figure8to fine-tune our MTL checkpoint with the high-quality version of our curated dataset (2,000 examples) .We evaluate our models on both seen and unseen testing datasets. As shown in Table4, fine-tuning Vicuna-7B with a mixture of datasets close the gap on instruction following score substantially (from 0.49 to 0.75)  while leaving headroom across the board. Next, models fine-tuned withReSetsignificantly outperform the MTL baseline on both seen and unseen testing datasets. Our results also suggest directly fine-tuning Vicuna-7B performs worse compared to fine-tuning a model checkpoint after MTL training. Last but not least, our results withReSet-S provide strong evidence that data quality is more important than data quantity and that using multiple iterations of sampling helps, asReSet-S achieves similar or better performance with 3-fold less training data. Due to space limit, we present qualitative model generations in AppendixA.7.
body
Dancing in Chains: Reconciling Instruction Followingand Faithfulness in Language Models
5ReSet: Reconciling Instruction Following and Faithfulness
2407.21417
6Related WorkInstruction Following of LMsThere are a variety of instruction following training datasetscovering tasks from wildly different domains such as poetry creation to context-dependent summarization. In this paper, we focus on a particular trade-off between instruction following and model faithfulness, which are fundamental objectives for modern LMs. In addition to training datasets, LM generations are often assessed by human expertsor model scorers. We use GPT-4 to access instruction following scores as in.Faithfulness and Groundedness of LMsBeing faithful or grounded is crucial for tasks like context-dependent QA or summarizations, reducing hallucinations for LMsUnlike task performance, faithfulness measures whether generated answers are based on the given context. For context-dependent QA and summarization benchmarks, common metrics include subsequence-based lexical matching, natural language inference, and more recently, LLMs as factuality scorers. We use lexical matching and NLI to measure faithfulness by checking whether the answer is grounded.Instruction Following Training with LMsBase LMs are often tuned to follow human instructions. Various post-hoc fine-tuning techniques have been proposed to align base LMs with human preferences, such as supervised fine-tuning on instruction-following datasetsor variances of RLHF and RLAIF. These techniques can require training policy and reward models which is not cost-efficient. Inspired by alignment training with reject-sampling, we propose a simple yet effective method that uses reject-sampling by self-instruct. On the other hand, there exist various advance MTL techniquesprior to the development of LMs. However,ReSetis the only variant that leverages LM’s self-instruct generations to further fine-tune the LM. More importantly,ReSetis complimentary to different instruct-tuning techniques by leveraging the LM and external evaluators to generate high-quality continued fine-tuning datasets.
Compared with recent works showing how instruct-tuning may cause hallucination, we provide a more rigorous analysis with our two-stage fine-tuning paradigm and focus on the interaction between instruction following and faithfulness.
body
Dancing in Chains: Reconciling Instruction Followingand Faithfulness in Language Models
6Related Work
2407.21417
7ConclusionIn this paper, we demonstrate a clear trade-off between instruction following and faithfulness when fine-tuning LMs on datasets with differing objectives.
To alleviate this trade-off, we proposedReSet, a simple yet effective iterative method that significantly outperforms the MTL baseline in both instruction following and faithfulness scores. The lightweight and iterative nature ofReSetmakes it extensible for future refinement at minimum cost and integrable with recent instruct-tuning techniques. We contribute to the broader goal of creating more reliable, accurate, and user-aligned language technologies.
body
Dancing in Chains: Reconciling Instruction Followingand Faithfulness in Language Models
7Conclusion
2407.21417
LimitationsThe limitations of our work are as follows:•Our study primarily focuses on the LLaMA-7B and Vicuna-7B models, that are among the best open-source models at the time of this work. While we posit that our findings and the proposedReSetmethod could generalize across other language models, our findings could remain speculative without evaluating on more current model types (e.g., LLaMA-2, Mistral or Mixtral at various scales) .•The datasets chosen for fine-tuning and evaluation, though comprehensive, are not exhaustive. There are other interesting datasets that are not covered in this study. For instance, long-form QA where the answers are much longer than 1-2 phrases or sentences. Our instruction following datasets can also be further categorized into creativity-driven, world-knowledge driven and others to help us to disentangle discrepancies in objectives better across datasets.•Our evaluation relies heavily on automated metrics and external judges like external LMs for assessing instruction following and faithfulness. While these methods are standard, they cannot fully encapsulate the nuanced understanding and preferences of human evaluators. For future research, evaluating responses with human annotators would provide additional validations.•Although the purpose of our study is to study the objective discrepancies in the datasets and come up with mitigation strategies without another novel training paradigm, it would strengthen our results if we can compare our method with more recent alignment training methods.•Although we evaluateReSeton unseen datasets, our method still has the potential to overfit to certain evaluators. Future work may use a different set of evaluators for a more robust evaluation. Human evaluation is challenging for our unseen testing datasets, such as WikiSum, because the input and response are extremely long, which could also make human ratings unstable.
body
Dancing in Chains: Reconciling Instruction Followingand Faithfulness in Language Models
Limitations
2407.21417
Ethics StatementIn this paper, we delve into the nuances of instruct-tuning LMs, a process that involves balancing a trade-off between instruction following and faithfulness. We feel that the immediate ethical and societal consequences of our research may be limited, and none which we feel must be specifically highlighted here.
conclusion
Dancing in Chains: Reconciling Instruction Followingand Faithfulness in Language Models
Ethics Statement